OpenGLnb = OpenGL with no bilenear filtering. IE, big ol' squares when running full-screen. Ddraw (windows only), overlay, and surface are probably better choices than this. I can't find screenshots, but I recall trying it and not being impressed. Nearest neighbor isn't always the same as proper scaling. It might be good for a slow computer, though, that can't do square-ish scaling by the CPU.
OpenGL = OpenGL with bilinear filtering. Fuzzy stretching. Ew.
OpenGLhq = my understanding is that this applies an arbitrary scaling factor shader with similar effects to hq2x, hq3x, etc..