I've used both default VGA and device specific drivers on my Windows 2000 Servers, and found that the VGA drivers are less problematic. The video drivers that come with Windows 2000 Server CD often have issues with screen artifacts when coming out of power saving mode, so you should make sure to get the latest video drivers for your server if you need more than 640x480 screen resolution.
Of course, you'll probably be doing most of your server administration remotely, so video drivers aren't a big issue.