the N68-VGS3 FX is a pcie 1.x board, but any VGA should work I think (it's also limited to Hyper Transport at 1000MHz, it uses a chipset designed for AMD K8)
but yes, we need more details to recommend something,
anyway, to me 7770 is quite decent for the money, as the 650 Ti, 650 Ti boost, 7790, 7850...