It seems from other threads that the general consensus is to opt for a higher resolution over using AA. A lot of people seem to use 1280x1024 or higher and keep AA off but have AF on 8x or 16x. I gave this a try, since I used to use 1024x768, 4x AA, 8x AF, but they ended up being pretty similar in both frame rate and image quality. It depends on the game though. UT2004 I can play at 1280x1024 and AA/AF, but with something like Joint Ops I shut off AA so that 1280x1024 is smooth.
Another thing to consider is that if your cpu is slower and you feel you are limited mostly by that, then drop down the resolution a little but crank up the AA/AF since with a powerful GPU these options are almost "free" at a lower resolution.