Ya if it's just a server don't worry about the card. No game I know of has the server do any actual 3d calculations.
Don't use the server as a gaming box either.
Fast CPU is nice. Dual CPU's is great. It helps with the mutlitasking greatly, not so important much for brute speed.
Plenty of RAM is nice, but excessive amounts gets wasted. It's nice for running mutliple servers out of one box, but if your only going to run one game server, then stick with a single CPU.
You have to monitor that, because if the games have to dip into the harddrive swap for proccessing data your server thruroughput is going to die. RAM is 100,000 times faster then Disk.
The speed of the disk isn't so important as long as you have enough RAM. Once the data gets loaded up in to ram the disk speed doesn't matter. But a nice 8meg 7200 is a defininate plus.
Unfortunately the big $$ is going to spent on the connection. Fast connections with lots of upload speed is going to be 100x more expensive in the long run then your server box (say over a period of a year or 2). A nice SDSL line can be a couple hundred a month.
So think about it this way:
If you have a 1Mb/s SDSL line. (1Mb/s upload and download. limit) That's about T1 speeds.
If you want 20 people on that then that means they each have a max of 51kb between them.
That's about the same amount of bandwidth you get from a very healthy dial up line.
1.1Mb/s SDSL line is the fastest I can get in my area. No adsl.
SpeakyEasy (a nice DSL ISP that is very supportive of sharing bandwidth with neighbors, and customers hosting servers and such) prices the 1.1Mb/s SDSL line at 200 dollars a month. The 768Mb/s line goes for about 159 bucks.
ADSL is often much cheaper. They have a special for a "gamer" ADSL setup. 6Mb/s (!) download, 768Mb/s upload. That's 106 dollars a month after the initial cheap special expires. (I want it, but it's not aviable in my area.)
But that would leave 20 players with a max 38Kb download speed (limited by your upload speed) each.
So that's probably going to be your biggest expense.