Kaido, rack-mounting is a lot more expensive in terms of capital equipment, but allows you to make things neat and secure (as in bolted down) which will help you in terms of operational cost. Do it if you can. Can't tell you how many times I've had folks bump into tower servers, or think they're just another PC and unplug them.
Dell and HP servers are *okay* - but you can definitely do better. I'm partial to Supermicro and Tyan pre-integrated whiteboxes, you can find vendors on the net who will build you a system based on them. These are motherboard+case+power supply+fans+drive backplanes all pre-integrated, just add CPU, RAM, and RAID controller. That's a lot lower risk than building your own totally custom. Supermicro in particular is very common in real data centers. When buying a rack-mount system, get a cabinet and rails, and investigate to make sure the rails are good. For example, the Dell PowerEdge 2650/2850 rails I think are pretty good. The Dell 1850 rails suck. The Supermicro rails are okay. With rackmount servers, you want a four-post cabinet and rails, or you will have much pain.
Opterons are good. I like Opterons. But there's a lot more volume of P4/Xeons, and more deals to be had on those.
I would strongly urge you to use ECC memory on every single PC you have, period. That includes both desktops and servers. If you don't have ECC RAM, you have a lot of very high speed storage that is critical to your system with absolutely no error detection or correction - and remember that JEDEC specs allow one in 10^-8 bit operations to err. In practice, the error rate on good quality RAM is a few order of magnitudes less, but you also have a whole lot of bit operations going on, especially on a loaded server. You WILL have errors creep into memory. Do you want them to corrupt your data, or not?
If you're worried about parts availability, buy another box and leave it in the rack as a cold spare. Don't buy desktop grade parts so you can buy replacements quickly at retail. Buy good parts, and keep your own spares. I also personally prefer to have cold spares rather than a vendor extended warranty. I've had way too many long phone arguments with call centers in India and/or Texas where my on-site x-hour warranty got turned into them shipping parts in a few days and having a technician available in a week or more. If it really has to work, I don't trust any outside vendor not to let me down...
Power wise, if you have to go UPS, check out Liebert. I recently found out they made UPSs for mere mortals. Data centers use a lot of higher-end Liebert gear, they really understand power (that, or they *really* understand sales, but the folks doing the buying for that level of gear usually wouldn't be fooled easily). APC is a very very popular vendor, but remember that these guys started out as a desktop PC UPS vendor and went up-market. I used to be a fan of Best Power, who started out building data center sized gear and moved downmarket, but they got acquired and disappeared. But Liebert is similar. I put a lot more faith in people who engineer high-end power solutions being able to move to lower-end stuff than folks who started low-end moving to higher-end stuff.
Oh, and make sure you have really good power feeds into your server room including really good grounds.
