What other hardware or hardware features should be paid close attention too?
RAID 1...the rest is fluff. You don't need 8 channel RAID controllers or Xeon processors, and some of the comments above regarding Xeon -vs- Core Duo sound like garbage you'd hear in a Mac forum so they can feel special.
The only meaningful difference between Xeon/Opteron server processors and their consumer counterparts is the server processors can be used in multi-processor arrays. For instance, there is no such thing as a dual processor Core Duo. Some other minor architectural differences allowing for better utilization of large amounts of server memory, etc.
A 2.8ghz Core Duo with simple RAID 1 controller will have identical performance as a 2.8ghz dual core Xeon Server 99.9% of the time, and the former will be a helluva lot cheaper.
Throw Apache on it, and use any basic Windows OS and it will be solid and pretty secure. Or, hold your breath and pay too much for a 5-cal Server 2003 box, which you don't need anyways.
At first it's fun to host your own Web box, but after awhile you'll get sick of your ISP's politics, dynamic IP's, crappy upload speeds annoying your clients, blah, blah, blah. At that point you'll find cheap hosting.