Another "it's simple" vote. I've built 3 over the last two years or so, the most recent just a few weeks ago. IMO, the hardest part is getting the floppy drive connected correctly - I always plug the cable in bass-ackwards

. Hint: if you boot the computer for the first time and the little green "access" light on the drive stays lit, it's connected incorrectly.
aside from that, the other small diff difficulties have already been pointed out, I think:
-getting all the little connectors from the front of the case (Power, Power LED, harddrive LED, speaker, etc) on correctly is a pain. Doing it wrong won't hurt anything, but the PC may not boot, or some of th lights may not come on.
-Always make sure you're electrically grounded when dealing with CPUs and circuit boards (mainboard & add-in cards). That way you won't fry anything with static electricity.
-be careful putting the heatsink on; a)you don't want to crack the core and b) if you have to use a flathead screwdriver, you don't want to slip & put a gouge in the motherboard. Oh, and make sure the CPU fan is connected or your computer won't last very long
Nate