Unless you have an immensely powerful magnet, say an electromagnet, it's unlikely that you could damage much of anything in your system, including your hard drive.
A few years back, my brother-in-law and I performed an experiment where we "magnetized" an already magnetic Phillip's head screwdriver to give it near super-magnetic properties. We stuck it to every component inside of an older 486 system, the CPU, RAM, hard drive, chipset, video board, sound - everything! We even over-did it and let it sit on certain components for a while.
The bottom line: nothing was effected. I'm not saying to throw caution to the wind, but I'd be more concerned about electrostatic discharge, which can do more damage than the puny magnet on your screwdriver ever will.