Certain types of connection, namely ethernet, SCSI and USB are designed to be hot-swapped. There should be no damage caused from hot-swapping such devices.
Most other things, such as keyboard, mouse, parallel or serial ports could potentially be damaged. I'll admit though, that I've done this occasionally, and never damaged anything.