You'll find hot swap drives mostly of the SCSI variety, and yes, they are most often used in servers, rarely in desktops.
ie; You are running a server. Most servers need to be up and running 24/7/365. So you are running 1 or more raid arrays of disks on that same server. If you are running raid 1 or 5 or some of the variants to them, you might get a PFA alert (SCSI has PFA, IDE has SMART) telling you the drive is about to fail or the drive may have failed already without warning. You can now "hot swap" the drive in question without bringing the server down or disconnecting users. Performance will suffer until the raid host adapter rebuilds the data content of the old drive to the new drive from the other drives in the array.
You need special drives and drive trays/carriages to do this on a server in order to protect the sensitive electronics as you hot plug them.
Some servers have hot swap fans, PCI slots ( slots cmust have power interupted to just the slot to replace card, and supporting software must be on system for PCI hot swap/management). There are also hotswap power supplies.
IBM (and some others) have hot swap memory - infact IBM is one of the few who have hot add/swap memory on the Intel platform - other platforms have been doing this for years....
IBM, and a few select others can also do processor hot swap, and they as well have been playing with that on Intel sytems though the cost of implementation is too high today to sell and it's not perfected except for very vertical applications.
It's all done to enable maximum server uptime, and allow techs to do scheduled upgrades during business hours if need be.