Hmm, sorry I was searching for this on google and it's high in the ranking in a search for "ground isolator" right now, so I thought I'd add my two cents.. A ground isolator might be a good first step (it's $20 and refundable if it doesn't work

). With computers, you have a sort of unique problem; while you won't get the characteristic 60 Hz hum associated with line potential differences across devices with linear power supplies, instead the computer (and perhaps your home amplifier) has a load sensitive power suppy that dynamically adjusts for load using high frequency switching to regulate the voltage and current . Therefore, the fact that the noise appears when using the breakout box on the front of the case may mean that somehow the DC offset from the switching noise in the power supply is leaking into the circuit and then being amplified on the amplifier connected to the headphones (or built into the headphones).
The sounds you're hearing when you move the mouse, load files off the hard drive, or close windows, are the changes in load on the power supply. I've experienced the same thing with my laptop, and the radio shack ground isolator fixed it up. while you lose some of the near DC sub bass ( >40Hz ) and perhaps saturate the sound a bit, it does remove the dc offset that may be interfering with the breakout part of your sound card. You'll still be sacrificing something in terms of sound quality, but I've found it's worth it. If you're extremely sensitive to the changes, it might be time to try another sound card that contains its own isolation system/ground lifting.
Of course it might be RF radiation, but most shielded audio cables, even if it is just a pigtail to a breakout box have fairly good noise rejection. On the other front, modern sound cards - even built-in ones - do have very good ground planes and are much more carefully designed than people give them credit for. The acid test is a regular pair of headphones. If you can't hear this noise on a set of passive, inductive (wire connects directly to the voice coil) headphones, the problem exists somewhere after the output stage of your sound card. If you do hear noise with headphones, then you have some kind of interference issue in your computer. and short of relocating wires or the position of the sound card itself, this may be difficult to resolve without replacing the unit.
I hope that helps
-Richard