Very interesting idea. If you find anything like this, I'd sure like to hear about it.
This has to be done in real time. The cancellation would be done by using a speaker to produce the exact same sound the computer is making but turning it perfectly out of phase. If you record it, the phase of the speaker output will never line up perfectly opposite the direct sound produced by the computer. In this case you will have the opposite effect that you want, you will double the volume instead of cutting it.
CoolEdit may have an invert function, but you have another problem. It has to be done in real time as explained above. Then, you have to remove the sound input to the mic from the output of the speaker or you will get feedback. Maybe not enough feedback to squeal, but certainly enough to ruin the process you are attempting. Although I'm not sure, my guess is if you put CoolEdit into input mode, you can't invert and if you aren't in input mode, you aren't in real time.
So, you have to find something that will take the input of the mic, invert it, and play it in real time (with VERY little delay to the output) without mixing ANY of the input with the output.
Great idea, tough to implement. Post here if you figure anything out.