First step would be to see if it's the case speaker, since that's easy. Just unplug it.
Next step would be to isolate whether it's the power supply. Unplug all the power connectors from the mainboard, drives, video card, whatever. Take the 20/24-pin ATX connector and a bent paper-clip or piece of wire, and short the one green wire on the connector, to any of the black wires on the connector. That will start the power supply.
If that seems to be okay, then you can move on to isolating other items. Pull out all the expansion cards (video and whatever else) and disconnect all drive cables and fan power connectors. Leave the memory and CPU installed, and the heatsink mounted but unplug the CPU fan cable, it'll be safe enough to run for a few seconds to test. Plug in only the ATX power connector and whatever auxiliary power connectors your mainboard uses (4-pin usually). Turn it on, and if you get the whine then there's something wrong with the mainboard in all likelihood. You can also try it with the memory modules pulled out, you should get an error beep that way.
If that works, then it may be a bad drive, or a fan. Test with the CPU fan plugged in, and any case fans, one by one. Then try with the video card plugged in. Then plug in the hard drives and optical drives one by one, until you end up either finding out which one causes it, or discover that disconnecting and reconnecting everything magically fixed it.
