It's common and harmless.... unless it annoys you.
As stated in Bozo's link, which references a HardForums post from a former SMPS EE, the whining comes from a vibration of the windings on a transformer or coil. The transformers are supposed to be "glazed" with a sealant to keep them from whining, but if there's any loose windings, it'll whine.
The motherboard making the whine is the same thing, only it's the windings of a coil vibrating against something else, which can also happen in a PSU. Easy way to troubleshoot this is to take a popsicle stick and touch the coils on the motherboard until the squealing stops. I wouldn't bother doing this with a PSU since you'd have to open it and void the warranty.
It's not leaking caps.