Well (oops, hit submit too soon.. sorry),
Anyhow, yes and no... Most likely you have a cap buzzing on the new card. This would be a fault of the cheap cap used in the build process. However a poor PSU with noisy signals can exaggerate these problems also. Infact, most PC problems can be attributed in some form to bad or dirty power. Good power supplies are, IMHO, the most important part to any build. Everything starts with a good anchor and base.
Now, personally I'd exchange the card itself. But if you're in the market anyhow, you could get a new PSU first to see if it helps any but unless yours is really bad, I wouldn't expect it to cure the problem.