the downsides?
well lets put it this way.. the benefits are not worth what the downfalls could be
i would be willing to bet a large sum of money, that if the PC is working fine now, not forever... i can guarantee this will take away the life of the PS, and maybe even the inner components if something goes wrong in the PC.. it could get ugly. another downfall? your case will heat up inside like mad, the PS box gets pretty freaking hot with no fan, esp. when it has a large draw (AMD machine anyone?) the heat will increase fairly largely inside the case if you ask me, and since you are going for a quiet PC, likely you don't have any exhaust fans. i bet that CPU will get pretty warm!!!
I'd never do that!!!