I never have much on my desktop, except maybe the odd Word doc that hasn't made it to my docs folder yet. I prefer the start menu or quick launch, since it doesn't require me to minimize everything to get back to the desktop. But I've seen people who have every possible spot of the desktop filled with something. Do whatever works for you.
The only time that I can think of where a lot of stuff on the desktop will slow you down is if you use roaming profiles. If you have a lot of data on the desktop, you will have to wait for that data to be copied from the server while you log in. So it can cause a delay when logging in before you see the desktop.