Some likely culprits:
a) Virus scanning / spyware scanning of the contents.
b) An explorer view that has to read additional data from / about the files is selected, e.g. document previews, image thumbnails, movie clips, whatever.
c) A 3rd party explorer add-in is running that slows it down, something like a CD/DVD burning plug-in, whatever.
d) It's scanning for things like network shares or removable media / removable disc devices that it can't access at the moment.
e) your drive is highly fragmented
f) Your physical ram is pretty ful and it is paging to disk a lot when you switch to explorer from whatever else was running.
g) you have billions of files/directories in the folder
h) You're using some kind of compressed / encrypted mode on that folder and it is slowing down.
i) you may have something "mounted" within that area as a reparse point / junction or whatever and that's slowing it down.
See if it does it in SAFE MODE.
Try disabling some of the START-UP, autostart, and task-bar type applications, see if it gets faster.
