I let IE delete them, so I'm sure I always have an updated page. If I was on a modem though I would probably keep them limited.
Under Tools/Options there is a checkbox to empty the temp files folder on exiting IE...however this won't work unless you shutdown O.E. as well.