And the time required to put it back in memory is minimal.if a driver or subsystem is idle enough that it gets paged out the chances of you needing it are slim.
Than you're probably just fine, the real problem will come in when you try and run applications that have greater memory requirements than your system has to offer and your system has to start paging pieces of those applications... If you arent experiencing that than you should be fine.I can't tell any difference at all with 512MB RAM.