Don't turn it off. One of the major reasons that people see Win7 as being more memory efficient than windows is because MS changed the way they report memory usage, They used to report it in terms of actual free memory, now they include the cached memory as freed memory.
Why would you want gigs and gigs of memory basically doing nothing? It is a waste. Freeing data from memory is an extremely fast process, so why not try to load it up with applications you frequently use? It essentially costs nothing and provides a big boost.