Well I don't imagine the cache filling up is an issue with the router. I can't even guess what cache it would be, since your router certainly isn't receiving data faster than it can send it to your computer, and your computer shouldn't be sending the data any faster than the router can forward it over the DSL (the TCP/IP standard includes allowing a device to tell another device to slow down transmission if it can't keep up). If it did cache your sent data, it would result in your traffic having higher latencies because your computer assumes the traffic is already gone and counts all the time it's cache. The router also wouldn't have any reason to be caching actual data other than possibly DNS information, which I don't think should be taking up 4MB even with several thousand servers listed.
What kind of router is it? Do you have the latest firmware on it?
Does the issue happen with any other games?