I'd be more worried about the simple consumer routers being vunerable then the modems.

I suppose DSL modems are complex enough to be vunerable, but I never thought about that before.
There have been a few router/firewall that have had serious flaws in them that can be exploited. Firmware updates can fix them.
Then again there are a few bad seeds like some of Belkin's home routers that were intentionally designed to hijack a browser requests from it's clients computers and send them to advetisments on the web!! I think it did one in 60 requests on port 80 or something like that. They were hoping nobody would catch on and they would make a bit on the side from the advertising revenue!
If I was a hacker that wanted to attack someone with a specific IP address, but had his computer turned off, I would simply write a script that would ping his computer every ten or fifteen minutes with one packet. Once it responded then I would have the script do whatever exploit I wanted to try out on him in order to get my code (the trojan, rootkit, whatever) to run on his computer. That way I don't have to try to hack a turned off computer!
I'm still interested in how you determined that your computer was hacked. Plenty of rootkits are easily detectable with anti-virus tools, and I suppose that many trojans can be identified as root kits by some companies because they sound scarier.
But I also know that since it's hard to detect some kits you can get false positives with some software.
I remember that chkrootkit (a common linux rootkit detection tool) has warned me several times that I may have a possible LKM rootkit installed on my computer. This was because it found proccesses running and detected them thru the /proc/ directory, but they didn't turn up in a "ps" command.(or visa versa, It was a while ago) Got me worried for a while! But it turned out to be a bug with the kernel and it didn't correctly indicate a couple kernel-specific deamons.
I suppose something like that could happen in windows easily, too. It would suck to have to reinstall over nothing.
Like it was said before the only realy reliable way to detect them is to boot up on seperate protected media (like a knoppix CD) and then try to detect them. That way the rootkit would have no way to protect itself from system scans and stuff like that since it would be dorment.