I work and have worked for several national ISPs. I write and maintain RADIUS servers. Almost all logins on the net are done using RADIUS. This is for dial, xDSL, cable, VPN, and a whole bunch of other stuff. Each RADIUS packet contains the 'assigned ip address'. Sometimes only the accounting stop record contains this but you always get it.
If RIAA or someone else (I have replied for request for IP addresses for a number of reasons, the most common being email spammers, I have never been asked for RIAA or similar) asks for 'who was using the IP address during this time period' I go through my logs and find the user that was using the thing during that time. In some cases, I have had online database lookups for this info going back one year. In other words, you give me an IP address and a time and I can immediately tell you the persons name, address, home phone, payment method for account, payment history etc. Of course this info can then be used to get even more info from credit bureaus etc.
What RIAA does is basically use a PC to search for users sharing target profiles, usually something like 1000 songs and some number of those top 40 say. Since most sharing is done p2p once they find you as a source, they know your IP address via netstat or buy using a sniffer installed on their PC or one another PC on the same segment. I suspect that they use sniffers to captuer the complete transaction search. Do a google search and take a look at Etherreal, a freeware sniffer than anyone can run on their PC (along with whatever) or to sniff the segment.
When you start talking about monitoring at the router level, that becames a big problem because of the overhead it would put on the router itself. Secondly, most networks now uses switches, MPLS, ATM, VPN protocols which make it very difficult to put a sniffer someplace in the backbone that can monitor everything. No network that I know of is willing to spend the millions of dollars to put permanent taps on their backbones. Not that it can't be done. But who would pay for it? RIAA finds target users and takes them to court, costing the pirate (or his/her parents) several thousand dollars and time. They then make a big splash with these examples in order to scare off other would-be sharers.
The technology they use is fairly simple. Some day somebody(s) are going to figure out how to allow for sharing and for artists/creators to get their money too. The result will end up making more money than microsoft has for bill gates. I personally don't mind paying for things I want and if no one pays, then entertainment will suffer. For me the neat thing about p2p is the ability to search and sample all kinds of weird music, old speeches, and things that noone cares about making money on. For instance, if you want to listen to mongolian throat music, you probably aren't going to be able to get that off of Napster or iTunes. But you could off of Kazza (lite, I wouldn't touch Kazaa media desktop with osama's PC).