Psuedo DDoS (& mitigation)

[randal]

Hey guys, thought I'd post my story and ask for everybody's input. I have a 7206 at a site that has two full DS3s coming into it. At that site is a free DNS provider called granite canyon. Somebody on the web pointed "intrenet.com" at GranCan's DNS servers without entering any zone information, resulting in non-cacheable NoSuchDomains being flung around the net.

Non-cacheable DNS == tons more queries == tons of bandwidth

The 7206 started dropping packets when it hit ~40mbps of throughput, simply because of the sheer number of itty-bitty DNS queries coming through. At ~80bytes/packet, that's 62,500pps. Tag on CEF, OSPF, BGP4, tons of ACLs and more, and the router started dropping packets. Bad times.

The short term answer was to rate-limit the client for an immediate performance improvement, then to put in bogus, cacheable DNS info for the intrenet.com domain so that requests would begin to peter off.

I know you guys must have some fantastic stories about your networks, so let's hear 'em!

 

[spidey07]

wow, that really, really, really should not have dropped a 7206. Some isn't right with it (switching path) as it can do 200K to 1000K pps. Something is causing it to use process swithcing, bad moj.

My story was an entire data center brought down because somebody plugged in a linksys router.

 

[randal]

Severely underpowered 7206 -- NPE100, non-VXR. Gotta roll old-skool. :-/ Also -- and I'm not sure on this but I heard it somewhere -- once NetFlow/CEF start topping out, the router drops to process switching. Had a link to it somewhere, will dig it up when I have a few minutes.

On the linksys, did they plug it in backwards? Do tell

 

[spidey07]

here's the router performance specs for all routers...

http://www.cisco.com/warp/public/765/tools/quickreference/routerperformance.pdf

yeah, NPE 100 is gonna hurt you, but it still should have handled it.

On the linksys, somebody put in a default route. Linksys decided to proxy arp for everything on its subnet (the backbone routing subnet), hence routers coudln't arp for each other because the linksys responded with its own MAC address.

 

[bgroff]

With the "tons of ACLS" I can believe it. Were you getting a full/partial BGP4 feed as well?

Better question, did that incident convince your client to retire that museum piece? (Okay, sorta museum piece. I've been sitting at a client site with AGS+ and IGS routers IN PRODUCTION.)

 

[n0cmonkey]

Originally posted by: spidey07
wow, that really, really, really should not have dropped a 7206. Some isn't right with it (switching path) as it can do 200K to 1000K pps. Something is causing it to use process swithcing, bad moj.

My story was an entire data center brought down because somebody plugged in a linksys router.

I remember that thread. :laugh:

:beer:

 

[randal]

Originally posted by: bgroff
With the "tons of ACLS" I can believe it. Were you getting a full/partial BGP4 feed as well?

Better question, did that incident convince your client to retire that museum piece? (Okay, sorta museum piece. I've been sitting at a client site with AGS+ and IGS routers IN PRODUCTION.)

Yea, one full & one partial feed. Also have a 4-way eth card and 2x FE cards, all with pretty big ACLs. I did some looking and the ACLs can have a large performance hit. Agreed, though, it's a relic and needs to go.

edit - just looked at it because I was unsure if it had 1 full/partial ... has *partial*

 

[spidey07]

with the newer NPEs the ACLs are done in hardware, so yeah you're going to take a processor hit with that original NPE. I think it is about 6+ years old.