AT got hit with DDoS today

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
I thought we already went through this in the late 90s?

The signatures for a DDoS are well formed. As such so are the tools built into the major network vendors and intrusion detection systems.

1) IDS/IPS see bad behavior
2) IDS sends dynamic ACLs to routers
3) Routers shun "bad" traffic

Am I missing something here? Or is this why I do this for a living?

AT isn't to blame of course, its their hosting service/provider that should have these countermeasures already in place.

I understand that a well crafted DDoS attack is very difficult to stop. But at the same time there is technology out there to thwart it. Heck there is basic mitigation in IOS.
 

Boscoh

Senior member
Jan 23, 2002
501
0
0
You can have all the protective measures you want, but in a real heavy DoS, or DDoS, the only way to stop it is to have your upstream provider shun the traffic. Even if your perimeter router or firewall blocks it, the traffic is still traversing the link to even get to your perimeter router or firewall.

A LOT of people don't understand that concept. "Oh hey, this firewall gives me DDoS/DoS protection. Cool. I'll NEVER have to deal with that problem!" :roll:

I think you're right in saying thats probably what happened today. The main site wasn't down for me however...just the forums.
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
Originally posted by: Boscoh
You can have all the protective measures you want, but in a real heavy DoS, or DDoS, the only way to stop it is to have your upstream provider shun the traffic. Even if your perimeter router or firewall blocks it, the traffic is still traversing the link to even get to your perimeter router or firewall.

A LOT of people don't understand that concept. "Oh hey, this firewall gives me DDoS/DoS protection. Cool. I'll NEVER have to deal with that problem!" :roll:

I think you're right in saying thats probably what happened today. The main site wasn't down for me however...just the forums.

Well that's what I'm saying. No matter what the customer does the only way to stop a DDoS lays on the provider.

I'm calling to suspect a provider that can't stop it. Then again this maybe why I'm still employed.
 

Mark R

Diamond Member
Oct 9, 1999
8,513
16
81
But isn't it the case, that if a DDoS is large enough, many providers may themselves be overwhelmed - meaning that the responsibility must lie with their bandwidth providers? In which case, who should it really be the ISP that is responsible?

What if the DDoS is indistinguishable from legitimate traffic? What can you do then, if you don't want to shut down your site?
 

Boscoh

Senior member
Jan 23, 2002
501
0
0
Smaller ISPs can get slammed often times. It would take a really massive DDoS to bring someone like SBC, Level3, or AT&T to their knees. Smaller ISP's would likely work with their upstream provider to deal with it. You get to a certain point, and it becomes all about who has more bandwidth.

I dont know how AT got DDoS'd....probably SYN or ACK flooding. You'd usually just block new connections on the premise of "if there are 'x' number of new connections in 'x' seconds, its probably not legit, so shun them all." Often times that shuts out legit customers, but when you're in a situation like that, you're pretty screwed any way you slice it.

In the past, I think many companies have emergency-transferred the caching of their site to big front-end caching services like Akamai. They have the resources to deal with not-quite-totally-massive DoS attacks, so the impact to your site would have a good chance of being minimal if they were fronting it. I seem to remember Microsoft doing that with Blaster, or whatever it was that orchestrated at DDoS on their site.
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
I'd like to continue the discussion of this that started in the Forum Issues thread...

About this reverse tracing thing...what is it, and how is it done? If the source IP is spoofed it's kinda hard...

I don't know what you mean when you say this stuff is well-established on how to prevent DDoSes. When you say signature, you mean something within the packets signifies/differentiates a DDoS one from a real one? Or you mean it's obvious to tell when you're getting hit? While I don't know the high-level stuff like Cisco at all, I have a well-established knowledge on the low-level of it and I don't think the Cisco stuff really matters when you get to the nitty gritty of what it's based on. If you overpower the attacker, the DDoS is effectively avoided as well.

An ACL is useless if the IPs generated are random. Only a dumbass would make all the DoS packets like out of a list of 1000 IPs. I think most DoS attacks are spoofed these days, or else it would be really easy to prevent them.

We don't know if they were hit by a botnet using legitmate IPs, or a botnet chugging out tons of random-IP'd packets.
 

Boscoh

Senior member
Jan 23, 2002
501
0
0
Are you talking about Reverse Path Forwarding? That checks the source IP of every packet coming into the router and makes sure that it could have actually come from one of the networks in it's routing table for the interface the packet came in on. So say a router only has the 2.2.0.0/16 network in its routing table, and it recieves a packet with the address 3.4.5.6. Obviously, that packet couldn't have legitimately come from 2.2.0.0/16, so the router shuns it.

The other method is to just back-trace each router that forwarded the packet and hope you eventually get to the source. I've never done this, but I'd think it would be difficult considering you'd have to get administrators from each ISP to check their stuff.

Doing a traceroute on a spoofed IP wont solve anything and, depending on the type of D/DoS it is, the majority of IP's are spoofed.
 

spidey07

No Lifer
Aug 4, 2000
65,469
5
76
I'm up watching a big storm hitting us, but RPF (as covered above) basically says "drop any packet on an ingress interface if it is sourced with an IP address that the router doesn't believe"

every router has a routing table, the heart of a router that tells it where to forward packets. Here's a quick example...

interface 1 - route 1.1.0.0/18 to this interface
interface 2 - 2.2.0.0/16
interface 3 - 3.3.0.0/18

So let's say int 2 get an IP sourced by 1.1.4.5. Router will drop it because 1.1.4.5 "must" come from the prefix (subnet) 1.1.0.0/18 and should be received on int 1. So its pretty obvious that this IP address is spoofed.

As far as bots with legitimate source IPs, that's where the number of SYNs per time period is flagged as "bad" and then dropped.

But spoofed IP addresses should not be a concern (unless they are spoofed from the same subnet as the hosts "real" IP. Bots with legitimate IPs will need behavior and rate information to be stopped.

here's a decent link...
http://www.cisco.com/en/US/netsol/ns615...ions_white_paper0900aecd8011e927.shtml
 

deadseasquirrel

Golden Member
Nov 20, 2001
1,736
0
0
I work at MCI (UUnet... and soon-to-be Verizon) and we're actually doing this for our customers now. I think marketing has called it WAN Defense or something like that.

Basically, we announce a BGP route that diverts the offending traffic to a mitigation center where we apply metrics and determine if the traffic is an attack. If it is, we separate that traffic from the legit, if there is any, and forward the legit on to the client.

We provide a full graphical log of packets, destination, sources, and an audit trail for any law enforcement requirements the company has. Pretty cool stuff.
 

Goosemaster

Lifer
Apr 10, 2001
48,775
3
81
Originally posted by: deadseasquirrel
I work at MCI (UUnet... and soon-to-be Verizon) and we're actually doing this for our customers now. I think marketing has called it WAN Defense or something like that.

Basically, we announce a BGP route that diverts the offending traffic to a mitigation center where we apply metrics and determine if the traffic is an attack. If it is, we separate that traffic from the legit, if there is any, and forward the legit on to the client.

We provide a full graphical log of packets, destination, sources, and an audit trail for any law enforcement requirements the company has. Pretty cool stuff.

cool