how can traffic bleed across ports on a switch?

[TechBoyJK]

I'm working on an issue that I'd like to understand.

Client A has a CentOS server connected to a Cisco 2950, which then connects to our core routers.

Client B has a server farm connected to a Cisco 2950, which then connects to the same Cisco 2950 that Client A connects to.



WAN > CORE ROUTERS > Cisco 2950 > Clients A/B

The problem we are facing is that Client A's CentOS server is seeing all kinds of http traffic come across its public interface that should be going to Client B. It's traffic destined for IP's that aren't even in his subnet. His server is just dropping the traffic, but he's concerned about it even getting to his interface.

We've checked all the arp tables and everything looks good. IP's are configured correctly. Neither client is complaining about lost traffic or ping loss.

Any ideas? I'd like to have a better understanding about this before I get my engineers involved...

 

[spidey07]

I'm willing to tell you exactly what your problem is. But it will cost you.

This is a common misconfiguration problem with spanning-tree.

Hint, you're dealing with a L2/flooding issue.

 

[TechBoyJK]

we've seen similar, more drastic cases that were due to DOS attacks where a 400Mbps dos attack comes in and since the network may only be on a 100Mbps port, traffic bleeds across the other ports, affecting other customers on the switch.

to my understanding switches only broadcast layer2 traffic, which isn't IP based. why would a layer 2 issue cause IP traffic to go to the wrong ports? Could it be due to the traffic being jacked up and not having a destination on the switch, therefore it has nowhere to go, so the switch is checking all ports? aka the traffic is just bouncing around on the network?

to my understanding our switches are setup as basic switches for portability so that if a switch goes down we can just swap cables to another switch. I don't even think spanning tree is configured. I know there is no kind of port security or VLAN configured.

 

[spidey07]

Originally posted by: TechBoyJK

to my understanding our switches are setup as basic switches for portability so that if a switch goes down we can just swap cables to another switch. I don't even think spanning tree is configured. I know there is no kind of port security or VLAN configured.

That's what I figured. Common problem. You're flooding, this is a function of spanning-tree. It's doing what you told it to do. (ie, a default config).

You have a L2 problem.

 

[TechBoyJK]

Originally posted by: spidey07

Originally posted by: TechBoyJK

to my understanding our switches are setup as basic switches for portability so that if a switch goes down we can just swap cables to another switch. I don't even think spanning tree is configured. I know there is no kind of port security or VLAN configured.

That's what I figured. Common problem. You're flooding, this is a function of spanning-tree. It's doing what you told it to do. (ie, a default config).

You have a L2 problem.

Client B has a dell switch connected to the Cisco Switch. Switch is running in default config. There are multilple MAC addresses associated with their port on the Cisco, but there are only 45 total Mac addresses in the table (24 port switch).

 

[spidey07]

Originally posted by: TechBoyJK
Client B has a dell switch connected to the Cisco Switch. Switch is running in default config. There are multilple MAC addresses associated with their port on the Cisco, but there are only 45 total Mac addresses in the table (24 port switch).

Yep, common problem. You're flooding. This is a function of spanning tree, it's working as designed.

I don't want to give you the answer because I could literally charge you a days work for this. If your techs don't know about this oh so common behavior then maybe you need to get me in for a few days.

-edit-
For security guys, this is why VLANs are not a security measure. It's easy to get a switch to flood.

 

[TechBoyJK]

Originally posted by: spidey07

Originally posted by: TechBoyJK
Client B has a dell switch connected to the Cisco Switch. Switch is running in default config. There are multilple MAC addresses associated with their port on the Cisco, but there are only 45 total Mac addresses in the table (24 port switch).

Yep, common problem. You're flooding. This is a function of spanning tree, it's working as designed.

I don't want to give you the answer because I could literally charge you a days work for this. If your techs don't know about this oh so common behavior then maybe you need to get me in for a few days.

-edit-
For security guys, this is why VLANs are not a security measure. It's easy to get a switch to flood.

well im a tech and I didnt realize all that. im trying to learn here though. thanks for your replies. most of our senior level techs werent in the office, and im just doing my job and trying to crack a case as much as possible before they get back.

 

[spidey07]

here's a hint....look up what happens when a topology change notification bpdu is sent (TCN).

 

[subflava]

to my understanding our switches are setup as basic switches for portability so that if a switch goes down we can just swap cables to another switch. I don't even think spanning tree is configured. I know there is no kind of port security or VLAN configured.

I think it's really bad practice to not separate client traffic from each other. If I was one of your clients, I would not have a lot of confidence in your security polices and procedures in other areas.

Also, by default spanning-tree is running on Cisco (and every other vendor I've come across) switches.

 

[subflava]

You're flooding, this is a function of spanning-tree.

Spidey - I don't completely understand your comment about flooding being specific to spanning-tree. Can you elaborate? Flooding is a normal function of a switch when it doesn't have a MAC address in it's table...I'm not seeing the specific connection to ST. Are you just saying because of the (likely) frequent TCNs in his setup, his table aging times are always/frequently low and therefore traffic frequently floods?

 

[nightowl]

When a TCN is sent out, it ages the CAM table more quickly to adjust for the change and therefore traffic is flooded until the mac-address are learned again by the switch.

http://www.cisco.com/en/US/tec...09186a0080094797.shtml

 

[spidey07]

Originally posted by: subflava

You're flooding, this is a function of spanning-tree.

Spidey - I don't completely understand your comment about flooding being specific to spanning-tree. Can you elaborate? Flooding is a normal function of a switch when it doesn't have a MAC address in it's table...I'm not seeing the specific connection to ST. Are you just saying because of the (likely) frequent TCNs in his setup, his table aging times are always/frequently low and therefore traffic frequently floods?

That's exactly what I'm saying. It happens all the time if you don't setup spanning-tree properly.

 

[jlazzaro]

Originally posted by: spidey07

Originally posted by: subflava

You're flooding, this is a function of spanning-tree.

Spidey - I don't completely understand your comment about flooding being specific to spanning-tree. Can you elaborate? Flooding is a normal function of a switch when it doesn't have a MAC address in it's table...I'm not seeing the specific connection to ST. Are you just saying because of the (likely) frequent TCNs in his setup, his table aging times are always/frequently low and therefore traffic frequently floods?

That's exactly what I'm saying. It happens all the time if you don't setup spanning-tree properly.

too many people take for granted that STP is enabled by default...everything should be a-ok right? a lot goes into adjusting default priorities, costs, roots, primaries, secondaries, etc so everything runs EFFICIENTLY as opposed to barely running at all.

 

[spidey07]

Originally posted by: jlazzaro
too many people take for granted that STP is enabled by default...everything should be a-ok right? a lot goes into adjusting default priorities, costs, roots, primaries, secondaries, etc so everything runs EFFICIENTLY as opposed to barely running at all.

One good thing about being a consultant, I see how things can be done REALLY badly.

Even huge shops don't pay attention to spanning-tree and assume that a complete network meltdown every few months is acceptable.

 

[subflava]

When a TCN is sent out, it ages the CAM table more quickly to adjust for the change and therefore traffic is flooded until the mac-address are learned again by the switch.

I know about this behavior of spanning-tree. I was just asking for clarification because it sounded like spidey *could* have been saying there was something specific with spanning-tree in which the protocol itself specified that traffic be flooded. I was not aware of any such behavior which is why I asked about it. For example, you could achieve the same effect without spanning-tree if you turned off spanning-tree completely, then manually changed the mac table aging to 15 seconds.

I just wanted to be clear about what Spidey was saying. I suspected this was what he was saying and he was just using shorthand/shortcut to get his point across without going into the details, but I wanted to be sure. Anyways, he's responded to my question so now I'm clear that we're talking/thinking about the same thing.

 

[spyordie007]

Originally posted by: spidey07

Originally posted by: jlazzaro
too many people take for granted that STP is enabled by default...everything should be a-ok right? a lot goes into adjusting default priorities, costs, roots, primaries, secondaries, etc so everything runs EFFICIENTLY as opposed to barely running at all.

One good thing about being a consultant, I see how things can be done REALLY badly.

Even huge shops don't pay attention to spanning-tree and assume that a complete network meltdown every few months is acceptable.

I can second that, one of our large healthcare customers had this very issue several months back...

 

[TechBoyJK]

yea our main engineer brought up the concept about CAM and spanning tree and attributed it to the dell behind the cisco messing up the arp tables.. or something like that.