vlan loop from a voip phone

[xSauronx]

got this from my boss this morning and thought I'd pass it on. copying his email straight from my inbox so...dont bitch at me

I thought this warranted a forward to the group!

Jarred and I spent the better part of a day dealing with what appeared to be DHCP server issues. The results were sporadic, but basically random PCs from a particular subnet/VLAN were getting IPs for a subnet that they should not have been part of. We had machines that were on an untagged VLAN 413 port (10.40.13.x) and they would occasionally receive IP addresses for VLAN 405 (10.40.5.x). The stranger pieces to this is that sometimes these machines would function fine on the “wrong” network and sometimes they would not?? Also, occasionally, we couldn’t get an IP address from the DHCP server? We rebooted DHCP the server several times, checking for any recently installed updates, issues with the NIC configuration (priority/vlan tagging), etc…

We had a similar issue about a year ago and had attributed it to a bug in the core switch software. So, because of that, the core was rebooted on Tuesday, problem showed back up on Wednesday, upgraded core to latest IOS (taking down the mill again), rebooted…same problem.

When we started performing some mirroring of the DHCP server port, we discovered multiple DHCP requests with the same transaction ID, seemed really odd..

*** im not bothering to upload and insert the pic. ***


After a few hours of various troubleshooting with HP, they suggested that we could have a loop between VLANS. We confirmed this by having a machine on 10.40.13.x and one on 10.40.5.x and then performing a ping from 10.40.5.x to 10.40.13.255 and we could see the broadcast ping between the two networks, which we shouldn’t have been able to. So this put us on a goose chase to determine where a loop could possibly be. Jarred discovered some unusual MAC addresses associated with one of the core ports that was on VLAN 405, I reviewed the port and had no documentation for this particular port on our topology maps, so that put us on the trail. We finally narrowed the switch port connectivity down to the sales conference room in the Admin building. You’ll notice by the attached picture that someone was nice enough to plug two wall ports into the same phone. It just so happens that they were on different VLANS J, thus creating a loop within the phone. We believe that all of the suspect traffic had been forwarding through this phone…so, the little Cisco phone was supporting about 50 users! Once we disconnected the 2nd cable, all of the machines that were incorrectly assigned 10.40.5.x addresses lost network connectivity J….they had to be restarted (release/renew) to gain their proper addressing (10.40.13.x). It was refreshing to finally figure this out, however it was quite frustrating for something so simple to wreak such havoc. Beware of the randomly connected Cisco phone!

taking down the switch for an OS upgrade seemed a bit odd to me, but i still thought it was interesting.

 

[drebo]

So....uhm......not to be an ass, but:

STP should have caught this and shut down one of the two ports, sending an SNMP trap or syslog which should have notified your team immediately which port was at fault.

Run a better network next time?

 

[theevilsharpie]

STP only protects against broadcast storms caused by network loops. The network was not looped in this case (at least from the point of view of the switch), so STP would not have done anything for him.

 

[drebo]

No, STP protects against loops forming, period. The fact that they were not the same VLAN is irrelevant. Especially on an HP switch, where your options are either CST or MST, both of which use VLAN 1 for transmission of BPDUs.

STP listens for BPDUs before bringing a port up. It would have seen BPDUs coming from itself, or another switch it already knew about, and not advanced the new port into listening state. That's how STP works. In fact, STP was designed specifically to prevent mistakes like the OP's boss'.

 

[mcmilljb]

Shouldn't have DHCP Snooping realized that it was getting DHCP server response messages from an untrusted port? I can see the request messages getting through to the wrong DHCP server. But once the server responds, I would have thought once it crossed the phone it would have triggered it on the switch.

Edit - Checked. HP has DHCP Snooping.

 

[mcmilljb]

So....uhm......not to be an ass, but:

STP should have caught this and shut down one of the two ports, sending an SNMP trap or syslog which should have notified your team immediately which port was at fault.

Run a better network next time?

Would the phone pass on the BPDU's though?

 

[mcmilljb]

Looks like the OP isn't the first to encounter this.

https://supportforums.cisco.com/thread/144774

I would love to sit in with TAC engineers for a day just to see the stuff they get.

 

[imagoon]

Would the phone pass on the BPDU's though?

They did in our environment. Granted my vlans assignments were different but looping a phone like that resulted in vlan100 looping and going in to BPDU guard. [Native vlan 100, Voice 3]

edit:

Oh when it happened we knew quickly because the BPDU guard sent an alert, and when the port went to blocking the stupid phone would reboot over and over again because it couldn't contact the PBX causing a stream of alerts every 12 seconds.

 

[drebo]

Would the phone pass on the BPDU's though?

Of course. The phone is just a 3-port switch.

That said, it probably depends on the model of phone. Some phones probably filter them.

 

[spidey07]

You bridged the vlans somehow (via the phone). This normally happens when you have native vlan mismatches on trunk links. The native vlan on one side bridges to a non-native on the other and vise versa (which is what happened here). Happens all the time.

The reason spanning tree didn't do anything is there were separate VLANs so it wasn't technically a bridge loop. This is also why BPDUguard should be run on all ports that aren't interconnects.

This isn't a bug or anything like that, it's doing exactly what it's supposed to.

 

[xSauronx]

yeah so i emailed my boss a bit...theyre only running even STP on trunk ports and apparently havent changed anything to keep this from occurring again...at least, he didnt tell me anything they did when i asked, so im assuming nothing happened.

awesome.