Cisco switch turned off a couple ports and won't come back on

[Brazen]

We have a Cisco Catalyst 3560. I don't think this is a hardware problem.

We have a few servers with two nics plugged in to this. Each pair is aggregated and set as vlan trunks. On one of these servers, we have a pfsense firewall. I bridged two vlans in the firewall and after a few seconds, both ethernet ports went dead. The lights went completely out like nothing is even connected. I thought it odd that both ports would have a failure at the same time, so I rebooted the server and unplugged/replugged the ethernet cables. The ports stayed dark.

I tried plugging in to another set of ports and those came up and worked for a few seconds, and then went dark. I thought maybe this had something to do with bridging the vlans, so I disabled the bridge and plugged in to a third set of ports. This time the ports worked fine.

I don't really need the bridging, but I can't get these 4 dead ports to come back up. Any help? I have not power cycled the switch and prefer not too as I will have to schedule some after-hours downtime.

 

[spidey07]

GAH! Why did you bridge the VLANs/cards? You essentially created a bridging loop. Remember a switch is nothing more than a multiport bridge.

The switch is shutting down all those ports to prevent a loop. The feature is called bpduguard and if you look at the logs in the switch it will show you why it err-disabled them. Get into the switch and "show log" and "show int status err-disabled" commands. Then to get the ports back enter configuration mode, then "int g0/4" or whatever port and then "shut" and then "no shut"

But you need to fix the loop you're creating first otherwise it's just going to shut them down. NEVER bridge network cards, especially if it is going into the same network. Other switches would have just let you do this and the whole network would cease to function in a matter of seconds.

 

[Brazen]

GAH! Why did you bridge the VLANs/cards? You essentially created a bridging loop. Remember a switch is nothing more than a multiport bridge.

The switch is shutting down all those ports to prevent a loop. The feature is called bpduguard and if you look at the logs in the switch it will show you why it err-disabled them. Get into the switch and "show log" and "show int status err-disabled" commands. Then to get the ports back enter configuration mode, then "int g0/4" or whatever port and then "shut" and then "no shut"

But you need to fix the loop you're creating first otherwise it's just going to shut them down. NEVER bridge network cards, especially if it is going into the same network. Other switches would have just let you do this and the whole network would cease to function in a matter of seconds.

I didn't bridge the same network. I bridged two separate vlans. There shouldn't be any other location where the vlans can talk to each other other than through the now-disabled bridge, so no loop was created. Unless I made a mistake in what I bridged, or there is a misconfiguration somewhere else on the vlans, or else the cisco is being retarded.

I'll try your suggestion for re-enabling it, and see what the logs say.

 

[spidey07]

By bridging the VLANs, BPDUs from one vlan were being forwarded by the server to the other and received on a switch port. That will cause the switch to shut down the port via bpduguard. It's normally best practice to use portfast bpduguard default which puts any ports using portfast to also enable bpduguard. I can almost guarantee that is what shut them down as it takes only 2 seconds for the switchport to receive a bpdu.

The switch is doing this on purpose to stop you from causing a problem.

 

[drebo]

You shouldn't need the VLANs to be bridge in order to talk to both of them (especially on the router that's routing between them).

What is it with badly-conceived network layouts lately?

 

[Pheran]

You shouldn't need the VLANs to be bridge in order to talk to both of them (especially on the router that's routing between them).

What is it with badly-conceived network layouts lately?

Drebo, my impression is that he is trying to implement a layer-2 "bump in the wire" firewall, which would require this type of configuration. But I agree that it is not clear from the OP.

 

[Brazen]

Drebo, my impression is that he is trying to implement a layer-2 "bump in the wire" firewall, which would require this type of configuration. But I agree that it is not clear from the OP.

That's right. Although it is not exactly my choice to do it that way and I will agree it is a bit "stupid." Although I wouldn't put it that harshly. I inherited this configuration and I'm trying to replace a 10 year old SonicWall with pfSense. The bridged networks are the internet connection and the DMZ. I would have just plugged back in to the same ports exactly as it was (no aggregation and no vlans) but I prefer to do it this way for the added redundancy and flexibility.

A better design to me, would be to have the DMZ on a private subnet and route between the DMZ and the internet connection, using 1-to-1 NAT to map public ip addresses to the servers (that would put less processing demand on the firewall anyway). I'm guessing maybe the SonicWall used bridging for the DMZ by default, or maybe that is just how the previous sysadmin wanted it. Anyway, I would have to reconfigure a lot of servers to switch from bridging to routing, which is probably what I will do.

I have not tried the "shut" and "no shut" commands mentioned above yet; I haven't been back in to work on it yet, but hopefully that will get my ports back alive. I guess I'll find out tomorrow.

 

[Brazen]

GAH! Why did you bridge the VLANs/cards? You essentially created a bridging loop. Remember a switch is nothing more than a multiport bridge.

The switch is shutting down all those ports to prevent a loop. The feature is called bpduguard and if you look at the logs in the switch it will show you why it err-disabled them. Get into the switch and "show log" and "show int status err-disabled" commands. Then to get the ports back enter configuration mode, then "int g0/4" or whatever port and then "shut" and then "no shut"

But you need to fix the loop you're creating first otherwise it's just going to shut them down. NEVER bridge network cards, especially if it is going into the same network. Other switches would have just let you do this and the whole network would cease to function in a matter of seconds.

by the way, the "show int status err-disabled" says: "channel-misconfig". I'm not sure if that is what you expected from being disabled by the bpduguard?

 

[Brazen]

"shut" and "no shut" didn't work

 

[Pheran]

"shut" and "no shut" didn't work

Shut and no shut aren't going to accomplish a thing if you haven't disabled BPDU guard on those ports (assuming that is in fact the problem here). Channel misconfig sounds more like an etherchannel problem, and you certainly should not have an etherchannel config in this situation. What does "show log" say? Also, the output of "show run interface" on your 2 interfaces may help.

 

[Brazen]

Shut and no shut aren't going to accomplish a thing if you haven't disabled BPDU guard on those ports (assuming that is in fact the problem here).

I haven't disabled BPDU guard (don't know how), but I have shut down the bridge (actually moved the firewall to different physical ports and disabled the bridge). I just want to get these ports back to how they were before.

 

[Brazen]

This is all the output from "show log", May 14th is when I killed the ports by setting up the bridge. I was plugged in to ports 8 and 18, then moved to 5 and 15. The etherchannel and vlans were already set up on these ( I don't know how) and I didn't change any config:

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.



No Inactive Message Discriminator.


    Console logging: disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 278 messages logged, xml disabled,
                     filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    File logging: disabled
    Persistent logging: disabled

No active filter modules.

    Trap logging: level notifications, 279 message lines logged
        Logging to 192.168.2.43  (udp port 514,  audit disabled,
              authentication disabled, encryption disabled, link up),
              279 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled

Log Buffer (4096 bytes):
o up
000243: .May 14 16:46:46.235 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to up
000244: .May 14 16:48:52.080 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to down
000245: .May 14 16:48:54.001 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to down
000246: .May 14 16:48:56.476 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to up
000247: .May 14 16:48:57.692 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to up
000248: .May 14 16:52:09.296 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to down
000249: .May 14 16:52:11.217 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to down
000250: .May 14 16:52:13.650 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to up
000251: .May 14 16:52:14.866 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to up
000252: .May 14 16:55:01.355 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/3, changed state to down
000253: .May 14 16:55:03.351 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/3, changed state to down
000254: .May 14 16:57:53.346 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/3, changed state to up
000255: .May 14 16:57:53.355 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/3, changed state to up
000256: .May 14 17:15:41.383 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/7, changed state to down
000257: .May 14 17:15:41.383 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down
000258: .May 14 17:15:41.383 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel3, changed state to down
000259: .May 14 17:15:43.178 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/7, changed state to down
000260: .May 14 17:15:43.270 CST: %LINK-3-UPDOWN: Interface Port-channel3, changed state to down
000261: .May 14 17:15:43.270 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/17, changed state to down
000262: .May 14 17:18:43.122 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/7, changed state to up
000263: .May 14 17:18:43.164 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/17, changed state to up
000264: .May 14 17:18:44.439 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/7, changed state to up
000265: .May 14 17:18:44.439 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up
000266: .May 14 17:18:46.150 CST: %LINK-3-UPDOWN: Interface Port-channel3, changed state to up
000267: .May 14 17:18:46.159 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel3, changed state to up
000268: May 18 09:08:13.296 CST: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (192.168.0.91)
000269: May 18 17:14:42.532 CST: %LINK-5-CHANGED: Interface GigabitEthernet0/8, changed state to administratively down
000270: May 18 17:14:51.130 CST: %LINK-3-UPDOWN: Interface GigabitEthernet0/8, changed state to down
000271: May 18 17:14:51.499 CST: %PM-4-ERR_DISABLE: channel-misconfig error detected on Gi0/8, putting Gi0/8 in err-disable state
000272: May 18 17:14:55.752 CST: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (192.168.0.91)
000273: May 20 06:39:02.696 CST: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (192.168.0.91)
000274: May 20 06:41:24.874 CST: %PM-4-ERR_DISABLE: channel-misconfig error detected on Gi0/8, putting Gi0/8 in err-disable state
000275: May 20 06:41:41.903 CST: %PM-4-ERR_DISABLE: channel-misconfig error detected on Gi0/18, putting Gi0/18 in err-disable state
000276: May 20 06:45:26.592 CST: %PM-4-ERR_DISABLE: channel-misconfig error detected on Gi0/5, putting Gi0/5 in err-disable state
000277: May 20 06:45:36.910 CST: %PM-4-ERR_DISABLE: channel-misconfig error detected on Gi0/15, putting Gi0/15 in err-disable state
000278: May 20 06:45:51.917 CST: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (192.168.0.91)

 

[Brazen]

Here is the "show run interface" output for two bonded ports (8 and 18):

#show run interface GigabitEthernet 0/8
Building configuration...

Current configuration : 236 bytes
!
interface GigabitEthernet0/8
 switchport trunk encapsulation dot1q
 switchport mode trunk
 srr-queue bandwidth share 10 10 60 20
 queue-set 2
 priority-queue out
 mls qos trust cos
 auto qos voip trust
 channel-group 4 mode on
end

#show run interface GigabitEthernet 0/18
Building configuration...

Current configuration : 237 bytes
!
interface GigabitEthernet0/18
 switchport trunk encapsulation dot1q
 switchport mode trunk
 srr-queue bandwidth share 10 10 60 20
 queue-set 2
 priority-queue out
 mls qos trust cos
 auto qos voip trust
 channel-group 4 mode on
end

 

[Pheran]

Oh, well no wonder it doesn't work, you are trying to bridge together two ports that are already bonded together in an etherchannel, which is going to go down in flames. I would also question why they are even in trunk mode if you are only dealing with one subnet, but that should still work as long as your firewall is prepared for 802.1q tags.

At minimum, get rid of the etherchannel, that's not going to work at all.

 

[spidey07]

Also need "show run int po 4". Channel-misconfig is an err-disabled condition where the physical ports disagree with the channel configuration or it receives PaGP frames that disagree with the current configuration. You can also use the command "show etherchannel detail" to view the state of the channel.

 

[Pheran]

Oh, well no wonder it doesn't work, you are trying to bridge together two ports that are already bonded together in an etherchannel, which is going to go down in flames. I would also question why they are even in trunk mode if you are only dealing with one subnet, but that should still work as long as your firewall is prepared for 802.1q tags.

At minimum, get rid of the etherchannel, that's not going to work at all.

I just occurred to me that I might be misunderstanding what you are trying to do. I was thinking mostly in terms of an untagged bridge, but it could be completely valid to have an etherchannel in trunk mode if you are doing 1-armed bridging between two different VLANs. Sorry about the misunderstanding if that is your goal - ignore my comments about removing the etherchannel if so.

 

[Brazen]

Sorry it took me a while to get to this.

Also need "show run int po 4". Channel-misconfig is an err-disabled condition where the physical ports disagree with the channel configuration or it receives PaGP frames that disagree with the current configuration. You can also use the command "show etherchannel detail" to view the state of the channel.

Here is the output of the "show run int po 4"

3560_Rack_SW2#show run int po 4
Building configuration...

Current configuration : 92 bytes
!
interface Port-channel4
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

And here is the output of the "show etherchannel detail" (snipped):

Group: 4
----------
Group state = L2
Ports: 2   Maxports = 8
Port-channels: 1 Max Port-channels = 1
Protocol:    -
Minimum Links: 0
                Ports in the group:
                -------------------
Port: Gi0/8
------------

Port state    = Down Not-in-Bndl
Channel group = 4           Mode = On              Gcchange = -
Port-channel  = null        GC   =   -             Pseudo port-channel = Po4
Port index    = 0           Load = 0x00            Protocol =    -

Age of the port in the current state: 10d:22h:16m:21s

Port: Gi0/18
------------

Port state    = Down Not-in-Bndl
Channel group = 4           Mode = On              Gcchange = -
Port-channel  = null        GC   =   -             Pseudo port-channel = Po4
Port index    = 0           Load = 0x00            Protocol =    -

Age of the port in the current state: 10d:22h:16m:50s

                Port-channels in the group:
                ---------------------------

Port-channel: Po4
------------

Age of the Port-channel   = 12d:13h:58m:11s
Logical slot/port   = 2/4          Number of ports = 0
GC                  = 0x00000000      HotStandBy port = null
Port state          = Port-channel Ag-Not-Inuse
Protocol            =    -
Port security       = Disabled

Time since last port bundled:    12d:13h:53m:13s    Gi0/18
Time since last port Un-bundled: 10d:22h:16m:55s    Gi0/18

The only difference in either of these two outputs, that I see, is the output of "show etherchannel detail" has "Down Not-in-Bndl" for the port states while other groups say "Up Mstr In-Bndl" for the port states.

 

[Brazen]

I fixed this.

I should have updated this thread right away so I could have given a better description, but basically I had to enable auto-recovery for Channel-misconfig. After waiting for the default time period (5 minutes I think it was), all the ports came back on.