MAC flapping issue between HP and Cisco

[dphantom]

Not a network guy but a server admin.

Ever since we put in a Cisco network, 6509s, Nexus 5ks in the data center and 3750s in the closets, we have been experiencing MAC flapping with our HP servers including blade chassis servers. MAC flapping is intermitent and ony occurs when the NICs are load balanced. When NICs are configured as network fault tolerant, the problem goes away.

Our network folk cannot find any problems. Our previous Nortel network did not exhibit any MAC flapping when NICs were load balanced.

I am hoping someone here can provide some ideas on things to look at regarding why this occurs. We do have TAC but in our environment, our network people want us to identify the problem before they will resolve it or open a TAC case.

So any ideas including WAGs are welcome.

 

[spidey07]

How are the servers connected to the switches? If different switches then you MUST use fail on fault, only one NIC (and it's associated MAC) can be active at a time. Otherwise the switch will see the mac on different ports and that's a huge no-no.

If they are connected to the same switch then you can use etherchannel/LACP to form a single logical channel across multiple ports and do load balancing.

If your connected to the nexus 5ks then you can use vPC/Etherchannel/LACP and load balance either to same chassis or different one. Same if you're using VSS on the 6500s.

Honestly though, if your networking people don't know this then they really should open a TAC case. This is basic stuff.

-edit-
A MAC address can only live on a single port. A channel/link aggregation/vitual port-channel is still a single logical port. This is all part of how switching/bridging works. When you use multiple NICs they still have the same MAC address when using NIC teaming features. So you have to make sure the network and NIC teaming software are configured to agree with each other. So you have some sort of configuration problem, you cannot use load balancing on the NIC team without using channeling features on the switch, otherwise you WILL see this exact thing - MAC flapping.

MAC flapping is when a source MAC address is seen on different ports. That could be an indication of a serious problem (like a loop) and the switch is letting you know about it. It can also cause excessive flooding and performance problems because the switch will be re-learning the MAC constantly every single time a frame is seen on a different port. What I'm trying to say is you NEED to fix this.

 

[freegeeks]

Not a network guy but a server admin.

Ever since we put in a Cisco network, 6509s, Nexus 5ks in the data center and 3750s in the closets, we have been experiencing MAC flapping with our HP servers including blade chassis servers. MAC flapping is intermitent and ony occurs when the NICs are load balanced. When NICs are configured as network fault tolerant, the problem goes away.

Our network folk cannot find any problems. Our previous Nortel network did not exhibit any MAC flapping when NICs were load balanced.

I am hoping someone here can provide some ideas on things to look at regarding why this occurs. We do have TAC but in our environment, our network people want us to identify the problem before they will resolve it or open a TAC case.

So any ideas including WAGs are welcome.

are you running Linux on the HP server? Have you correctly configured LACP (including kernel modules?). You need to configure bond mode=4.
Have you correctly configured the port channel on the cisco?

If you run windows, disregard my post

 

[dphantom]

How are the servers connected to the switches? If different switches then you MUST use fail on fault, only one NIC (and it's associated MAC) can be active at a time. Otherwise the switch will see the mac on different ports and that's a huge no-no.

If they are connected to the same switch then you can use etherchannel/LACP to form a single logical channel across multiple ports and do load balancing.

If your connected to the nexus 5ks then you can use vPC/Etherchannel/LACP and load balance either to same chassis or different one. Same if you're using VSS on the 6500s.

Honestly though, if your networking people don't know this then they really should open a TAC case. This is basic stuff.

-edit-
A MAC address can only live on a single port. A channel/link aggregation/vitual port-channel is still a single logical port. This is all part of how switching/bridging works. When you use multiple NICs they still have the same MAC address when using NIC teaming features. So you have to make sure the network and NIC teaming software are configured to agree with each other. So you have some sort of configuration problem, you cannot use load balancing on the NIC team without using channeling features on the switch, otherwise you WILL see this exact thing - MAC flapping.

MAC flapping is when a source MAC address is seen on different ports. That could be an indication of a serious problem (like a loop) and the switch is letting you know about it. It can also cause excessive flooding and performance problems because the switch will be re-learning the MAC constantly every single time a frame is seen on a different port. What I'm trying to say is you NEED to fix this.

Perfect Spidey. We will get with our network guys and see if they actually created any vPCs. We don't think they have.

 

[Cooky]

Is this a wide spread problem that's occurring to many servers, or only a few of them?
I've seen two of our HP servers somehow ended up w/ the same virtual MAC when doing NIC teaming, and caused MAC flapping between them.

If it's wide spread, then look into the config between servers & switches per spidey's suggestion.
We do vPC on our N5K's and etherchannel's w/ HP servers w/o any issues.

 

[spidey07]

Is this a wide spread problem that's occurring to many servers, or only a few of them?
I've seen two of our HP servers somehow ended up w/ the same virtual MAC when doing NIC teaming, and caused MAC flapping between them.

If it's wide spread, then look into the config between servers & switches per spidey's suggestion.
We do vPC on our N5K's and etherchannel's w/ HP servers w/o any issues.

Good point. I have seen HP issue the same MAC on different servers/NICs. But given it goes away when properly configuring the NICs to fault tolerance points to a channel/teaming config mismatch.

 

[dphantom]

Good point. I have seen HP issue the same MAC on different servers/NICs. But given it goes away when properly configuring the NICs to fault tolerance points to a channel/teaming config mismatch.

That is a good observation Cooky but my team believes it is a network misconfiguration. Now we just have to convince our network team. It does happen with multiple servers, both pizza boxes and blades. Reconfiguring the server NICs to NFT fixes the problem every time.

 

[spidey07]

That is a good observation Cooky but my team believes it is a network misconfiguration. Now we just have to convince our network team. It does happen with multiple servers, both pizza boxes and blades. Reconfiguring the server NICs to NFT fixes the problem every time.

Then I can just about guarantee the channel/vPC isn't setup on the switch side or it isn't setup properly.