Replacing Router - same IP different MAC, no dice. ARP to blame?

[Sauro]

Setup: Replacing 5520 ASA with another 5520 ASA that has been updated to the last firmware, RAM added, same config on both (same config expect for the converted NAT rules since Cisco changed their NAT format). Powered up new ASA next to the old ASA, waited until it was completely booted and swapped jumps over, 1:1. Outside interface peers with a network which I do not have access to. I should also mention that their are hosts NAT'd across the outside interface, so I am also assuming that the ASA is properly responding as a proxy ARP.

Traffic would not flow into, or out of the router on the outside interface - inside worked fine. One idea as to why this did not go as planned is the ARP table on our peer. I am assuming, concerning ARP, that if an entry exists for a L2 host, but the L2 host is not reachable, it will re-ARP to get the new hardware address. Am I mistaken in assuming this? If so, how else, besides being in contact with the peer's admins and getting them to flush the table, can I get their side to reflect the L2 change?

If it isn't ARP screwing me up, any ideas what else it may be?

 

[Nothinman]

If it was ARP then it should've cleared up on it's own as soon as the old cache entry expired, which shouldn't be more than a few minutes at the extreme end. Unless the other admin added a static ARP entry, then you'll have to call them to have it changed.

The only other thing I can think of right now is is port security, if the port was set to a max of 1 MAC address you'll need to clear that before the new one can be used.

 

[Sauro]

Thanks for your suggestions Nothinman, here's a quick reply/update:

I checked with an admin for the other network and no static ARP exists. Port-sec is a good idea - I'll have to confirm that (since it's not an access port I wouldn't expect it to be on, but who knows - worth checking). The TTL on cached ARPs on the other side is set to the default of 4 hours.

The question to which I cannot find an answer is what happens to a cached ARP entry when its L2 host no longer exists. Will the gateway continue to forward frames even though the hardware destination no longer exist? Since there is no checking at L2 (I don't think there is any, maybe I'm mistaken?) will the frames just get dumped once the MAC table entry for the old ASA expires? Would it re-ARP after this, or would the L3 protocol not care since the ARP entry is still cached?

The next time I attempt to bring the new ASA up I'm either going to have the other admins flush the ARP table or just copy the MAC from the old ASA to the new one, but in the meantime I would certainly like to figure this out.

 

[drebo]

It is likely an ARP issue, yes. Typical ARP timeout is 4 hours.

If you can, coordinate with whoever manages the other network to clear ARP when you make your switch. Or, simply power cycle the equipment you connect to.

The reason I say this is an ARP issue is because an ASA (and, incidentally, a PIX) does not advertise its secondary IP addresses over ARP. When you plug it in, it'll only make known its actual interface IP address, as it needs to find the default gateway on the outside network. The router, already having a MAC address in its forwarding table will not requery ARP for the secondary IP addresses (such as 1:1 NATs or PATs if not using the interface IP). Clearing ARP on the upstream router is the only way to resolve that.

 

[drebo]

The question to which I cannot find an answer is what happens to a cached ARP entry when its L2 host no longer exists. Will the gateway continue to forward frames even though the hardware destination no longer exist? Since there is no checking at L2 (I don't think there is any, maybe I'm mistaken?) will the frames just get dumped once the MAC table entry for the old ASA expires? Would it re-ARP after this, or would the L3 protocol not care since the ARP entry is still cached?

Their router will continue to forward frames to the MAC it has cached until it expires. Once it expires, it will requery ARP for that IP address when it needs to forward traffic to it.

When you plug in a device and it needs to send traffic, for instance to a default gateway, it'll query ARP for that default gateway's address. The act of doing that will cause the router to update its MAC table for the IP address of your ASA. The ASA, however, only does this for the actual interface IP assigned to the ASA's outside interface. It'll listen for and respond to ARP requests for other IP addresses configured on the outside interface, but it will not actively make those addresses known.

 

[Sauro]

drebo -Thanks a bunch for the response. It is quite clarifying, but I do have one more question.

In response to this statement:

Their router will continue to forward frames to the MAC it has cached until it expires. Once it expires, it will requery ARP for that IP address when it needs to forward traffic to it.

Are you referring to the ARP timeout or MAC table timeout. If you are indeed talking about the MAC table, can you confirm that there is no L2 checking for a dead host and it will continue to forward frames regardless of whether the host is up or down. Thanks a bunch.

 

[spidey07]

That's likely ARP as drebo described in detail - 4 hour timeout. The ASA is supposed to send gratuitous ARPs when the interface comes up to update ARP tables, but a lot of time spanning tree isn't done and it's missed (use portfast). I don't know if it sends gratuitious ARPs by default or not.

A trick you can do for any kind of ARP issue for this is ping the router from the new MAC. That will update the router's ARP entry for that IP without having to get into it and clear the arp entry or entire cache.

 

[Sauro]

That's likely ARP as drebo described in detail - 4 hour timeout. The ASA is supposed to send gratuitous ARPs when the interface comes up to update ARP tables, but a lot of time spanning tree isn't done and it's missed (use portfast). I don't know if it sends gratuitious ARPs by default or not.

A trick you can do for any kind of ARP issue for this is ping the router from the new MAC. That will update the router's ARP entry for that IP without having to get into it and clear the arp entry or entire cache.

That makes sense, but, as drebo brought to my attention, I would still be SOL as far as the ASA answering proxy ARP requests.

 

[spidey07]

That makes sense, but, as drebo brought to my attention, I would still be SOL as far as the ASA answering proxy ARP requests.

There is no such thing as a proxy arp request. Only how how l3 devices respond to l2 arps. Proxy arp is the devil and why you always turn it off.

 

[Sauro]

I was using the term loosely as for the ASA responding for the devices behind its outside int.