Site-to-Site VPN traffic monitoring question (Cisco ASA)

[seepy83]

I've got a site-to-site VPN between a couple of Cisco ASAs. It's been up for a very long time (years), but just in the last couple of days I was looking at utilization on the firewall's interfaces and i'm seeing something I can't explain.

There is a constant 10 Mbps of traffic between the two ASAs (which is, historically,not normal in this environment), but I only see that volume of traffic on the Outside interfaces of each ASA. The source/destination for the traffic is the public IP for the other ASA. But there is no traffic on the Inside/LAN interfaces of either ASA.

I can pull a packet capture off the outside/WAN interface, but it's all VPN tunnel traffic so it's encrypted and I can't see what's actually going on.

If I saw a comparable volume of traffic on the Inside/LAN interfaces, I wouldn't think twice about it. But since it's definitely VPN tunnel traffic, and it's reaching the other ASA and then not going anyway, I want to know what's going on.

Any suggestions?

 

[seepy83]

I ended up bouncing the outside interface on one of the ASAs and the traffic/utilization went back to normal rates.

I ended up asking one of the engineers at our ISP to look into it. Both sites are on Opt-E-MAN connections, and we've been waiting for some changes to bandwidth allocation to get completed. I thought maybe they made some changes, or partial changes in preparation for a cut-over, and it caused the problem. They say they didn't change anything, but I'm not entirely convinced.

Oh well...at least it's "fixed" now.

 

[Martin Wilson]

Interesting. Sounds like it might have been a bug?

 

[seepy83]

To be honest, I think the engineer at our ISP isn't giving me the whole story. It's too much of a coincidence that it has been up and running error free for over 3 years, and all of a sudden we have a problem after we've made a request for a service change but before that service change is completed. I haven't seen a "bug" like this appear out of nowhere in networking gear...it's more reminiscent of a design/config problem.

I don't have the time or expertise to try to figure out how the ISP may have made a config change upstream from our firewall that would cause something like a routing loop to occur and go away when the connection is dropped and the VPN tunnel needs to be re-negotiated.

I'd love to hear thoughts on this from anyone here gets to spend most of their time at doing strictly network admin/engineering.

 

[SecurityTheatre]

Meh, 3 years is pretty good. I see corporate-grade router/firewall devices simply hang or crash on about that basis.

Had to restart the dataplane a new firewall twice last week at two different customers that had run just fine for a year.

Then again, it could be someone messing with the config. It's pretty easy to bork VPNs. Some designs revalidate or cycle certificates on a regular (daily) basis and some very subtle problems can make them fail at odd times (like 20 hours after a change) due to this behavior.

Some modes of failure can result in loops and/or broadcast storms. But without knowing WHAT that traffic was, it's hard to say beyond that.