Cisco VPN concentrator 3005 drops connection

[Cooky]

We have a VPN Concentrator 3005 for employees to connect from home or off sites. Recently some users have reported dropped VPN connections, even though their Internet connection is ok. Below is part of th log, could someone please tell me what needs to be done to fix it?

26872 11/29/2005 11:39:04.200 SEV=4 IKE/120 RPT=2211 24.3.x.x
Group [groupname] User [username]
PHASE 2 COMPLETED (msgid=637ef260)

!At this point it appears that user was able to connect fine.

26873 11/29/2005 11:39:06.650 SEV=5 IKE/50 RPT=1672 24.3.x.x
Group [groupname] User [username]
Connection terminated for peer username.
Reason: Peer Terminate
Remote Proxy 172.16.x.x, Local Proxy 0.0.0.0

26876 11/29/2005 11:39:06.650 SEV=5 IKE/194 RPT=1885 24.3.x.x
Group [groupname] User [username]
Sending IKE Delete With Reason message: No Reason Provided.

26878 11/29/2005 11:39:06.660 SEV=4 AUTH/28 RPT=1725 24.3.x.x
User [username] Group [groupname] disconnected:
Session Type: IPSec/NAT-T
Duration: 0:00:02
Bytes xmt: 0
Bytes rcv: 0
Reason: User Requested
!What does this mean?? User said she didn't disconnect anything.

 

[Cooky]

Anyone?

 

[spidey07]

Well it sure looks like the client sent the disconnect request.

Key question - did anything change in anyway? software? Is it repeatable and does it happen often? User have firewall or NAT router?

I'd call cisco and open a case.

 

[Cooky]

The only change made within the past 6 months was an additional entry for RADIUS authentication. It shouldn't be used unless the preceeding authentication types ran into error (server down, etc)
Most users are ok and it only happens to some of them. There's no pattern and I haven't been able to reproduce the error.

Yes users have NAT routers but the problem only occurs to them occationally, not all the time so I don't think NAT is an issue here.
They reported that the connection would stay for about 3-5 min, then it just drops. They all have high speed Internet like DSL or cable modem.

There's really nothing I can find on this problem on Cisco or Google. (well, some guy posted the same problem in a forum but nobody has replied)

What sucks more is our SmartNet expired and management wouldn't approve the renewal proposal so I'm stuck...>-<

 

[spidey07]

Originally posted by: Cooky
The only change made within the past 6 months was an additional entry for RADIUS authentication. It shouldn't be used unless the preceeding authentication types ran into error (server down, etc)
Most users are ok and it only happens to some of them. There's no pattern and I haven't been able to reproduce the error.

Yes users have NAT routers but the problem only occurs to them occationally, not all the time so I don't think NAT is an issue here.
They reported that the connection would stay for about 3-5 min, then it just drops. They all have high speed Internet like DSL or cable modem.

There's really nothing I can find on this problem on Cisco or Google. (well, some guy posted the same problem in a forum but nobody has replied)

What sucks more is our SmartNet expired and management wouldn't approve the renewal proposal so I'm stuck...>-<

Well then its not your problem anymore.

Never, ever, ever run gear without a current maintenance contract. Its just part of doing business. Because I can guarantee they would be able to fix it.

You could just let cisco get into your concentrator and say "fix it"

But I digress. Maybe you can open a case for a fee?

 

[Zstream]

Originally posted by: Cooky
The only change made within the past 6 months was an additional entry for RADIUS authentication. It shouldn't be used unless the preceeding authentication types ran into error (server down, etc)
Most users are ok and it only happens to some of them. There's no pattern and I haven't been able to reproduce the error.

Yes users have NAT routers but the problem only occurs to them occationally, not all the time so I don't think NAT is an issue here.
They reported that the connection would stay for about 3-5 min, then it just drops. They all have high speed Internet like DSL or cable modem.

There's really nothing I can find on this problem on Cisco or Google. (well, some guy posted the same problem in a forum but nobody has replied)

What sucks more is our SmartNet expired and management wouldn't approve the renewal proposal so I'm stuck...>-<

It does not look like the client is disconnecting but rather once the connection between the client and server get established your server is booting it off because of packets not getting through. As you probably already know it will be hard without looking at the packets sent.

 

[bgroff]

Originally posted by: spidey07

Originally posted by: Cooky
The only change made within the past 6 months was an additional entry for RADIUS authentication. It shouldn't be used unless the preceeding authentication types ran into error (server down, etc)
Most users are ok and it only happens to some of them. There's no pattern and I haven't been able to reproduce the error.

Yes users have NAT routers but the problem only occurs to them occationally, not all the time so I don't think NAT is an issue here.
They reported that the connection would stay for about 3-5 min, then it just drops. They all have high speed Internet like DSL or cable modem.

There's really nothing I can find on this problem on Cisco or Google. (well, some guy posted the same problem in a forum but nobody has replied)

What sucks more is our SmartNet expired and management wouldn't approve the renewal proposal so I'm stuck...>-<

Well then its not your problem anymore.

Never, ever, ever run gear without a current maintenance contract. Its just part of doing business. Because I can guarantee they would be able to fix it.

You could just let cisco get into your concentrator and say "fix it"

But I digress. Maybe you can open a case for a fee?

Absolutely, amen. Go renew your smartnet. That's why you buy Cisco in the first place. For the support! The 8x5xNBD isn't that expensive, especially on a 3005.

 

[Cooky]

Thanks for the replies guys.

We've tried to get it but failed. What should I tell the management if they ask why I can't figure it out?? They'd say the reason they hired me is because I'm supposed to fix these problems.

 

[spidey07]

Originally posted by: Cooky
Thanks for the replies guys.

We've tried to get it but failed. What should I tell the management if they ask why I can't figure it out?? They'd say the reason they hired me is because I'm supposed to fix these problems.

"This requires the support of the manufacturer. It looks like a software bug."

Wish I could help, but its been ages since I troubleshooted IPsec. Maybe throw a sniffer on the outside interface and see what you can see. Punt? Reboot the box.

On a side note there will ALWAYS be things you can't figure out, no matter who you are. Even with the great resources of the intarweb.

a few things to chew on...
http://www.cisco.com/en/US/tech/tk583/t...logies_tech_note09186a0080093f87.shtml
http://www.cisco.com/en/US/products/hw/...oducts_tech_note09186a0080094eca.shtml

 

[Zstream]

Originally posted by: Cooky
Thanks for the replies guys.

We've tried to get it but failed. What should I tell the management if they ask why I can't figure it out?? They'd say the reason they hired me is because I'm supposed to fix these problems.

Hey can you show some of the packets being sent through the system? I would believe you are correct about the smartnet, that is why you were hired.

 

[spidey07]

how big is your address pool for the clients? Any way you could be running out. I can see that causing the sporadic nature.

 

[nightowl]

Does the disconnect always come with in 3-5 minutes? It could very well be a problem with the end user's ISP or NAT device. My VPN connection to a Cisco concentrator has been solid for the most part at my apartment with Time Warner Cable now that I have a Cisco AP for wireless. Before that, I would drop the connection from time to time over wireless with my Linksys wireless router. I have seen a couple disconnects with using the Cisco AP but no where near as many as before. I have seen this on SBC and other DSL providers too. So, since you said it was just a few users I would not necessarly say the concentrator is the oly cause of the problem. The times when I have been disconnected my Internet connection appeared to be fine too.

 

[Cooky]

I'll redirect traffic from the VPN switch port to another one for a SPAN session and then see if I can capture anything tomrrow.
I looked in the concentrator and didn't see any place where you can capture packets.

The pool had 128 IP's, which was later reduced to only 32 because at any moment we don't see more than 10 people connecting.

 

[spidey07]

Originally posted by: Cooky
I'll redirect traffic from the VPN switch port to another one for a SPAN session and then see if I can capture anything tomrrow.
I looked in the concentrator and didn't see any place where you can capture packets.

The pool had 128 IP's, which was later reduced to only 32 because at any moment we don't see more than 10 people connecting.

Also read the links I posted. Tons of good information, especially logging on the client end.

 

[Cooky]

Just when I've got the SPAN session ports all set up none of the users are having any problem today...which is a good thing I guess.
This has happened before: for a while it'll work fine, but then one day people get dropped connections for no apparent reason.

The 2 links that Spidey posted are very useful indeed. Thank you.
I'll have the end users send me logs next time they experience any issues.

We did suspect ISP or end user equipment but this happened to multiple users which is why we weren't sure. Yes it may have been the case.

Thanks for everyone's help!! Really appreciate it.