UDP packet errors on a 6509?

[polm]

I am having some UDP issues.

I am running 2 Geotel (Cisco Call Routing) servers. Each connected to a seperate Cat 6509.

Each Cat 6509 has double redundat fiber trunk connections, using STP, but etherchanneling the Fiber ports as well.

The Geotel servers typically require a NIC1 --> SWITCH, and a NIC2 --> Cross-Over to other Geotel Server.

They operate the call routing service on NIC1 . They maintain a UDP heartbeat (100ms) across NIC2.

Up until last week my 2 Geotel servers ran both the call routing service and the heartbeat across NIC1, with NIC2 actually disabled. (BTW, OS is NT 4.0)

I noticed that the #2 Geotel server was beginning to reboot frequently, with no indication of why coming from NT. The Geotel service seemed to be rebooting the box due to a failure to recieve the heartbeat from NIC1 of #1 Geotel. This results in a software shutdown, and then the server is rebooted (don't ask me why Cisco does this, they just do)

If I conect both Geotel servers to the same switch , the heartbeat is maintained and the servers run fine.

The servers must remain connected to seperate switches for redundancy standards.

So far I have replaced all CAT5 between the servers and the switch. Replaced all the NICS. Updated drivers. Reinstalled OS. Reinstalled Application.

I am feeling very confident in the servers, and think my problem lies somewhere between Switch 1 and Switch 2.

The switches are Catalayst 6509s with MSFC2 modules.

The switches are set at logging level 7, and have not generated any frame errors.

Can I investigate UDP traffic without running "debug" commands ?

The MSFC2's are showing some UDP packet errors, but only via SNMP. I'm not sure how to investigate them any deeper.

 

[spidey07]

quickly read over your post.

So.....let's move up the stack.

You're sure of layer 1 (seems you already covered it)
You're sure of layer 2 (you checked packet and framing errors, etc)
Layer 3 seems fine.
We're at layer 4.

But I'm gonna back down to layer 2.
The link between the switches. Keep in mind that they each maintain their own CAM table and there could be something funky going on, especially in a failover scenario...and even more so when spanning-tree is involved.

Keep a look out for any spanning-tree events (the heartbeat of 100ms would mean any slight hiccup at all would trigger the failover). you can check with "show spantree statistics port vlan" to see exactly what and when any event is happening.

Because quite frankly I think you've described your problem "box reboots service because not receiving" which would point me to a layer1/2 problem as long as they are on the same network and don't involve routing.

 

[Slvrtg277]

Are the two servers in the same vlan, or are you routing between two separate vlan's? Being that it works when both are plugged into the same switch, I'd check that the heartbeat is reaching the second switch first. If you are routing the information from one msfc to the other, and the heartbeat is a broadcast, that could be your problem.

If you wanted to do a debug on the UDP, the best way I can recommend is for you to create an acl specifying the exact port number, then debug the acl. That way, you will only get the information that you need and you won't have to worry so much about overloading the proc.