ICMP Checksum errors

[Crusty]

I'm trying to troubleshoot some network issues with an application and I need help trying to decipher the traceroutes.

First, I ran one locally here in the office to a remote server in NJ, it appears to hit a set of routers that are using 192.168.*.* IP's and then times out from then on. I assume it's some sort of load balancer sitting in front of the servers. Once I saw that, I logged into one of our web servers @ theplanet and ran the same traceroute, only this time I received lots of ICMP checksum errors from some ATT routers.

Can anyone shed some light on these traceroutes?

From my office:

traceroute 207.17.34.246
traceroute to 207.17.34.246 (207.17.34.246), 30 hops max, 40 byte packets
1 DD-WRT (10.0.0.1) 3.366 ms 4.645 ms 4.922 ms
2 * * *
3 ge-1-3-0-181.aggr02.austtx.grandecom.net (66.90.138.254) 25.281 ms 25.540 ms 25.533 ms
4 so-2-1-0-0.core02.dllstx.grandecom.net (24.155.121.27) 30.589 ms 30.851 ms 30.843 ms
5 so-2-0-0-0.core02.smrctx.grandecom.net (24.155.121.227) 36.199 ms 36.454 ms 36.445 ms
6 bcr2-so-3-0-0.dallas.savvis.net (208.172.129.137) 41.144 ms 35.960 ms 35.932 ms
7 cr2-pos0-0-3-0.dallas.savvis.net (204.70.193.14) 37.597 ms 38.234 ms 38.206 ms
8 cr2-loopback.nyr.savvis.net (206.24.194.71) 83.874 ms 88.800 ms 92.931 ms
9 204.70.197.9 (204.70.197.9) 79.627 ms 282.566 ms 282.309 ms
10 208.174.224.138 (208.174.224.138) 79.609 ms 87.955 ms 83.239 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

From webserver @ theplanet:

traceroute 207.17.34.246
traceroute to 207.17.34.246 (207.17.34.246), 30 hops max, 38 byte packets
1 d9.df.344a.static.theplanet.com (74.52.223.217) 0.897 ms 0.859 ms 0.817 ms
2 vl2.dsr02.dllstx2.theplanet.com (12.96.160.42) 0.416 ms 0.555 ms 0.521 ms
3 7d.fd.5746.static.theplanet.com (70.87.253.125) 0.544 ms 0.717 ms 0.528 ms
4 et5-2.ibr04.dllstx3.theplanet.com (70.87.253.29) 0.502 ms 0.659 ms 0.456 ms
5 12.87.41.149 (12.87.41.149) 1.073 ms 0.866 ms 0.843 ms
Icmp checksum is wrong
6 tbr2.dlstx.ip.att.net (12.122.100.98) 40.935 msIcmp checksum is wrong
40.636 msIcmp checksum is wrong
40.567 ms
Icmp checksum is wrong
7 tbr1.sl9mo.ip.att.net (12.122.10.89) 38.555 msIcmp checksum is wrong
38.583 msIcmp checksum is wrong
38.534 ms
Icmp checksum is wrong
8 tbr1.wswdc.ip.att.net (12.122.10.29) 40.195 msIcmp checksum is wrong
40.302 msIcmp checksum is wrong
42.343 ms
Icmp checksum is wrong
9 cr2.wswdc.ip.att.net (12.122.16.5) 39.935 msIcmp checksum is wrong
40.152 msIcmp checksum is wrong
39.930 ms
Icmp checksum is wrong
10 cr2.n54ny.ip.att.net (12.122.3.37) 37.017 msIcmp checksum is wrong
36.936 msIcmp checksum is wrong
37.211 ms
Icmp checksum is wrong
11 tbr2.n54ny.ip.att.net (12.122.16.198) 40.138 msIcmp checksum is wrong
40.552 msIcmp checksum is wrong
40.756 ms
12 gar1.nw2nj.ip.att.net (12.123.219.185) 37.094 ms 37.077 ms 37.072 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *


It just times out @ 30 tries just like the first one too.

 

[m1ldslide1]

I don't intuitively see anything wrong with your traceroutes. The timeouts are pretty normal for many remote networks, as people can block ICMP traceroute. I'm not sure about the checksum, I ran the same tracert from a DOS cmd window and didn't get those (but did get "Destination net unreachable" from my ISP).

What is the application problem? It might be possible to narrow it down some so that you'll know for sure if it's the network, and then maybe know which part of your network could be causing the problem.

 

[Crusty]

Well, the application is a real time trading suite that uses a heartbeat/ping to monitor the connection between the server and client program. One desktop has the program crash repeatably when put under decent load, around 80% CPU and upwards of 500kbps in and out streaming. According to the tech support at the vendor the software is crashing because the heartbeat latency is jumping, although I'm having a tough time believing that now. This desktop is the only machine to have the problem, and it has no other network issues. I've replaced all the network hardware including cabling all the way to the switch, where I've switched ports and still have the same issue. I attached a network sniffer on the machine as well and I do not notice any spikes in latency on the heartbeat packets.

I'm just trying to get as much information on this problem before I call back their tech support.

 

[spidey07]

The only real way to see what is going on is with a packet trace, you can provide that to the app people. I've seen high latency or a high variance in latency cause all kinds of problems with timesensitive trading.

Also you could ping the server from the client to see if the problem is related, however keep in mind ping is not an accurate measure of latency. The best is a packet trace.

 

[Crusty]

Ping requests are blocked so that doesn't help, but I'll try and grab a few packet traces from when the app crashes and forward that onwards.

Thanks for the input.