Really high packet loss in server hop

[oblivion53074]

Basically this is a trace to a Chicago server which I live less than 100 miles from. For some reason I'm getting really high packet loss at 107.14.19.97 which is a Road Runner address in LA. My ISP is indeed Road Runner.

My questions here is why is my server hop going all the way to LA which is out of the way and is there anything I can do to bypass this hop?

 

[John Connor]

I just did a tracert and a ping test and that IP works.

 

[oblivion53074]

Well, we have different internet hops and routes. What IP did you test?

 

[John Connor]

The one you gave.

 

[oblivion53074]

Right, I see the bad hop I have is on 107.14.19.97 which is Road Runner LA. Why would my internet be going out to LA and back to Chicago when I live less than 100 miles from Chicago? You are getting good results because your not hitting the same routes as me obviously being from a different location.

This one hop is giving me 10% loss is there anyway I can avoid it?

 

[John Connor]

Well. I'm not a server expert so maybe someone else with experience can chime in. I did a tracert to the server address and here is what I get. I think it's your ISP.

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\AAron>tracert 208.172.0.1

Tracing route to ges1-ethernet.Chicago.savvis.net [208.172.0.1]
over a maximum of 30 hops:

1 1 ms 1 ms 2 ms Skynet [192.168.1.1]
2 50 ms 49 ms 51 ms ges1-ethernet.Chicago.savvis.net [208.172.0.1]

Trace complete.

 

[John Connor]

BTW, when I try connecting to 208.172.0.1 I get nothing. What kind of server is this?

 

[oblivion53074]

I'm not one either I guess its the routes my ISP is taking on the way to the server at the end?

Look at this 23% packet loss on the way to the end server. Seriously? What can I do..

 

[Gryz]

Inside (small) networks packets flow via the shortest paths. Small networks like company network, campus networks, etc. Even inside an ISP's network.

Between large networks the routing is not done in such an automatic way. Routing is done mostly via human-made policy. Because you don't only want packets to flow via the shortest path, but also via the cheapest path. Or at least make sure someone pays to use your network as transit. Etc. So any routing from your ISP to another ISP is decided by human-made policy, not by routing protocols. That's why traffic might flow via unexpected ways.

There's nothing you can do yourself. It's all in the hands of your ISP.

Having said that, 10% packetloss is high. And most ISPs won't like that. Your ISP might not be aware of the problem. So just call them, tell them about the problem. They might change their routing a bit, and let the traffic flow via another path another peering or another transit ISP. I think it's worth a try.

 

[SecurityTheatre]

Inside (small) networks packets flow via the shortest paths. Small networks like company network, campus networks, etc. Even inside an ISP's network.

Between large networks the routing is not done in such an automatic way. Routing is done mostly via human-made policy.

This is accurate. Their peering agreement between ISPs may only allow them ingress via a few points (perhaps in LA and NYC), or they have an oversubscribed gateway in Chicago and are re-routing some traffic to reduce load on a busy node.

Internet routing is astoundingly complex (while being simultaneously elegantly simple). But there is no way for you to change it from there.

As for the packet loss, I've done a series of pings and traces both surrounding the destination server, as well as the gateway that you seem to be having trouble with and although I can't get that exact route, I can hit some of the surrounding nodes and none seem to be behaving too poorly.

I do get some high-ish RTT (response time) from several of them around there, so my guess is that there is a node there that is bandwidth-saturated or has a bad link.

Is it still going on?

 

[oblivion53074]

It has been going on for a long long time now and still is going on. So your telling me going to Chicago to LA back to Chicago is a man made route by my ISP? That seems bizarre. The server I was tracing is a game server the address as you see is 208.172.0.1, ges1-ethernet.Chicago.savvis.net. I notice that the bad hop I am getting 10% loss in is my own ISP route which is out of the way again being Chicago to LA back to Chicago. Seems like they gone out of their way to save money?


Here are some other Chicago traces I did

Why does everything I trace to in Chicago run like ass?

 

[drebo]

Traceroutes are handled by the controlplane of a router and are not necessarily indicative of packet loss to the end destination, nor are the indicative of delays to the end destination. Most high-end ISP routers have separate control and data planes. The control plane getting hammered (by people doing pings/traceroutes or large routing table updates, etc) will cause pings and traceroutes to time out or take a long time. That doesn't necessarily mean that the data passing through the router is affected.

Notice, for instance, that there is no packet loss to your end destination, despite there being loss to some of the hops along the way.

The latency in your case is due to the packets going to California before they leave RR's network and enter Saavis's network. There is no way to get your ISP to reroute your traffic.

Basically, there isn't necessarily a problem here, nor is there anything you can do about it.

 

[oblivion53074]

Traceroutes are handled by the controlplane of a router and are not necessarily indicative of packet loss to the end destination, nor are the indicative of delays to the end destination. Most high-end ISP routers have separate control and data planes. The control plane getting hammered (by people doing pings/traceroutes or large routing table updates, etc) will cause pings and traceroutes to time out or take a long time. That doesn't necessarily mean that the data passing through the router is affected.

Notice, for instance, that there is no packet loss to your end destination, despite there being loss to some of the hops along the way.

The latency in your case is due to the packets going to California before they leave RR's network and enter Saavis's network. There is no way to get your ISP to reroute your traffic.

Basically, there isn't necessarily a problem here, nor is there anything you can do about it.

What about the other traces to IL2 and IL4? So your telling me the packet loss in the hops on the way to the server won't bother anything as long as there is no loss to the end destination?

Look at the IL2 trace picture. Why the hell do I get a worst spike of 900 on the destination server. Why does my ping to the savvis server jump up to 260? How come when I play the game my ping goes from 17-50 up to 300-900 for a second and is just all over the place.

This is the latency on the game server I am experiencing all within a minute apart

Well, you seen my traces so who is to blame for this? My ISP, the routes or the server host.

 

[drebo]

What about the other traces to IL2 and IL4? So your telling me the packet loss in the hops on the way to the server won't bother anything as long as there is no loss to the end destination?

No, I'm telling you that packet loss in a traceroute or ping to a router along the way does not indicate that there will be packet loss to the end destination.

When you ping a router or traceroute through a router, that router has to respond, which requires CPU cycles to do. If you ping a device on the other side of the router, the router simply forwards the ping request to the next hop, which requires significantly fewer CPU cycles (or none, depending on the architecture of the router.)

The difference is that connectivity to the device on the other side of the router is not dependent upon the router responding to ping or traceroute packets, and thus issues with ping and traceroutes are not necessarily indicators of a problem with connectivity to the end device. Ping and traceroute are useful troubleshooting tools, but unless you own the network end-to-end and know exactly what's supposed to be happening, they are not definitive.

Basically, in regards to a router, answering a request and forwarding a request are two completely different actions handled by two completely different parts of the router. A failure in one doesn't indicate a failure in the other.

 

[drebo]

Look at the IL2 trace picture. Why the hell do I get a worst spike of 900 on the destination server. Why does my ping to the savvis server jump up to 260? How come when I play the game my ping goes from 17-50 up to 300-900 for a second and is just all over the place.

This is the latency on the game server I am experiencing all within a minute apart

Well, you seen my traces so who is to blame for this? My ISP, the routes or the server host.

Could be quite a few different things. It could be that the server itself is experiencing periods of high CPU utilization and thus taking extra time to respond to ping. It could be congestion somewhere in the network. It could be congestion on YOUR network or in YOUR connection to the Internet.

Like I said in my last post, unless you own the network end to end and know exactly what "normal" is on that network, pings and traceroutes only tell you about 10% of the story.

 

[oblivion53074]

How do I begin testing to troubleshoot the issue then? Please keep in mind that I ONLY experience these ping spikes on this specific game. I could be playing any other game in the world and get stable ping that only jumps by 10 ms on different servers.

Amazing how I play another game on a Chicago server and I sit at 30-50 ms and I go to Dallas server and it sits at 50-60 ms zero spikes or jumps no issues. There has to be something wrong with this specific server.

I notice that the server issue gets REALLY bad in the morning around 2 AM and later.

 

[SecurityTheatre]

How do I begin testing to troubleshoot the issue then? Please keep in mind that I ONLY experience these ping spikes on this specific game. I could be playing any other game in the world and get stable ping that only jumps by 10 ms on different servers.

Amazing how I play another game on a Chicago server and I sit at 30-50 ms and I go to Dallas server and it sits at 50-60 ms zero spikes or jumps no issues. There has to be something wrong with this specific server.

I notice that the server issue gets REALLY bad in the morning around 2 AM and later.

Sounds pretty clear that it's this specific server...

The issue is.... ? --)

 

[Zargon]

No, I'm telling you that packet loss in a traceroute or ping to a router along the way does not indicate that there will be packet loss to the end destination.

When you ping a router or traceroute through a router, that router has to respond, which requires CPU cycles to do. If you ping a device on the other side of the router, the router simply forwards the ping request to the next hop, which requires significantly fewer CPU cycles (or none, depending on the architecture of the router.)

The difference is that connectivity to the device on the other side of the router is not dependent upon the router responding to ping or traceroute packets, and thus issues with ping and traceroutes are not necessarily indicators of a problem with connectivity to the end device. Ping and traceroute are useful troubleshooting tools, but unless you own the network end-to-end and know exactly what's supposed to be happening, they are not definitive.

Basically, in regards to a router, answering a request and forwarding a request are two completely different actions handled by two completely different parts of the router. A failure in one doesn't indicate a failure in the other.

this.

drebo is spot on.

though you could open a ticket with your ISP , it may be a routing bug/bad route that they arent aware of