Strange traceroute issues..Can someone expain?

[stimpyman77]

Hello everyone..

Enclosed are the trace routes from 2 boxes I have first is a Suse 9.1 Pro and the other is Win2k SP4. I found it hard to understand why the information differs because they are both behind the same router and firewall. There are no static routes involved on either box aside of default gateway. Could someone enlighten me? Why does the linux box see another hop and the Windows box does not?

I discovered this today whey trying to Remote Desktop to a clients system and could not establish any connection. I have verified on their end that they are up and there internet is fine.. we are both on Comcast internet btw. Funny because it looks like it is a routing issue to me, but Comcast said they could not help me with it... :disgust: I also verified on a friends system (closer to their node) and he got destination net unreachable also? Any ideas?

Linux traceroute..

traceroute to 24.128.120.xxx (24.128.120.xxx), 30 hops max, 40 byte packets
1 192.168.0.1 1.116 ms 1.101 ms 2.248 ms
2 10.210.160.1 12.953 ms 18.595 ms 21.948 ms
3 bar01-p12-1-0.drryhe1.nh.attbb.net (24.147.0.101) 26.069 ms 30.096 ms 34.615 ms
4 24.128.191.65 39.265 ms 37.892 ms 11.733 ms
5 10.212.240.1 13.573 ms 19.571 ms 24.050 ms
6 (24.128.120.xxx) 768.218 ms 778.726 ms 791.232 ms

Windows traceroute..

tracing route to 24.128.120.xxx over a maximum of 30 hops

1 <10 ms <10 ms <10 ms 192.168.0.1
2 20 ms 10 ms 10 ms 10.210.160.1
3 10 ms 10 ms 20 ms 24.147.0.101
4 20 ms 10 ms 10 ms 24.128.191.65
5 24.128.191.137 reports: Destination net unreachable.

Trace complete.

Thanks!

Stimpyman77

 

[Nothinman]

It looks like the Linux one actually makes it to the destination host and the Windows one doesn't, probably because Linux traceroute uses UDP instead of ICMP ECHO to do the tracing and a lot of places like to filter different types of ICMP for security reasons.

If you can't remote desktop in it's probably being filtered at 24.128.191.137, to rule out your Windows box being the problem why not try rdesktop on the Linux box?

 

[stimpyman77]

Nothinman,

Thanks for the quick reply and your thoughts.. I was just in the middle of downloading X11 libraries for Rdesktop 1.3.1 to install it. (Dam!! I love how Linux has a program for everything!!!) I did not know that about Linux traceroute using UDP .. Thanks for that info! ... I don't feel that it is an issue with the box as I was able to do this without issue for months and not much has changed on my box,all this crapola started last Friday.. I feel that somewhere there is a routing loop or bad config somewhere on their end. Time will tell.. Thanks again..

 

[stimpyman77]

Good morning bumpage...

 

[Fiveohhh]

Originally posted by: stimpyman77
Good morning bumpage...

Didn't Nothinman answer your question allready?

 

[stimpyman77]

Fiveohhh,

Nothinman did answer my question for me.. I had bumped before I thought about his answer...but I am still a little confused. I am not sure where the filtering would be? When host 24.128.191.65 gets a packet destined for the 24.128.120.xxx why does it choose to send it 2 different ways?? Is that not done by address not protocol? I am not sure why the Linux packet did not get to the 24.128.191.137 address as well.. Sorry for my newbieness....

Stimpyman77

 

[classy]

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

 

[classy]

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests. I am a learning Linux newbie . But I did a google and found this thread post which is pretty good at explaining the differences. Maybe this might help with your problem stimpy. The thread is here.

 

[Nothinman]

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

The system will never 're-route' itself, the routing is purely determined by the decisions by each router enroute. Any router that honors source routed packets is setup wrong. There is a chance that the extra hop that Windows reported was just pure luck, routes can change at any time depending on what's happening on different parts of the network.

The 137 router that dropped the packet may only be setup to filter ICMP, meaning UDP and TCP could make it through when a ping or windows traceroute won't.

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests.

There's a switch to tell it to use ICMP if you really want to do that.

 

[classy]

Originally posted by: Nothinman

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

The system will never 're-route' itself, the routing is purely determined by the decisions by each router enroute. Any router that honors source routed packets is setup wrong. There is a chance that the extra hop that Windows reported was just pure luck, routes can change at any time depending on what's happening on different parts of the network.

The 137 router that dropped the packet may only be setup to filter ICMP, meaning UDP and TCP could make it through when a ping or windows traceroute won't.

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests.

There's a switch to tell it to use ICMP if you really want to do that.

Maybe I am using the wrong wording when I said "re-route". I guess my question is if you send out a packet to a remote network, if one router fails for some reason to pass the packet doesn't the packet retry by using another route to reach the network? Because the internet uses ospf which is suppose to calculate changes in the routes correct. So if one route fails shouldn't the packet be redirected to another route? Or am I missing something in respect to using tracert in particular or am just plain out in left field, lol?

 

[Fiveohhh]

Originally posted by: classy

Originally posted by: Nothinman

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

The system will never 're-route' itself, the routing is purely determined by the decisions by each router enroute. Any router that honors source routed packets is setup wrong. There is a chance that the extra hop that Windows reported was just pure luck, routes can change at any time depending on what's happening on different parts of the network.

The 137 router that dropped the packet may only be setup to filter ICMP, meaning UDP and TCP could make it through when a ping or windows traceroute won't.

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests.

There's a switch to tell it to use ICMP if you really want to do that.

Maybe I am using the wrong wording when I said "re-route". I guess my question is if you send out a packet to a remote network, if one router fails for some reason to pass the packet doesn't the packet retry by using another route to reach the network? Because the internet uses ospf which is suppose to calculate changes in the routes correct. So if one route fails shouldn't the packet be redirected to another route? Or am I missing something in respect to using tracert in particular or am just plain out in left field, lol?

My knowledge on the subject is limited, but from what I know the packet itself won't reroute itself(it doesn't think on its own) It goes where its told. To have the packet rerouted the owner of that router would have to set up to route a different way, or a router sometime before that would need to reroute it. If a network went down and packets were trying to get through that network, the packet woudn't say "allright this networks down lets try another". It would have to be redirected by a router before it got to that network, otherwise it would just drop.

Again my knowledge on this is fairly limited, but thats the way I understand it.

 

[spidey07]

Originally posted by: classy

Originally posted by: Nothinman

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

The system will never 're-route' itself, the routing is purely determined by the decisions by each router enroute. Any router that honors source routed packets is setup wrong. There is a chance that the extra hop that Windows reported was just pure luck, routes can change at any time depending on what's happening on different parts of the network.

The 137 router that dropped the packet may only be setup to filter ICMP, meaning UDP and TCP could make it through when a ping or windows traceroute won't.

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests.

There's a switch to tell it to use ICMP if you really want to do that.

Maybe I am using the wrong wording when I said "re-route". I guess my question is if you send out a packet to a remote network, if one router fails for some reason to pass the packet doesn't the packet retry by using another route to reach the network? Because the internet uses ospf which is suppose to calculate changes in the routes correct. So if one route fails shouldn't the packet be redirected to another route? Or am I missing something in respect to using tracert in particular or am just plain out in left field, lol?

I'll try to help clarify.

A router routes. That is it receives packets on one interface and decides which interface to spit it out. In order to do this it looks at the desitnation address of the packet, looks in its routing table and goes "OH! This packet is destined for 24.128.120.0 network. I know that I need to send this to my other buddy router at address 10.212.240.1."

So the packet doesn't really no or care because the packet is never modified (well the time to live is decremented by one so we don't have packets from 1998 flying all around).

A router knows which interface to spit the packet out of by talking with other routers and determing the best path to a network and figuring out what the next hop address is. That is called its routing table - a table of IP networks and the next hop and the path properties.

All this means it is a routers responsiblity to deliver the packet. In your first scenario the traceroute packets were delivered successively along the path to each intermediate router and finally to the final destination.

In the second scenario you got a destination net unreachable message. That's an ICMP from a router that says "HEY!!! I don't know where to send your packet so I'll be nice an let you know I threw it away"

Most likely there is a filter or "black-hole" route for the type of packets windows is using to traceroute. I ran a packet capture and windows uses ICMP echo-request packets to do trace route. That's why you get a "destination net unreachible"

That last router in the windows traceroute is "bit bucketing" ICMP echo-request packets. any ICMP echo-request packets are being routed to a special interface called "null0" telling the router to drop it. It is better than an access list because it uses the speed of the router instead of an ACL and the additional processing that goes with performing and ACL on every frame.

So long story short, as was mentioned the different OSs use different means of tracing a route. Windows is using ICMP and the packet is getting dropped. Thankfully the router is nice engough to tell you about it.

 

[classy]

Originally posted by: spidey07

Originally posted by: classy

Originally posted by: Nothinman

Hmmm........Nman might be right, but that doesn't explain why windows won't reach that network. Even if it was filtered at that router which is possible, it still doesn't explain why it wouldn't re-route itself to another router to reach its destination as it should, right? Can you ping the ip addy? If you can ping it you should be able to establish a connection.

The system will never 're-route' itself, the routing is purely determined by the decisions by each router enroute. Any router that honors source routed packets is setup wrong. There is a chance that the extra hop that Windows reported was just pure luck, routes can change at any time depending on what's happening on different parts of the network.

The 137 router that dropped the packet may only be setup to filter ICMP, meaning UDP and TCP could make it through when a ping or windows traceroute won't.

Good stuff Nman . I never knew Linux used udp packets instead of ICMP echo requests.

There's a switch to tell it to use ICMP if you really want to do that.

Maybe I am using the wrong wording when I said "re-route". I guess my question is if you send out a packet to a remote network, if one router fails for some reason to pass the packet doesn't the packet retry by using another route to reach the network? Because the internet uses ospf which is suppose to calculate changes in the routes correct. So if one route fails shouldn't the packet be redirected to another route? Or am I missing something in respect to using tracert in particular or am just plain out in left field, lol?

I'll try to help clarify.

A router routes. That is it receives packets on one interface and decides which interface to spit it out. In order to do this it looks at the desitnation address of the packet, looks in its routing table and goes "OH! This packet is destined for 24.128.120.0 network. I know that I need to send this to my other buddy router at address 10.212.240.1."

So the packet doesn't really no or care because the packet is never modified (well the time to live is decremented by one so we don't have packets from 1998 flying all around).

A router knows which interface to spit the packet out of by talking with other routers and determing the best path to a network and figuring out what the next hop address is. That is called its routing table - a table of IP networks and the next hop and the path properties.

All this means it is a routers responsiblity to deliver the packet. In your first scenario the traceroute packets were delivered successively along the path to each intermediate router and finally to the final destination.

In the second scenario you got a destination net unreachable message. That's an ICMP from a router that says "HEY!!! I don't know where to send your packet so I'll be nice an let you know I threw it away"

Most likely there is a filter or "black-hole" route for the type of packets windows is using to traceroute. I ran a packet capture and windows uses ICMP echo-request packets to do trace route. That's why you get a "destination net unreachible"

That last router in the windows traceroute is "bit bucketing" ICMP echo-request packets. any ICMP echo-request packets are being routed to a special interface called "null0" telling the router to drop it. It is better than an access list because it uses the speed of the router instead of an ACL and the additional processing that goes with performing and ACL on every frame.

So long story short, as was mentioned the different OSs use different means of tracing a route. Windows is using ICMP and the packet is getting dropped. Thankfully the router is nice engough to tell you about it.

Ok cool. So I can assume this just applies to ICMP packets correct?

 

[stimpyman77]

Classy, Nothinman, Fiveohhh and Spidey07 (Spidey07 -- I was hoping that you would see this) thank you for taking time to help me understand this. Just to update everyone a little bit, I had talked to someone at Comcast's NOC and they did verify with me that the 24.128.191.137 because of its model and make of which I forgot the name .. can't touch ICMP because it will hang and reboot. It is the head end router for that area and plans are in the works for its replacement. I understand now why I got the messages I did for the traceroute now. But in my newbieness, I have to ask another one... when 24.128.191.65 gets the packet destined for 24.128.120.xxx it forwards it to 10.212.240.1 (because it knows how to get to 24.128.120.xxx) and from there it hits the destination. That packet was UDP. but when the Windows machine sends it to the router at 24.128.191.65 it goes and sends it straight to 24.128.191.137 why did it not send it to 10.212.240.1 like the other one (route is based on IP not UDP or ICMP)??

If sending it to 10.212.240.1 was the fastest way before .. why is it not now? Traffic is to the same destination just protocol has changed. I know that 24.128.191.137 can not send ICMP so that is why my pings never get there. I am still confused why it did not go like this:

Windows traceroute..

tracing route to 24.128.120.xxx over a maximum of 30 hops

1 <10 ms <10 ms <10 ms 192.168.0.1
2 20 ms 10 ms 10 ms 10.210.160.1
3 10 ms 10 ms 20 ms 24.147.0.101
4 20 ms 10 ms 10 ms 24.128.191.65
5 ??ms ??ms ??ms 10.212.240.1
6 24.128.191.137 reports: Destination net unreachable.

I am really sorry if I am ignorant about this and making it harder than it is, but I really want to understand it correctly and I feel I am missing something still. Is there something in the upper layers I am getting wrong?

Thank you everyone.

Stimpyman77

 

[spidey07]

Linux traceroute..

traceroute to 24.128.120.xxx (24.128.120.xxx), 30 hops max, 40 byte packets
SAME - 1 192.168.0.1 1.116 ms 1.101 ms 2.248 ms
SAME -2 10.210.160.1 12.953 ms 18.595 ms 21.948 ms
SAME - 3 bar01-p12-1-0.drryhe1.nh.attbb.net (24.147.0.101) 26.069 ms 30.096 ms 34.615 ms
SAME - 4 24.128.191.65 39.265 ms 37.892 ms 11.733 ms
5 10.212.240.1 13.573 ms 19.571 ms 24.050 ms
6 (24.128.120.xxx) 768.218 ms 778.726 ms 791.232 ms

Windows traceroute..

tracing route to 24.128.120.xxx over a maximum of 30 hops

1 <10 ms <10 ms <10 ms 192.168.0.1
2 20 ms 10 ms 10 ms 10.210.160.1
3 10 ms 10 ms 20 ms 24.147.0.101
4 20 ms 10 ms 10 ms 24.128.191.65
5 24.128.191.137 reports: Destination net unreachable.

working on this later...gotta go. Something fishy is going on here. Try messing with the options on your linux box for trace route. But at its core the reason why you're seeing this behavior is one is using udp, the other icmp-echo request.

 

[ScottMac]

I may be able to toss in a few cents worth here.

Some branches of the SBC portions of the Internet will filter things like 64 (and 1500, and ?1496?) byte pings. The rationale is that many / most / all worms / trojans / viruses use ping to continue propagation ... default Windwos ping size is 64 bytes .... so they filter it.

If you try a ping with a different (non-filtered) size, the ping makes it through (try ping -l 65 XX.XX.XX.XX ...that's a "dash-lower-case-L" to describe the ping packet size).

Tracert and other similar applications may also be filtered ... I don't know. *nix uses 100 byte pings, perhaps they also use a different size Traceroute packet ... I don't know ... just tossing something out for discussion.

I believe the fiiltering policies are not just limited to SBC, I'm sure other ISPs have done similar things (at least temporarily) to help contain the malicious propagation.

That might explain why Windows and *nix are not displaying identical paths / routes / behavior.

FWIW

Scott