ICMP Echo - Strange Issue - Need Help

loupag007

Junior Member
Jul 24, 2015
2
0
0
Hi Everyone,

I have an industrial device in the network. From Windows, if I ping that device from CMD with the default parameters (32 bytes of data), then the device replies to the echo request fine with all 32 bytes returned exactly (all good).

Problem is:

If I ping the device again from CMD with only 5 bytes of data ( - l 5 parameter), then it returns 18 bytes of data. I see in Wireshark that the Echo Request sends 5 bytes, and the Echo Reply contains 18 bytes. Actually, the Echo Reply contains the first 5 bytes as the request, but then adds padding to the response by adding some extra ~000 at the end.

Has anyone encountered this before?
I wonder, under what reasons a device return with more data than requested.

p.s. - We have other devices on the same network and subnet, that respond to 5 bytes of data just fine.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
It is likely a bug / poor developer decision in the TCP/IP stack implemented on the device. It is a fairly common issue in the industrial world to have loose or poor implementations of the standards.

RFC792 specifies to return the echo data unaltered. The fact that it is nulls (byte 00) on the end it likely a developer feeling (s)he needed to pad the frame out to a certain size. IE the TCP/IP minimum frame size of 46 bytes or ethernet's 64byte minimum.
 

Gryz

Golden Member
Aug 28, 2010
1,551
204
106
Where do you see that number 18 ? Does your ping application tell you ? Is it WireShark that explicitly mentions the number 18 ? Or is it what you counted yourself when looking at the hexdump of the packet ?

That number 18 is not a coincidence.
I'll give you 2 clues. :)

First one:
https://www.google.nl/search?complete=0&ion=0&q=minimum%20framesize%20ethernet
https://en.wikipedia.org/wiki/Ping_(networking_utility)#ICMP_packet

Second one:
64 - 18 - 20 - 8 = ?

Let me know when you get it. :)
If not, please copy&paste a hexdump of the full Ethernet frame here (including Ethernet-frame-header), so I can see exactly what values are in the headers. Thanks.


====
Edit:
OK, it's not as straightforward as I thought.
But I'm pretty sure that number 18 is because of padding.
Note the difference in terminology here between packet (ip, layer-3) and frame (ethernet, layer-2).

Ethernet frames that are smaller than 64 bytes need to be padded to 64 bytes length.
The padding should be done in the frame *after* the layer-3 packet.
The frame-length should be 64, and never smaller.
You should also know that there are 4 types of Ethernet-encapsulation.
1) Arpa, 2) OSI/LLC, 3) SNAP, and 4) some old Novell crap that I hope nobody uses anymore.
Note that encapsulations 3) and 4) use the 2 bytes after ether-DA and ether-SA as a length field. But Arpa-encapsulation does not use those 2 bytes as length. Arpa-encaps uses it as the type field. (Note, all types are supposed to be larger than 1518, so there's no confusion. IP is type 2048, which is larger than 1518, so no confusion).

So we can conclude that when we use arpa-encaps (like IP does), the frame-length is implicit, not explicit like a field inside the frame-header.

Another note: Ethernet-headers are 18 bytes. That's 14 bytes at the start (6 destaddr, 6 srcaddr, and 2 type/length). And 4 bytes CRC at the end. Wireshark seems to not show the CRC. So that is guaranteed to cause confusion.

Normally icmp-echo-requests have 32 bytes of random data attached to it. This makes the total frame size: 18 ethernet-header + 20 ip-header + 8 icmp-header + 32 icmp-random-data = 78 bytes. My WireShark (windows version. 1.12.2) says 74 bytes. It forgets about the 4 bytes CRC.

When I ping with length zero (e.g. ping -l 0 8.8.8.8) I see a packet going out of size 42. Let's add the 4 bytes CRC that WireShark forgets about. That's 46 bytes then. That means that that frame will have exactly 18 bytes of padding. Wireshark just doesn't show it to us. Inside the ip-header it says "total length = 28". That's 20 bytes ip-header + 8 bytes icmp-header. It all adds up.

Now I look at the return packet. From the router (a Fritzbox by AVM) to my PC. Now WireShark says it sees a 60-byte return frame. Add the 4-bytes CRC, and we are at the magic 64 bytes minimum Ethernet-frame size ! If we look at the total-length field in the ip-header, it says 28 bytes, just as it should.

My conclusion:
WireShark does not see the ethernet-frame padding on outgoing packets.
But it does see the ethernet-frame padding on incoming packets.

Maybe winpcap is to blame ? (Winpcap is the driver that hands-over frames/packets from the windows drivers to WireShark).

Now that I knew where to look, it was easy to find confirmation via Google.
First hit:
http://palewhiteman.blogspot.nl/2010/02/why-wireshark-doesnt-show-padding-on.html

My advice:
1) Look at the total-length field in the ip-header. If that is correct, then you got nothing to worry about.
2) Attach a 3rd device to your network. (A bit hard to do these days, with switches everywhere, and no hubs for sale anymore). Do the WireShark sniffing on the 3rd device. I bet you will now see padding on frames in both directions.

Hope this helps.
This was actually a fun 15 minutes to look into it.
It's been a long time ago since I looked at this. I needed to verify my old memories. :)
 
Last edited:

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Where do you see that number 18 ? Does your ping application tell you ? Is it WireShark that explicitly mentions the number 18 ? Or is it what you counted yourself when looking at the hexdump of the packet ?

That number 18 is not a coincidence.
I'll give you 2 clues. :)

First one:
https://www.google.nl/search?complete=0&ion=0&q=minimum%20framesize%20ethernet
https://en.wikipedia.org/wiki/Ping_(networking_utility)#ICMP_packet

Second one:
64 - 18 - 20 - 8 = ?

Let me know when you get it. :)
If not, please copy&paste a hexdump of the full Ethernet frame here (including Ethernet-frame-header), so I can see exactly what values are in the headers. Thanks.

Wireshark specifically calls out padding. Additionally:

TCP/IP level: Padding is inserted in the options portion of the frame, not the data. (Aligning the options to a 32 bit boundary)
Ethernet: Padding is added to the end of the data portion it is located via math (ie 46 - length field as long as length is < 46, ie Length 8 = 46 - 8 So "Pad" = 32)

Wireshark normally misses the padding on the Ethernet frame when sending (since it doesn't hook at the wire level on the sending PC and often doesn't have access that deep in the NIC..
The padding will show on the receiving machine typically at a frame length of 60bytes since the 4 bytes for the FCS are stripped out at the card level.

So wireshark should be showing in the reply:

[IP][ICMP[ICMP Data]] or in bytes [20][8][5] or 33Bytes. Add the frame:

[Header][data (ICMP packet)][pad][FCS]
Since length = 33 bytes
[14][33][13][4]=64bytes

Wireshark will often only show 60 because the FCS isn't sent to the computer itself or:

[14][33][13] = 60bytes

If the "Data" portion of the ICMP packet is showing the 5 bytes of data and padding zeros (13 bytes should be 26 zeros) then device is padding the data field or in reporting an invalid length such as a junior mistake of doing the 46 byte check in the TCP stack when it should be left to the NIC.

PS I mistakenly mention TCP/IP has the 46 byte minimum size, that is incorrect the Ethernet data payload size is what I meant to say.

Point is the padding is at the Ethernet frame and padding out the "data" to make it at least 46 bytes prior to sending it to the NIC is a software defect on the device and a violation of the RFC.
 

loupag007

Junior Member
Jul 24, 2015
2
0
0
Hello,

First, I appreciate the in-depth information. This is very helpful, not just for determining the root cause, but I also picked up a few things from your responses.

We found the issue from a pretty rare Winsock message ERROR 11050 (IP_GENERAL_FAILURE), ICMPSendEcho that appeared in our software. These captures were then made by Wireshark on the Source Server, the one sending the Requests. Sorry, but I cannot upload the actual capture files.

Here is the hex stream from the ICMP Echo request:
00602d015829d89d677772e00800450000210f980000ff01000088f14c0b88f14d0a0800f537006963ce50494e4700

And here is the hex stream from the ICMP Echo reply:
d89d677772e000602d01582908004500002e0ffb00004001bfdc88f14d0a88f14c0b0000fd37006963ce50494e470000000000000000000000000000

Thanks again,
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Hello,

First, I appreciate the in-depth information. This is very helpful, not just for determining the root cause, but I also picked up a few things from your responses.

We found the issue from a pretty rare Winsock message ERROR 11050 (IP_GENERAL_FAILURE), ICMPSendEcho that appeared in our software. These captures were then made by Wireshark on the Source Server, the one sending the Requests. Sorry, but I cannot upload the actual capture files.

Here is the hex stream from the ICMP Echo request:
00602d015829d89d677772e00800450000210f980000ff01000088f14c0b88f14d0a0800f537006963ce50494e4700

And here is the hex stream from the ICMP Echo reply:
d89d677772e000602d01582908004500002e0ffb00004001bfdc88f14d0a88f14c0b0000fd37006963ce50494e470000000000000000000000000000

Thanks again,

Please include the Wireshark breakout. The raw string isn't quite as useful. Mostly because depending on which one you grabbed, differing parts will be inside it.
 
Last edited:

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Original Echo request:

Frame 1 (47 bytes on wire, 47 bytes captured)

Arrival Time: Jul 27, 2015 20:24:07.000000000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 47 bytes
Capture Length: 47 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:icmp:data]

Ethernet II, Src: d8:9d:67:77:72:e0 (d8:9d:67:77:72:e0), Dst: AlertonT_01:58:29 (00:60:2d:01:58:29)

Destination: AlertonT_01:58:29 (00:60:2d:01:58:29)
Source: d8:9d:67:77:72:e0 (d8:9d:67:77:72:e0)
Type: IP (0x0800)

Internet Protocol, Src: 136.241.76.11 (136.241.76.11), Dst: 136.241.77.10 (136.241.77.10)

Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
Total Length: 33
Identification: 0x0f98 (3992)
Flags: 0x00
Fragment offset: 0
Time to live: 255
Protocol: ICMP (0x01)
Header checksum: 0x0000 [incorrect, should be 0x014c]
Source: 136.241.76.11 (136.241.76.11)
Destination: 136.241.77.10 (136.241.77.10)

Internet Control Message Protocol

Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0xf537 [correct]
Identifier: 0x0069
Sequence number: 25550 (0x63ce)
Data (5 bytes)

0000 50 49 4e 47 00 PING.

Data: 50494E4700

Here is the decode of the reply:

Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Jul 27, 2015 20:22:08.000000000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 60 bytes
Capture Length: 60 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:icmp:data]
Ethernet II, Src: AlertonT_01:58:29 (00:60:2d:01:58:29), Dst: d8:9d:67:77:72:e0 (d8:9d:67:77:72:e0)
Destination: d8:9d:67:77:72:e0 (d8:9d:67:77:72:e0)
Source: AlertonT_01:58:29 (00:60:2d:01:58:29)
Type: IP (0x0800)
Internet Protocol, Src: 136.241.77.10 (136.241.77.10), Dst: 136.241.76.11 (136.241.76.11)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
Total Length: 46
Identification: 0x0ffb (4091)
Flags: 0x00
Fragment offset: 0
Time to live: 64
Protocol: ICMP (0x01)
Header checksum: 0xbfdc [correct]
Source: 136.241.77.10 (136.241.77.10)
Destination: 136.241.76.11 (136.241.76.11)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Code: 0 ()
Checksum: 0xfd37 [correct]
Identifier: 0x0069
Sequence number: 25550 (0x63ce)
Data (18 bytes)
0000 50 49 4e 47 00 00 00 00 00 00 00 00 00 00 00 00 PING............
0010 00 00 ..
Data: 50494E470000000000000000000000000000
[Length: 18]


The device is violating the ICMP spec. Length:18 is incorrect here. It should be length 5 based on the request and then have "Padding: 13" in the IP portion of the response. I can also see you are using a TCPIP offload card (checksum error in the sending frame, checksum of 0 means "let the card do it."
 
Last edited:

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Finally had a chance to run this on my sniffing environment. Here is the Length 5 response coming back from a Cisco router. As you can see there is 13 extra junk chars in there to pad out the frame but the Length is still 5 and the data is "abcde."

Code:
No.     Time           Source                Destination           Protocol Length Info
    372 4.981779000    192.168.20.251        192.168.22.157        ICMP     60     Echo (ping) reply    id=0x0001, seq=7203/8988, ttl=255 (request in 371)

Frame 372: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
    Interface id: 0 (\Device\NPF_{EAFD43E2-756F-4DCF-9334-F10C0C7AB96F})
    Encapsulation type: Ethernet (1)
    Arrival Time: Jul 27, 2015 14:38:03.948575000 Central Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1438025883.948575000 seconds
    [Time delta from previous captured frame: 0.000399000 seconds]
    [Time delta from previous displayed frame: 0.000399000 seconds]
    [Time since reference or first frame: 4.981779000 seconds]
    Frame Number: 372
    Frame Length: 60 bytes (480 bits)
    Capture Length: 60 bytes (480 bits)
    [Frame is marked: True]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: Cisco_24:e5:80 (88:1d:fc:24:e5:80), Dst: Dell_61:5b:2e (d4:be:d9:61:5b:2e)
    Destination: Dell_61:5b:2e (d4:be:d9:61:5b:2e)
        Address: Dell_61:5b:2e (d4:be:d9:61:5b:2e)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: Cisco_24:e5:80 (88:1d:fc:24:e5:80)
        Address: Cisco_24:e5:80 (88:1d:fc:24:e5:80)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IP (0x0800)
    Padding: aa636465aa636465aa636465aa
Internet Protocol Version 4, Src: 192.168.20.251 (192.168.20.251), Dst: 192.168.22.157 (192.168.22.157)
    Version: 4
    Header Length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 33
    Identification: 0x4a3f (19007)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 255
    Protocol: ICMP (1)
    Header checksum: 0xc4b3 [validation disabled]
        [Good: False]
        [Bad: False]
    Source: 192.168.20.251 (192.168.20.251)
    Destination: 192.168.22.157 (192.168.22.157)
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xba14 [correct]
    Identifier (BE): 1 (0x0001)
    Identifier (LE): 256 (0x0100)
    Sequence number (BE): 7203 (0x1c23)
    Sequence number (LE): 8988 (0x231c)
    [Request frame: 371]
    [Response time: 0.399 ms]
    Data (5 bytes)
        Data: 6162636465
        [Length: 5]

0000  d4 be d9 61 5b 2e 88 1d fc 24 e5 80 08 00 45 00   ...a[....$....E.
0010  00 21 4a 3f 00 00 ff 01 c4 b3 c0 a8 14 fb c0 a8   .!J?............
0020  16 9d 00 00 ba 14 00 01 1c 23 61 62 63 64 65 aa   .........#abcde.
0030  63 64 65 aa 63 64 65 aa 63 64 65 aa               cde.cde.cde.
 

Gryz

Golden Member
Aug 28, 2010
1,551
204
106
Agree with imagoon. I got little time this week to look into this. But the key is the Total-Length field. It should be set to whatever length the packet should be, even if it's shorter than 46 (46 + 18 = 64). The cisco set the total-length field to 33, while your device sets it to 46. This indicates that the device adds the padding to the ip-packet or the icmp-packet. It shouldn't do that. It should add the padding to the ethernet frame.