Metro Optical Ethernet troubles, please help - FIXED!

[spidey07]

Originally posted by: randal

Originally posted by: cmetz
randal, have you tried hooking two servers directly to the 100Mb/s Ethernet handoffs (assuming they're copper) and running netperf? That would help determine whether the 7200s are introducing problems, and should be able to help you fight with the telco if that test fails.

Also ensure that the telco's duplex settings are right. I have similar circuits from Verizon, though 10Mb/s, and a recent problem I had with those showed me that their default config on their gear is pretty bogus.

Yes. I hooked up the MOE delivery directly to the "ra" server in Denver and my laptop here in COS and had the same throughput issues, host to host. I did -not- however, do any looking for OOO packets, retrans, etc. just pure throughput. I will do this again today, host-to-host without any intermediary switches/routers.

well then that seems to rule out any of your gear, sounds like the provider has a provisioning/setup problem.

 

[randal]

The new server is formatting. It has multiple ethernet interfaces, so one side will have a routable live IP (for remote management) and the other side will have 10.10.10.10/24 nonroutable for the MOE.

New diagram: http://isis.data102.com/~randal/moedrawing2.gif

This should separate the networks 100%. Please note that the new machine "Ammen" has -NO- gateway services on it. Also, please note that I do not have 10.10.10.0/24 in any routers routing table. That will separate things quite a bit and make for cleaner testing.

edit - RE: carrier problem, I think so too, but I haven't done any tcp tracing or deep inspection in such a scenario besides simply measuring throughput, hence the new server testbed.

 

[bruceb]

Randal .. I understand how they deliver it .. I used to install these circuits
You can trust my on the fact that a Framing Problem will cause trouble
for your thoughput .. No matter what the end handoff is, there is always
framing on both the User Level Facility and also on any higher Facility such
as an OC48 or OC96 that you circuit rides on ... you may be the First ckt
installed on a new Optical Ring .. if so, the telco may have all sorts of
problems and never know it

Does the Telco Handoff to you on a Fiber Optic Jack ? ?
Did you put a Fiber Optic Light Metet on it and measure both the
Light Level and the Laser Frequency ? ? ... How does it compare
to the Good Direction ? ? ? ... It could be nothing more than a badly
done Fiber Splice or a Bad End Connector on the Fiber Strand

 

[randal]

The handoff to me is ethernet on cat5. Whether or not -they- have framing issues, I do not know, nor am I able to inspect it.

The "first user on a new circuit" holds a LOT of weight, as this thing took almost 3 months to get installed due to "capacity issues", as indicated in this email below, dated 07/11/2006 ... and we just got it installed end of August, a month later!

Randal,
Due to some problems we have run into regarding the long haul capacity, we have to push the due date for this circuit out to 07-28-06 Our ops techs and outside plant personnel are working to get this up as soon as possible, if we are able to turn it up sooner, I will let you know. Please let me know if there are any questions.

The LEC just got purchased by Level(3), and I am concerned that this is a turnup on the new Level3 fiber instead of their own, and things are totally screwed up.

 

[randal]

I finished testing from server-to-server with no intermediary devices, and this is the netstat info from the RECEIVING host in denver (5 minute test run, counters cleared right before):

ra# netstat -s -s -p tcp
tcp:
24908 packets sent
12 data packets (1136 bytes)
7 data packets (2288 bytes) retransmitted
19890 ack-only packets (2472 delayed)
4998 window update packets
1 control packet

27373 packets received
10 acks (for 1138 bytes)
1 duplicate ack
14922 packets (21597232 bytes) received in-sequence
1 completely duplicate packet (48 bytes)
12439 out-of-order packets (18011400 bytes)

1 connection accept
1 connection established (including accepts)
1 connection closed (including 0 drops)
10 segments updated rtt (of 9 attempts)
7 retransmit timeouts
14921 correct data packet header predictions
1 syncache entrie added
1 completed
9964 SACK options (SACK blocks) sent
ra#

>>>>>>>>>>>>>>>


In looking at the tcpdump of that 5-minute test, I am seeing lots and lots of fast retransmits and retransmissions and tons of duplicate ACKS (aka out of order), but ethereal won't highlight anything as Out of Order. Either way, the circuit is trash. If somebody wants to take a look at the .cap file and confirm, I'd be glad to link it.

edit: bolded the important things

 

[bruceb]

I will keep an eye on this thread .. I am very curious as to the end cause.
It would be nice to know if what I suspect is the case, a problem with
the Telco Facility is really the cause

 

[spidey07]

Originally posted by: bruceb
I will keep an eye on this thread .. I am very curious as to the end cause.
It would be nice to know if what I suspect is the case, a problem with
the Telco Facility is really the cause

i'm inclined to agree. framing in the mux.

-edit- and with all those retrasmissions and out of order packets tcp is spending all of it's time trying to maintain sequencing/windowing/acking. I bet if you look at your window size for that connection it will be very small - under 8 KB. This hurts your performance even more. TCP can't really "ramp up" and you're faced with the low throughput.

 

[randal]

hmm, well, the program I am using is iperf - a ttcp derivative - and it reports that the Denver machines (recv) TCP window size is 64.0KB, and the COS (xmit) TCP windows is 32.5KB.

edit: I misread your reply. Yes, I agree that the window size has got to be getting really tiny while it is moving traffic, killing throughput./edit




When configured with 8KB windows throughput goes down to ~650kbps, with 128KB it goes up to ~1.1mbps ...

>>>>>>>>>>>>>>>>>>>>>>>

ra# iperf -s -w8k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[ 4] local 10.10.10.20 port 5001 connected with 10.10.10.10 port 57079
[ 4] 0.0-10.4 sec 824 KBytes 652 Kbits/sec
ra# iperf -s -w128K
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[ 4] local 10.10.10.20 port 5001 connected with 10.10.10.10 port 64954
[ 4] 0.0-11.1 sec 1.38 MBytes 1.04 Mbits/sec

===================================================
===================================================

ammen# iperf -c 10.10.10.20 -w8k
------------------------------------------------------------
Client connecting to 10.10.10.20, TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[ 3] local 10.10.10.10 port 57079 connected with 10.10.10.20 port 5001
[ 3] 0.0-10.1 sec 824 KBytes 665 Kbits/sec
ammen# iperf -c 10.10.10.20 -w128K
------------------------------------------------------------
Client connecting to 10.10.10.20, TCP port 5001
TCP window size: 129 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
[ 3] local 10.10.10.10 port 64954 connected with 10.10.10.20 port 5001
[ 3] 0.0-10.0 sec 1.38 MBytes 1.16 Mbits/sec

 

[Garion]

Do you have root on these boxes to take a tcpdump or a snoop? That might be the most useful thing to show what's actually happening at the network layer. Try and snoop one of those tests and save it as a .cap file and pull it up in Ethereal. There's a way to build a filter to colorize TCP retransmits - Put one of those on and see if you have a problem (or stick it up on a FTP site somewhere and post the link here or PM it to a few of us to look at it..)

- G

 

[randal]

This is my capture file on the RECEIVING DENVER server from earlier today (1:20pm MST) while I went and got lunch. 5 minutes of throughput testing, then another 4 minutes or so of idle. I took a look at it in ethereal already, which is where I got the retrans/fast-retrans/dup-acks info. Please disregard the ssh traffic

http://isis.data102.com/~randal/1320.cap.gz ~1.1MB compressed

 

[randal]

The layer 3 tech has finally returned from vacation, and amidst catching up gave me a ring. They understand what I'm saying (thank God) and are going to take a deeper look at it tomorrow. Hopefully they'll find out what's up and let me know -- as soon as I hear something, I'll post it here.

 

[ScottMac]

Have you checked / replaced the cabling between the handoff switch and your distribution (or, on your direct connection)? Both sides?

Bad or poorly terminated cabling can cause all of these problems ... and the part about our switch / host / server falling back to half duplex is a pretty good symptom in that regard.

Aside from that, it almost sounds like a bad Virtual Tributary (VT) mapping to the OC, that would be a cause for your out-of-order frames/packets.

FWIW

Scott

 

[spidey07]

scott,

So you think the mapping is off? I don't get into the deep mappings of the muxes, so care to enlighten? The metro/ethernet over sonet is pretty new to me and I'd like to understand exactly how it works. It's not a POS interface, it's something else. Something in the card that takes the frames and puts/addresses it on the ring.

 

[randal]

Well, I have an answer, and it is my problem. Well, it is Cisco's problem. There are a couple fault points - the first being that our head to head tests weren't, thanks to some superb work by the guys at our Denver POP. When I finally did get a real head-to-head test, everything was clean. After lots of troubleshooting this weekend, the issue is this:

>>>>>>>>>>>>>>>

Cisco Bug ID: CSCea56745
Bug Details

Deferred frame counted, traffic delayed when duplex set as full

Product 2950 Model 2950-24
Component hardware Duplicate of
Severity 2 Severity help Status Resolved Status help
First Found-in Version 12.1(13)EA1 All affected versions First
Fixed-in Version 12.1(13)EA1b, 12.1(14)EA1 Version help
Release Notes

[Symptom]
When interface configured its duplex as full, deferred counter increments, then packets through that link are delayed. When traffic is ping, some ICMP hit timeout. When traffic is ftp, too many retrasmissions cause low troughput.

[Condition]
12.1(13)EA1
interface duplex configured as full duplex.
Symptom can be seen both 10/full and 100/full.
Symptom not seen when 100/half, 10/half, and autonegotiation.

[Workaround]
-use autonegotiation
or
-use previous release image.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

The fix? Then the Denver 2950 from Full/100 to auto. Works great since. Upgrading the switch as we speak.

Thanks for everybody's input on this maddening issue - I guess sometimes you can't overlook buggy software, no matter how much you think the hardware is working.

 

[spidey07]

doh!

That's a pretty nasty bug. Thanks for the heads up.

See, I told you it was a speed/duplex/negotiation issue.

 

[bruceb]

I am glad you finally got it fixed.
And until you ran a real head to head test
framing issues could not be ruled out.
Hope Cisco will fix the bug in the next software
release