Suggestions on MetroE troubleshooting

[kevnich2]

I've been working with an optical based MetroE network. The network consists of 7 spoke locations all with 10mb/s symmetrical and a hub site with 70mb/s aggregate bandwidth. Layer 2 speed test shows everything flowing correctly. Jperf is showing 10mb on the traffic the from hub site to each remote site but only 5mb/s on the traffic from the spoke sites to the hub site. The upload is also being very spotty, sometimes being 5mb/s, other times being 2mb/s where the download is a solid 10mb/s each time.

My first thought was a duplex mismatch somewhere. We looked at every interface, everything is set correctly for duplex.

Anyone have any possible suggestions that I could look at?

 

[drebo]

A lot of times, a MetroE provider will require you to traffic shape your outbound traffic to confirm to your subscribed bitrate. Make sure your traffic shape rules are properly written, as incorrectly written rules can cause what you describe.

 

[kevnich2]

All internal traffic between locations is handled by the carrier's cisco switches, our equipment is only handling traffic going via internet.

Any other suggestions? Would like a couple things I can try to look at.

 

[Emulex]

sounds like over committed bandwidth. you got a CIR that guarantees minimum bitrate? if you don't own the fiber - you probably are sharing it with as many people as they can get to make money.

do you have any route control? BGP4?

 

[kevnich2]

sounds like over committed bandwidth. you got a CIR that guarantees minimum bitrate? if you don't own the fiber - you probably are sharing it with as many people as they can get to make money.

do you have any route control? BGP4?

Yes - we have guaranteed bandwidth of atleast 10mb/s for each site. What's weird is that it pretty much stops at the 5mb/s mark, almost like there's a setting somewhere for the upload from the remotes that they forgot about.

As far as oversubscription - nah. They actually showed us the cabinets and splitters and how many customers are sharing the fiber with us. All nodes are actually 10gbE and they have a max of 32 customers per splitter, depending on the bandwidth that you purchase. We could actually purchase 10gb per location if we wish and in which case, we'd have our own dedicated fiber with no splitter.

We've been working with the vendor for about a week on this trying to figure out where the issue is and I'm trying to think of any possibilities myself to see if we can get it fixed faster.

 

[spidey07]

Duplex mismatch?

 

[macssuck]

I had what soundslike a similar issue with some circuits from ATT. We had 100Mb and 1Gb at the hub site. Traffic shaping is suggested but that still did not resolve the issue for us. In the end ATT had to adjust a setting on the Cisco Metro E switches they provided. I think it had something to do allocating more memory to a buffer.

 

[kevnich2]

Duplex mismatch?

That was actually my first idea but the carrier's engineer's and myself went through every single interface and verified the duplex was correct. The symptoms originally looked exactly like a duplex mismatch.

 

[kevnich2]

I had what soundslike a similar issue with some circuits from ATT. We had 100Mb and 1Gb at the hub site. Traffic shaping is suggested but that still did not resolve the issue for us. In the end ATT had to adjust a setting on the Cisco Metro E switches they provided. I think it had something to do allocating more memory to a buffer.

Hmm - I'll ask them about that and see if they can look at that.

 

[freegeeks]

some shaper / qos profile that is not configured correctly?

 

[kevnich2]

some shaper / qos profile that is not configured correctly?

This is what its looking like. Ran another test last night while working with an engineer with the carrier and had them temporarily kick our bandwidth up. He made a new profile to test with and with the new profile, speed was it should have been. I have a conference call with them tomorrow so we will say how things go.