Cell delineation issues on Cisco router

[kt]

Once in a while we get this message on the Cisco router and we would lose connection.

07:02:43: %ATM_AIM-5-CELL_ALARM_UP: Interface ATM0/1 lost cell delineation.
07:02:45: %ATM_AIM-5-ACTIVE_LINK_DOWN: Interface ATM0/1 of IMA Group ATM0/IMA1 i
s now inactive.

There are a total of 4 T1 circuits in the IMA group and when any one of them gets the message above the connection goes down. Each time it goes down, I have to reset the IMA interface to restore the connection. Any ideas what is causing all these cell delineations? We've had the circuit running for years without any issues. Only recently has it been acting up like this almost 3-4 times a day.

Any help is appreciated.

 

[spidey07]

Sounds like physical layer problems. Check clocking and framing on both ends.

 

[kt]

Sounds like physical layer problems. Check clocking and framing on both ends.

The other end is AT&T. It takes an army to get them to do anything.

Clocking configuration on our end:

network-clock-participate wic 0
network-clock-participate wic 1
network-clock-select 1 T1 0/0
network-clock-select 2 T1 0/1
network-clock-select 3 T1 0/2
network-clock-select 4 T1 0/3
!
controller T1 0/0
mode atm aim 0
framing esf
clock source line primary
linecode b8zs
!
controller T1 0/1
mode atm aim 0
framing esf
linecode b8zs
!
controller T1 0/2
mode atm aim 0
framing esf
linecode b8zs
!
controller T1 0/3
mode atm aim 0
framing esf
linecode b8zs
!

We've had this configuration for years. Why would it suddenly start breaking? Can I safely assume that AT&T made changes on their end without notifying their customers?

 

[jlazzaro]

unlikely its configuration related...probably a line defect or some other physical layer issue. intrusive line testing, and keep on them 🙂

 

[spidey07]

I can just see the ticket with AT&T now.

"Trouble cleared while testing, no problem found"

Check for errors on your T1.

 

[kt]

unlikely its configuration related...probably a line defect or some other physical layer issue. intrusive line testing, and keep on them 🙂

Line tests came back clean (according to AT&T). They ran an intrusive line test last night while the circuits were down. I've been at this for this whole week without any resolution or explanation why this is happening.

 

[spidey07]

Call cisco then. I searched for that log message and it's coming back on a lot of bugs.

 

[kt]

I can just see the ticket with AT&T now.

"Trouble cleared while testing, no problem found"

Check for errors on your T1.

What kind of errors should I be looking for? I am getting overwhelmed just looking at this:

T1 0/0 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20051006, FPGA: 20, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (7 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 25 15 minute intervals):
     5 Line Code Violations, 8 Path Code Violations,
     1 Slip Secs, 0 Fr Loss Secs, 5 Line Err Secs, 5 Degraded Mins,
     6 Errored Secs, 1 Bursty Err Secs, 0 Severely Err Secs, 35 Unavail Secs
T1 0/1 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20051006, FPGA: 20, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (27 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 25 15 minute intervals):
     0 Line Code Violations, 10798 Path Code Violations,
     2 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     2 Errored Secs, 0 Bursty Err Secs, 6 Severely Err Secs, 118 Unavail Secs
T1 0/2 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20051006, FPGA: 20, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (28 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Line Code Violations, 0 Path Code Violations
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 25 15 minute intervals):
     1 Line Code Violations, 3 Path Code Violations,
     3 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     3 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 33 Unavail Secs
T1 0/3 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Version info Firmware: 20051006, FPGA: 20, spm_count = 0
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  CRC Threshold is 320. Reported from firmware  is 320.
  Data in current interval (31 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 25 15 minute intervals):
     0 Line Code Violations, 2 Path Code Violations,
     2 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     2 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 33 Unavail Secs

 

[jlazzaro]

if you're entitled to support, a TAC case never hurt. there are some bugs related to cell delineation (CSCsv04275)...what platform & IOS are you on?

I would still try another intrusive window with varying patterns; these sporadic issues are rarely apparent to the carrier on the first go-round.

as for the errors, clear your counters and track if any violations, slips, or errors are incrementing.

 

[Cooky]

Besides opening a ticket w/ ATT, you should also contact your Service Executive from your account team. That person will help you push the NOC for resolution.
Since it's been a week, escalate through your account team until it gets resolved.

Looking at your output, I spotted the following for your 0/1 interface:
0 Line Code Violations, 10798 Path Code Violations.

I think line code violations means you have a local physical issue, and path code violation means the errors occurred several hops away (in the carrier network).
If you have these errors after intrusive testing & a clear counter, you need to show these stats to ATT as proof that they need to look at their network.
To be honest I'm not certain if this statement is 100% accurate, but it's worth mentioning to ATT.

 

[Cooky]

We recently had a chronic issue w/ a remote 2X T1 multilink bundle provided by ATT.
After several attempts, the LEC (Verizon) finally figured out it was caused by a burnt-out battery for the MUX on customer premises.

 

[kt]

Thank you guys for all the help. Looks like I will be opening a TAC case with Cisco support since it looks like the IOS version we are on is bugged with the loss of delineation issue.

Cisco IOS Software, C2600 Software (C2600-ADVIPSERVICESK9-M), Version 12.4(9)T,
RELEASE SOFTWARE (fc1)

I don't think our account executive will be of any help. We've been switching away from AT&T as soon as our service contracts expire. SBC was great, but ever since they merged with AT&T it has gone down hill.

Cooky - thanks for the information. I will do that and show the stats to AT&T.

 

[spidey07]

if you're entitled to support, a TAC case never hurt. there are some bugs related to cell delineation (CSCsv04275)...what platform & IOS are you on?

I would still try another intrusive window with varying patterns; these sporadic issues are rarely apparent to the carrier on the first go-round.

as for the errors, clear your counters and track if any violations, slips, or errors are incrementing.

Agreed. Those are some pretty dirty T1s. Notice the unavailable and errored seconds. It has happened plenty of times before where the LEC accidentally changes clocking. Hence my "trouble cleared while testing" comment.

The clock can slip so bad you actually lose framing. Always good to ask them to check provisioning and specify you are NOT providing clock and are getting it from them, which is most always the case unless you have a point-to-point T1.

 

[Pheran]

I agree with spidey - unless those errors are left over from the intrusive testing, those T1s look like crap. Line code violations, path code violations, errored seconds, you've got it all. Clear the counters and see if they are still periodically incrementing.

 

[spidey07]

I agree with spidey - unless those errors are left over from the intrusive testing, those T1s look like crap. Line code violations, path code violations, errored seconds, you've got it all. Clear the counters and see if they are still periodically incrementing.

100 bucks says it's a card in the mux. I can guarantee all those Ts are on the same card. That or clocking.

Don't fuck with the physical layer.

 

[SpunkyJones]

Cisco tac will recommend that you update your IOS to the latest, but that your issue is with the carrier. Case closed!

 

[Emulex]

could be on either end too. took the CO 8 hours to find a flakey card (seriously to find it in their co). that's their problem - they should be proactive to fix it unless they aren't monitoring you

 

[kt]

Just thought I would update this thread. I spent about 4hrs this weekend on the phone with different AT&T support groups and finally got to their IMA group after jumping through multiple hoops. A ticket was opened with the IMA group and they said they would run some tests then give me a call back.

About 3 hours later, I got a call back. They said everything is good and dandy again. No explanation, just that they found the problem and "fixed" it. No configuration changes were made on our Cisco router. Our connection has been stable since that call. So whatever it was, it was on their end but they will never admit to it. 20 hours of my time wasted.

Good job, AT&T. Guess what, our contract is expiring soon.

 

[spidey07]

Cisco tac will recommend that you update your IOS to the latest, but that your issue is with the carrier. Case closed!

True. Very true. If they get some bug hits on it that is their first course of action. Escalate if need be.

 

[spidey07]

Just thought I would update this thread. I spent about 4hrs this weekend on the phone with different AT&T support groups and finally got to their IMA group after jumping through multiple hoops. A ticket was opened with the IMA group and they said they would run some tests then give me a call back.

About 3 hours later, I got a call back. They said everything is good and dandy again. No explanation, just that they found the problem and "fixed" it. No configuration changes were made on our Cisco router. Our connection has been stable since that call. So whatever it was, it was on their end but they will never admit to it. 20 hours of my time wasted.

Good job, AT&T. Guess what, our contract is expiring soon.

Didn't I tell you! "trouble cleared while testing"

Oh, and you'll get exactly the same run around from any other carrier. I've dealt with them all. It's almost always some kind of LEC issue, you just need to keep pushing the ticket. For an issue like this I'd go directly the the AT&T account rep and say "fix it, this is a chronic problem and subject to our service level agreements". The rep can pull some strings.

 

[Cooky]

OP - even though you are switching provider, having had your ATT Service Executive push their NOC for resolution could've saved you some hours and frustration.
That person's job is to make sure customers' issues are taken care of.
It's funny they wouldn't have had to create such a position if the support folks had just done their job at the first place.

 

[kt]

OP - even though you are switching provider, having had your ATT Service Executive push their NOC for resolution could've saved you some hours and frustration.
That person's job is to make sure customers' issues are taken care of.
It's funny they wouldn't have had to create such a position if the support folks had just done their job at the first place.

Our SBC account exec was great. When SBC and AT&T merged, they assigned us a new account exec and everything went down hill from there. He may have got the hint when we stopped renewing our contracts as they expire. So he's not going out of his way to do anything for us, but then again he wasn't doing much before to keep us happy neither.

 

[jlazzaro]

Our SBC account exec was great. When SBC and AT&T merged, they assigned us a new account exec and everything went down hill from there. He may have got the hint when we stopped renewing our contracts as they expire. So he's not going out of his way to do anything for us, but then again he wasn't doing much before to keep us happy neither.

lol, going through the same thing with Sprint. they're great when you want to buy something but non-existent when they realize you're going month-to-month and not planning to renew. hopefully our move to AT&T is not indicative of your experiences 🙂

in reality though, they are all the same. good account management is a crap-shoot nowadays.