T1 & Cisco Issues - Serial Up, Line Protocol Down

[randal]

I just had a T1 installed out to the boondocks of Colorado. It is a 18 mile point to point link, going from our NOC to ICG through Qwest to a shed in a field.

Physical layout is:
Cisco 4500/s2--8' cable--Osicom DSU/CSU--ICG--Qworst's stuff--NIU-Adtran TSU100--8' cable--Cisco 2501/s0

The DSU/CSUs synch up just fine, and I can loop the ends up/down and run QRS / all 1s / any other testing pattern without fail. ICG has done the same, end to end, and the line is clean. The NOC DSU/CSU is set to Internal timing, the Shed set to network timing; both are ESF/B8ZS. Both Cisco's are set to HDLC, keepalive at default 10sec, with IP addressing set correctly (NOC xxx.xxx.246.1 / Shed xxx.xxx.246.2, both 255.255.255.252).

The 4500 reads Serial2 Up, Line protocol is down, as well as:
DCD=up DSR=up DTR=up RTS=up CTS=up

which means that everything should be working fine. I have no idea what the other side says, as it is very far away, and our telephone line has not been installed -- however, the router left the NOC working in a back-to-back configuration, and nothing has changed AFAIK.

The cisco starts out UP/UP, but dies after the 10 second keepalive. I debug int serial on the 4500 and watched for keepalive & hdlc info, and it sends out keepalives, but gets nothing back from the Shed, and goes to UP/DOWN after 10 seconds. Turning off keepalive makes it go to UP/UP permanently, but it does not pass traffic.

No amount of shut/noshut and loop up/down (local & remote) will make the routers talk.

I want to say that the Shed router is somehow fubar'd, and will check that in the morning. Anything else I should look for while I'm out there?

Thanks,
randal

 

[spidey07]

You've got a common problem. If you are ABSOLUTELY SURE your clocking/csu are good, meaning you've looped both ends and are sucessfull from a csu-csu point.

Then there are only a few reasons why you would get a serial up/down:

1) up means you have good carrier and CTS, but are not receiving HDLC keepalives from the remote end.
2) generally this could come from a framing/linecode error, but you've ruled that out.
3) Check both sides of the serial link for layer2 errors (show int) and layer 1 errors (csu, show controller/service module)

my guess is the far side doesn't have carrier/CTS, that's why you're not getting hellos from it. check ckt provisioning as well.

good luck, these can be tricky especially when you carrier says "we've looped both sides and run bert test, the ckt's good"

"DUUDE!!!! I DON'T HAVE A CARRIER, NOR DO I HAVE A CLOCK, what planet are you on and will you please take both smart jacks out of loop!!!!"

seriously, ask them to send a delatch signal to both ends.

good luck...pm me and I can talk you through it tomorrow morning, EST.

 

[ScottMac]

Since you had the routers working back-to-back in your Lab, is it possible that you forgot to change the timing to "loop" or "recover clock" ?

Unless your carrier is supplying your with dark copper (not very likely), you'll want to set the clock on both ends to get clock from the loop / cloud. Normally, if you had a clock / sync issue, you'd have alarms up the wazoo (or at least some functionality with some "slippage"). if the unit on your end is trying to supply clock, you may be getting an error that looks like something else.

Check your clock source settings, make both sides DTE, get your clock from the loop.

(you probably already know this, but also verify ESF, B8ZS ... these are the most common)

Also, if you're using UTP (Category anything) cabling from the smartjack to the CSU, and there is some distance involved, there's a good chance that it's the cabling. A T1 line *- By Spec - * uses "premises cabling:" two individually shielded pairs inside another shielded jacket. Most T1 CSUs or CSU/DSUs will operate fine with UTP, but under some circumstances (distance extremes, weak signal, deaf receivers, marginal impedences, collective tolerances, etc) the UTP just isn't good enough. Sometimes you can compensate enough by adjusting your Line Build-Out ("LBO", closer to zero is a stronger signal).

If this is NOT a full T1 (Fractional T1) make sure your channel count and channel assignments match at both ends (probably so, since it worked back-to-back).

Also do a "sh CDP nieghbor." CDP will frequently work, even though the line is showing down because it uses a proprietary signalling / framing protocol ... that'll at least show whether you have honest-to-gawd connectivity between the two points.... just some parametric issues or perhaps some other problem.

Also check to see if maybe your config is incorrect for the line type . You may have back-to-backed for a channelized (or unchannelized) T1 (real T1), but the circuit that was delivered as a PRI - basically the same thing, but you may need to add ISDN Switch-type National-XXXX (XXX could be 5ess, dms100, national / NI2 ....). The PRI will also need to have a serial "D" channel assignment (i.e., S0/1:23).

That's it off the top of my head, we get calls like this all the time ... these are frequent mis-configurations.

Good Luck

Scott

 

[randal]

Thank you very much for your knowledgable and prompt replies. First thing I'm going to look at tomorrow is making sure that both sides are set to network timing; I don't think there is really a clocking issue because I can test from the NOC csu/dsu to the far end and it comes back without error. I'll make sure to run the QRS/1s tests from the Shed when I am out there tomorrow morning.

Spidey, I ran into the looped smartjacks issue this afternoon when I couldn't get the 4500 to go out of up/up(looped); I called the provider and had them drop the Shed smartjack loop, and now I can run my tests without fail, but the routers won't synch up. I want to say that there is something wrong at the Shed.

At the Shed, it comes in as premise cabling into the smartjack, then goes about 3 feet on cat5 into the dsu/csu -- just enough to penetrate the wall. On the NOC side, which is about 7 floors from the ICG NOC, it is dropped on standard 50-pair, crosspunched and then ran into our server room on cat5 -- I don't think cabling is an issue, nor LBO as the CSUs can see each other (or at least think they can see eachother).

It is a full T1, not channelized, and AFAIK is not provisioned as a PRI. It's a really generic, straight up T1 line -- nothing special about it in the lab, nothing special about it in the real world; just normal T1 installation fun

As mentioned, first step is to verify network/loop timing on both ends and that the routers are set to dte (I haven't set a clock rate, so they should be dte already), then look at the router configs. I'm pretty sure the Shed has DCD, but DSR and CTS were down if I recall. The CDP idea is a good one, and I'll definitely give that a whirl.

Thanks again for the replies -- at least I'm armed with some ideas for the morning -- I'll definitely post how it turns out, either in frustration or asking for some more pointers.
randal

 

[randal]

Hey guys,
Just got out here to the Shed and looked at WTF was going on, and as I have learned from both experience and the numerous preachings here, I checked the cables first.

Turns out the v.35 cable was in the wrong port on the dsu/csu. Moved it to the correct port on the DSU/CSU and it came right up, and I'm posting from the T1 as we speak.

Thanks for all the help guys!
randal

 

[astralnaut]

Whew. Wipes sweat off brow thinking sold router was DOA.

 

[ScottMac]

Troubleshooting 101:

Is it plugged in?
Is it turned on?
Are you currently experiencing a power failure?
(and so on ...)

Glad to hear it was something simple!

Good Luck

Scott

 

[Garion]

The classic answer for this kind of problem. ALWAYS investigate the physical layer first and you'll be right about 80% of the time.

- G

 

[spidey07]

Originally posted by: Garion
The classic answer for this kind of problem. ALWAYS investigate the physical layer first and you'll be right about 80% of the time.

- G

Spideyism - "Don't fvck with the physical layer"

 

[KC5AV]

there are only 3 types of problems in networking.
1. cables
2. cables
3. cables

 

[ScottMac]

HAH!

They're only a problem when neglected, ignored, abused, or otherwise out of spec.

Be nice to your physical layer and it'll be nice to you .....

JM.02

Scott