Problems with my T1 line w/thunderstorms - help!

[scsi stud]

I own an eCommerce business and I have had my T1 line for about 3 years now. The line runs perfectly smooth all year around, except when summertime hits -- as soon as we get a thunderstorm, my line goes down. With the recent onslaught of thunderstorms in the Washington DC area, my line has been down about 6 times the past week. Phone calls to my ISP (Netifice) are useless, they keep escalating the issue but the local carrier (the devil named Verizon) don't take a look at the line to see where the problem is occuring. They keep bringing the line back up but don't fix the physical problem. It's as if I have a flat tire, they keep plugging the holes but they don't replace the tire.

Can anyone provide any sort of insight as to why my line goes down when there are thunderstorms? During the winter when we get lots of precipitation (snow and rain), the line is still silky smooth. Only during electrical storms is when this happens.

Any help would be tremendously appreciated. Thanks.

 

[spidey07]

ah, the joys of dealing with a flaky T1.

Gather all your ticket number (problem tickets) and send them to whoever your contact is with the ISP. Then just say you want them to run a new replacement T1 and you're not paying for it.

Flaky T1s can be very difficult to get resolved because of the various pieces involved with them. If you can make the case of it being down 6 times they should work with you to get it resolved but you have to keep pressing and escalating. Log all your tickets and outage times because you probably have some sort of SLA that guarantees uptime.

what you probably have is bad cabling or a T1 card in their (LEC) gear. If it only happens during eletrical storms that would point even stronger to cabling/installation.

-edit- Have them send a tech out to inspect your end or the "demac" You can also request intrusive testing and that may help them pinpoint where the problem is.

 

[jersiq]

You need to be careful on how you ask a prove to the demarc, they can and will charge you for maintenance/dispatch.
Request your ISP open a chronic ticket with verizon, and escalate the ticket yourself with verizon.
I would give you my contacts, but they are for the wireless provider group only.

Faulty office repeaters, doublers getting wet....as spidey said there is a lot that can go potentially wrong with a T1. In my office we have 5 OC-48 backhauled from the entire city of Houston from one provider alone, and on any given day we have at least one T1 outage.

But your problem is definitely chronic. Either your ISP isn't yelling loud enough at VZ, or you may want them to step up their "current yelling volume"

 

[ScottMac]

Sounds like a "Wet Pair" issue.
The ground (vault/pedastal water level) just needs to get real wet for this to occur.

I don't know for sure, but I'm betting Verizon has most of their service force in "Emergency Mode" for the Pensylvania flood areas and has re-directed at least some of the field force from surrounding areas to keep their stuff as intact as possible.

This, of course isn't helping you much, and as Spidey says, track your outages, your downtime, and your SLA so that , at the least, you can probably get some flavor of rebate against your SLA.

Some time in the future, you can probably request that the pairs be swapped for others with hopefully positive results.

Good Luck

Scott

 

[cmetz]

scsi stud, ask your ISP to run a remote loopback pattern test on your line for 24 hours. This means you WILL be down for 24 hours, but will hopefully document the problem.

Push the SLA angle. $$ credits wake people up.

Verizon is currently a bit busy in the DC area. While their T1 support does normally suck big time, right now I expect that they are simply overwhelmed and actively trying not to get too deeply involved in problems if they can avoid it. This is one of those times when the relationship between your ISP and Verizon matters - Verizon plays favorites in a huge way when prioritizing resources to problems, and also having folks inside VZ to call matters a lot.

If you own an eCommerce business, I hope that all your customer-facing stuff is coloated, and not at your site. If that's not currently the case, let this be a good reason why you need to do that. Connectivity to random office or home locations is never going to be as high quality, or get fixed as fast, as you get with a good colocation facility.

Also, I've never heard of the ISP Netifice, and that tells me you are probably not working with a good one. A good ISP costs more, but it's times like these that the difference shines through. A lot of smaller/cheaper ISPs just don't have the capability to really troubleshoot lines, and even if they do, they don't have the contacts and credibility to get Verizon to believe them.

PS: do you get any crackling on your POTS lines during electrical storms, esp. when lightning strikes?

 

[bruceb]

You could also ask Verizon to install your T1 on a Fiber Optic Line
You might need to pay for the Fiber Install and a small Fujitsu Mux,
but in the end, you will have a T1 line that will almost never go down
unless the fiber fails or your power goes out .. for that you install
backup power ... also with the fiber run installed, you now have
expansion capability .... say perhaps to a full T3 system which
can handle 28 T1 Systems .... 24 channels per T1

 

[scsi stud]

Thanks for the replies, it is greatly appreciated. Netifice is actually pretty large from what I know. They recently merged with MegaPath so that could be a reason why things are so hectic around there right now.

I host my eCommerce stuff locally at my store because our system (inventory, accounting, order management) is all live (3 stores connected together) and it is ideal to be local instead of colocating it.

I just got a phone call back saying that the field techs from both Verizon and the ILEC (what is an ILEC?) are coming on-site Monday morning because their results are conflicting. Is there anything I should point out besides doublers getting wet?

Cmetz: what do you mean by POTS lines?

Also, if they still do not resolve my issue, I want to have a Plan B option (switch providers). Any recommendations for good prices/service/reliability (low latency, 60ms or thereabouts) in the Northern VA/Washington DC area?

Thanks again for all the help.

 

[Fallen Kell]

Originally posted by: scsi stud

Cmetz: what do you mean by POTS lines?

POTS = Plain Old Telephone Service and is sometimes interchanged with PSTN (Public Switched Telephone Network)

 

[jersiq]

Usually it won't matter if you switch providers, the last mile will still be owned by VZ.
There are some circumstances where there MAY be two providers to the last mile.
What yur current ISP does is to lease the line at a wholesale rate from VZ, and resell it to you. If there is a problem with the T-1 line itself, changing providers will most likely give you the same line, and the same problems.

 

[ScottMac]

ILEC = Incumbent Local Exchange Carrier = "The Guys That Own (most of / all of) the Local Infrastructure"

They are the primary carrier in your area.

Compared to the Competitive Local Exchange Carrier ==CLEC = "The Guys That Purchase Wholesale From the ILEC and Resell it as their Own"

SO what they told you was that your carrier (CLEC) and Verizon (the ILEC) are coming out to resolve the finger pointing. It is likely that they will have to take your connection down for testing back to the Central Office and (probably) the Frame-Relay switching cloud.

Initially, I'd expect them to just jack into the NIU/SmartJack and pull the "PM Stats" (operational statistics, as seen by the NIU/SmartJack) ... this is usually non-invasive (invasive = connection down, like a loop-up).

If the PM Stats don't provide any significant/specific error info, then they'll probably want to put an analyzer in-line and run some stress tests back to the CO and / or the FR switch cloud (done with the cooperation of the CO techs (ILEC) and or FR cloud techs (CLEC) and with them on the phone ... it's frequently a coordinated test).

If they *still* don't find errors, they may ask to test your inside cabling between the NIU/SmartJack and the router/(CSU/DSU).

Something ought to pop up somewhere by this point, then they can fix it.

If something doesn't show by then, then they'll either blame your equipment, say it's all in your head (j/k ;-) ), or they'll start swapping things "on suspicion" ... probably starting with your "pair," then the NIU/SmartJack, then the OCU (Office Channel Unit - the card that drives the line from the CO), then maybe a DACS card or port, then maybe a FR switch port or card.

It's gonna be a hoot. Try to enjoy the experience.

Good Luck

Scott