muffed up network stories - tell 'em here.

[spidey07]

Seems like everybody has a story of a network that was just FUBARRED. Something so unusual as to make a man ask GOD to come down and fix it.

I'll start -
Office kept having trouble with a hub based ethernet network. about 100 nodes on 5 hubs in a central closet. Armed with a trusty sniffer I'm hired to check things out. All of a sudden the line is nothing but garbage, decoded as random hex in a constant stream. Well of course this causes everybody's computer to freeze and loose a connection to the lone NW3.12 server. What the heck is going on????? I check cable, I check hubs, I replace hubs. Same symptom, at random, any traffic load, just garbage for anywhere from 30 seconds to 5 minutes.

One day right as it happened I noticed somebody had JUST hit the copy button on the copy machine rather loudly. "hey can you hang out there and press that button when I ask please?". Sure enough, when the copy machine was used the network turned to garbage. It was a rather lonely copy machine and only got used a dozen times a day.

Turns out that all the cable runs were in the wall behind the copier, all nice and coiled up.

I'll never forget that one.
Cost of one week network analysis = $4500
Rolling the copier four feet = priceless

 

[Tallgeese]

You mean:

* The backbone chassis we had to periodically visit to reseat all the blades since the damn things were located in broom closets and were corroding from the cleaning agent fumes. Our "stepladders" were the rims of the over-sized sinks.

* The hub located BEHIND and one inch away from an 8 inch steam pipe. Next on "You Scald Your Face," a man attempts to read status LEDs with a pocket mirror, while standing on top of a filing cabinet, and trying desperately to ignore a professor having a shouting match with a TA!

* The e-mail gateway which was on a lamp timer (fer Pete's sake) to reboot it daily. And the user who set up a test e-mail that was sent every 15 minutes (I sh!t you not), and who let us know like clockwork EVERY %@$#%!& DAY the moment the gateway went down and when it came back up.

* The computer lab using 1 wire for every two computers, with jacket stripped and wires untwisted to the tune of about 2 feet. Specs for the room (I looked up the old proposal) proclaimed "Only CAT5-compliant components."

*The Apple IIes and IBM PS2s (that's plural, folks) that were in PRIMARY service in networked computer labs in 1995. If you've never installed and configured a MicroChannel NIC, then you haven't LIVED, folks!

* My jack@$$ boss who, when building our new, razzle-dazzle, dedicated server room (in the CEO's words "we want this to be a showcase") for our new corporate office, instructed the CFO (his boss) that there was "no need for a ceiling in the server room." AS400, telecom equipment, data racks (open and closed) all on a tile floor. His answer, when questioned, was "Well we don't have one in this (a remote) server room." His response when shown ceiling in question: "Well, whaddya know?" I simply requistioned earplugs and a case of Scotch.

* Switching an area to a new fiber backbone (in a relatively new job) and finding out your predecessor had used the run as a pull string (through two floors, NOT in CONDUIT) on a particularly bad day nearly six moths hence. And after the whole *streeeetch* *BREAK* BPF moment, your predecessor decided not to admit anything, and subsequently not have any other fiber terminated AT ALL in that area to conceal the fact.


Scariest part:
First 5 are at a major university (charter member of Internet 2).

 

[neuralfx]

<< * The computer lab using 1 wire for every two computers, with jacket stripped and wires untwisted to the tune of about 2 feet. Specs for the room (I looked up the old proposal) proclaimed "Only CAT5-compliant components >>



I saw this at a school .. it's ridiculous .. i think there are like "School Networking Seminars" where they promote these ridiculous ideas, along with the use of Novell .. heh .,.
-neural

 

[ProviaFan]

None of my network stories compare to yours at all, but I recall the time when I was trying to set up a LAN with another computer, and it would show connect lights and transmit lights and all that crap, but the computers could not ping, browse the network neighborhood, or anything. It turned out that I just needed to add the NIC to the "local zone" in ZoneAlarm, but I only found this out several months after trying to set up the LAN (and subsequently giving up on it).

 

[CBuxton]

School district computer lab. The previous administrator had setup a 35 computer lab by chaining together 8 - Five port 10Mbps hubs. Things would connect sometimes, usually not. When traffic would pick up things would slow to a crawl. One day I stuck my head up above the drop in ceiling to find the ethernet cables wrapped around the power line conduits and then a big bunch of wires pass BEHIND the 200amp sevice box. Little did I know that I would find this same sort of thing all over the school district. We also had a "core" switch consisting of an 8 port 3Com switch AND we had 1 server for the whole system (2000 users).

 

[ProviaFan]

<< School district computer lab. The previous administrator had setup a 35 computer lab by chaining together 8 - Five port 10Mbps hubs. Things would connect sometimes, usually not. When traffic would pick up things would slow to a crawl. One day I stuck my head up above the drop in ceiling to find the ethernet cables wrapped around the power line conduits and then a big bunch of wires pass BEHIND the 200amp sevice box. Little did I know that I would find this same sort of thing all over the school district. We also had a "core" switch consisting of an 8 port 3Com switch AND we had 1 server for the whole system (2000 users). >>


Replace "school district" with "church," and the networking terms with sound equipment tips, and you have an excellent description of the sound system that I work on at church. Oh, wait, sorry, this thread is only about networking stuff.

 

[spidey07]

<< chaining together 8 - Five port 10Mbps hubs >>


bwahahahahahahahaaha

yep. Or the times you walk into a 150 person office only to discover 14 (count 'em 14) linksys switches all connected togther and not able to move a single frame. Damn thing lit up like a christmas tree even when the office was empty.

 

[Garion]

Some of my worst ones.. Mostly at the physical/facilities level.

Having my whole data center go dark after a power failure when some idiot accidentally thew the breaker for the transfer switch to the generator weeks earlier. The UPS kicked in as it was supposed to and we all raced in, just to see run out and *click* go dark.

Another "friend" decided to run a test on the transfer switch (different data center) to see if it was still working. At 10:00 AM. ZZZZT! POP! Nope, it's not working and we found out the hard way that we had a bad cell in the UPS, taking the whole DC down hard.

Yet another rocket scientist decided to install a new switch trunked to our core data center switches (which pretty much run one of the biggest banks in the US) and set his switch to "server" in the STP domain. Nuked all the VLANS and took us down for four hours. Not only that, but he did it AGAIN that afternoon. Gotta love outsourcing.

A guy plugging in a Linksys router into my data center segment with Proxy arp enabl... Err. Wait a second.. That was a "virtual experience" *grin*

Having one segment network shut down for a few minutes every morning- Solved by moving the cables in the ceiling AWAY from the flourescent lights.

One of my techs tripping on a fiber cable and taking out the backbone - Always exciting. A few months later, we put in a fully-meshed fiber network, needless to say.

At my old job, our chillers (the "outside" part of a big data center cooling system) were EXTREMELY sensitive to power fluctuation. One power spike and they turned themselves off. Big problem, as we were aboput 100' away from a power substation that sucked in the lightning and spiked us all the time. Probably had 100+ degree temps in there at least a dozen times over two years, likely cutting the MBTF on the equipment (about $10 million in there) in more than half.

- G

 

[Mucman]

I would add something, but the goof-ball actions have been performed by yours truly, and I do not need any more laughs going in this direction

 

[BlitzRommel]

Here's one I've had recently.

One of the networks I work on (Notice I didn't say "monitor" for obvious reasons... It never operates properly.) has 24 PCs, mostly 486/DX's at 33MHz (AST Bravo/33 or something like that), a few Pentiums are in there. They're in six rows of four in a large classroom. At the end of each row, is a standard school desk (You know, with the opening on a side and the rest is enclosed metal/wood) that the school has these huge-ass 24-port 10B-T Baystack managed hubs. Keep in mind, these are ALL donations.
Our school (Catholic grade school) is so poor, I had trouble talking the head nun into buying Y adapters so children can use the computers and listen to the sound together.

Anyway, no one watches the room when I'm not working there. So what do the kids do? They screw with the computers' settings, plug mice/keyboards into different slots (Yeah, no AT keyboards! All PS/2) and just cause general mayhem. Well last week, ALL of the mouse balls from the mice were taken out. I figured, no real big deal, we got them back and the kid probably got the whoopin of his life from his parents. Damn kids.

What's worse though, is that the kids were unplugging/replugging the power to the hubs and eventually, they all just died. Just like that. Needless to say, I was awfully pissed. We eventually found someone that donated a buncha teeny Encore 8-port hubs and I daisy-chained the six together just like with the rackmount 24-ports (lol...) and it worked fine SOMETIMES. Sometimes, stuff just can't connect. So it's taken me a week to explain to the principal why things are becoming really unreliable whenever the kids try to print or whatever. The DHCP server then took a crap from something. Heh, just random events.

Anything our school buys for my computers is from donated Cambell Soup labels. 'Tards.

 

[BlitzRommel]

<< Replace "school district" with "church," and the networking terms with sound equipment tips, and you have an excellent description of the sound system that I work on at church. Oh, wait, sorry, this thread is only about networking stuff. >>



Trust me, I SO feel your pain. We should get together and swap "war stories"