Multiple cable drops fail at the same time

[alyarb]

so you have some machines that are perfect and the others that have 2% loss, or now all of them are a showing packet loss? bring your laptop or some trusted NIC down to one of the workstations with the new cabling and see if you still get loss. then make sure it's not the switch. i imagine NICs and/or switch are easier to replace than patch panel.

 

[imagoon]

Sounds like your boss doesn't actually want the issue fixed. That is like the shop manager telling your generator repair guy that he is not allowed to have a multimeter. If he continues to work on he just wastes more time. I would point that out to him. He is going to pay for not renting a meter in wasted time and database corruption.

PS: you may want to stick something like Wireshark on a machine (server maybe) and sniff the network. It will show you if you have a ton of runts and damaged packets and may even be able to sniff out the MAC address if a card is bad.

 

[Mogadon]

Considering the fact that you've now run three new drops you (your boss) should by default certify that these are functioning at the correct spec and you would need a scanner to be able to do this. At least that's the policy i've always worked with.

Most people don't have the kind of testing hardware that you require lying around. If you happen to find someone that does chances are they'd be a competitor of yours and i'm not sure you're going to want them on site, especially considering what has transpired recently.

 

[Fardringle]

you sure you didn't have a single device that wasn't surge protected - think laserjet - maybe that powerstrip wasn't a surge suppressor - let a spike get into the network?

also floating grounds - check every UPS and power strip for ground faults - you can cause all heck to break loose if you have a single ground fault

All computers have at least a surge protector. Some have UPS batteries as well. Quite a few of the surge protectors are old and don't have ground fault indicators so I'd have to manually replace/test them. I might have to do that eventually but it will have to wait.

So what happens if you cross-connect pairs of computers with new patch cables? Just two computers hooked together with a trusted switch and a couple of new cables? Do you still get packet losses?

Maybe you've got a NIC or two that are misbehaving and messing up the whole network?

I connected a few of the machines to a new switch away from the rest of the network during my initial troubleshooting and got the same results (good computers stayed good, bad ones stayed bad). A single bad NIC could very well be causing the problem. I know for sure that the NIC in the server was damaged and I already replaced that one but there could be another on the network that is still causing the problems I am seeing now.

so you have some machines that are perfect and the others that have 2% loss, or now all of them are a showing packet loss? bring your laptop or some trusted NIC down to one of the workstations with the new cabling and see if you still get loss. then make sure it's not the switch. i imagine NICs and/or switch are easier to replace than patch panel.

Yes, some machines still seem to be working perfectly, but a few of the ones that were previously perfect are now getting a small amount of packet loss as well. A known good laptop with a good patch cable got the same results at the "bad" locations that I tested as the computers in those locations. This is why I suspected a cabling issue from the beginning. It does seem to be at least partially a cabling problem since the really bad stations are at least partially working now that I ran new wires to them from the patch panel.

Sounds like your boss doesn't actually want the issue fixed. That is like the shop manager telling your generator repair guy that he is not allowed to have a multimeter. If he continues to work on he just wastes more time. I would point that out to him. He is going to pay for not renting a meter in wasted time and database corruption.

PS: you may want to stick something like Wireshark on a machine (server maybe) and sniff the network. It will show you if you have a ton of runts and damaged packets and may even be able to sniff out the MAC address if a card is bad.

He wants it fixed, but he's an insurance guy and he doesn't want to take responsibility for an expensive machine that he doesn't own. It's silly, but that's the reasoning he gave.

After properly grounding the equipment (see below) I ran Wireshark on the server since almost all traffic goes to or through the server but I'm not sure how to interpret the results. I initially ran it for 10 minutes to get some basic results and plan to let it run non-stop for the rest of the day to get a better idea of what is happening. There aren't many people in the office today so only a few computers are creating network traffic. However, all of those active machines produced at least one Malformed Packet error, including two that are not experiencing any apparent connection problems. Also, one computer had repeated connection problems throughout the test period even though the Wireshark report only listed a single packet error for that computer.

Considering the fact that you've now run three new drops you (your boss) should by default certify that these are functioning at the correct spec and you would need a scanner to be able to do this. At least that's the policy i've always worked with.

Most people don't have the kind of testing hardware that you require lying around. If you happen to find someone that does chances are they'd be a competitor of yours and i'm not sure you're going to want them on site, especially considering what has transpired recently.

I finally got an electrician to check the grounding in the server room. It was very inadequate (just a ground wire from the closest electrical socket), so we had him drill holes in the floor and run ground rods into the soil beneath the building. Now that they are properly grounded, I'm using Wireshark to attempt to determine if the remaining connectivity problems are just being caused by a rogue NIC or if there is a more significant problem that will convince the boss to let me rent/hire a Fluke scanner for proper testing and certification.

 

[westom]

I finally got an electrician to check the grounding in the server room. It was very inadequate (just a ground wire from the closest electrical socket), so we had him drill holes in the floor and run ground rods into the soil beneath the building. Now that they are properly grounded,

You are confusing grounds. For example, if all grounds are same, then motherboard ground is same as chassis ground is same as wall receptacle safety ground is same as breaker box neutral bus bar is same as earth ground. Therefore you can earth a lightning rod to the motherboard and had lightning protection.

Wall receptacle in a computer must not be connected to earth ground rods. That is a complete code violation. And that does not solve your problem.

Earth ground is the only ground that is relevant. And every wire in every cable entering the building must first connect short (ie 'less than 10 feet') to that single point earth ground. Either directly (cable TV, satellite dish). Or via a protector (telephone, AC electric).

From here on, assume the blackout was preceded by a surge. Surges are electric current. That means it has an incoming path and an outgoing path to earth. Otherwise the surge does not go through or damage that item.

You have one network card damaged. Therefore that card had an incoming and outgoing path from frame (box, chassis) to network cable. Now you have identified one path that a surge current took.

Based upon what the electrician implied, you let a surge into the building. Once inside, a surge will hunt for earth ground destructively. Either that surge must be earthed before entering the building. Or you have surge damage no matter how many frames you 'safety ground'. Yes, safety ground is not earth ground. Only earth ground is relevant.

Well, you did not explain what that router was connected to. For example, a surge incoming on AC mains may have found earth via the 1 inch conduit, easily arced across to network cables, then through that router to some earth ground path.

One excellent path to earth ground is the telephone wires. Why? Because all phone lines already have an earthed whole house protector where the telco's wires connect to yours. Just another fact you may need to know to understand why you had damage, to find what might be damaged, and to avoid all future damage.

Ok. Just because a network card is working does not mean it is working properly. Surges can degrade or overstress parts. Then shotgunning (replace a part to see if it works) only results in confusion, myths, and frustration.

Moving on to more facts. A best diagnostic is provided by NIC manufacturers. If numerous NICs are from the same NIC manufacturer, then superior diagnostics is the last test. One NIC outputs a constant (worst case diagnostic) test signal. All others echo that signal. Then the computers (running the NIC manufacturer's diagnostic and no Windows) reports actual threshold numbers. The only limitation - all NICs in that test must be from the same NIC manufacturer.

That NIC test first confirms all functions inside the NIC are actually working (yes a defective card can still conduct data). Then the last and optional test tests that NIC interface to other NICs. A poor man's solution to no Ethernet scope.

You pulled cables. But if arcing had causes a failure, you must strip the insulation from those cables to find the arc point. A reason why informed techs have sophisticated test equipment.

Now, on to a solution. Nothing in the computer room will provide effective surge protection. All that talk about rack grounding may, at best, only cause a slight improvement or add just a little more 'robustness'. As stated earlier, every wire in every cable must make that 'less than 10 foot' connection to earth ground where all wires enter the building - at the service entrance. And that earth ground (that only you - building owner - are responsible for providing) determines whether the surge dissipated harmlessly outside the building. Or when hunting for earth destructively via appliances.

Those plug-in protectors only give surges more potentially destructive paths through the network. We did the same analysis. All computers were powered off. Two had plug-in protectors. Protectors too close to electronics and too far from earth ground earthed that surge destructively through adjacent computers. Now the surge has an incoming path (protector bypassing protection in the computer's power supply) to motherboard. And an outgoing path via the network. We literally replaced semiconductors to trace that destructive path to earth via a third computer (all computers separate in the building). The best path to earth was via the third computer's modem, down phone line, to earth ground via the telco's 'whole house' protector.

Provides are some concepts. If you had surge damage, then you had some wire entering the building without earthing (directly or via a 'whole house' protector). That surge energy was hunting for earth destructively via that network card and other items. Your symptoms suggest the surge may have arced inside those cables - maybe from conduit through cable, through router, to earth.

To better answer your question, well you must describe all wires in the building. Because averting or tracing surge damage means tracing every possible wire, duct, conduit, and pipe. But this is apparent. Your single point earth ground is deficient. And you apparently do not have the always required 'whole house' protector on AC mains with a short (ie 'less than 10 foot') connection to earth. And you may have made the problem even worse if you earth grounded that wall receptacle.

Protection is about single point earth ground. If you added earth ground elsewhere in the building, then you made surge damage more likely. Again, protection is always - always - about where energy dissipates. Either energy dissipates harmlessly outside the building. Or energy hunts for earth destructively inside the building - with or without plug-in protectors. Your protector is only as effective as its earth ground. Your damage is what happens when that energy (current) goes hunting for earth destructively inside the building.

Suspect or replace any part that might have been in a surge path that you already know was through that one network interface. One reason you might be confused. You are not thinking that the surge is a current that always has both an incoming and outgoing path. A current that increases voltage as necessary to blow through anything that might stop it.

And yes, it one of two connections is now somehow shorted to safety ground, then signals start acting strange or intermittent as other building appliances power on or off.

Just a few introductory concepts – and yes, I am not kidding. I have only introduced some concepts relevant to your problem.

 

[imagoon]

You earth grounded the racks? Typically you ground them to the building super structure.

 

[Fardringle]

You earth grounded the racks? Typically you ground them to the building super structure.

Just the patch panel for now, but yes. There's absolutely nothing in that room (or the rest of the building, as far as we can tell) that can be used for grounding except for the grounding wire/pin in the electrical sockets, which is the way it was set up by whoever arranged things before I took over (they had the wire rack that the server sits on grounded directly to the ground pin of the electrical socket). According to the owner, there is no metal in the building construction at all. All LAN and phone conduit is PVC and the electrical wiring has no conduit of any kind. The building frame is wood with drywall over it and it is all sitting on top of a solid concrete slab with no rebar inside. Other than earth grounding, what option would you use? Grounding directly to the wiring in the electrical wall socket? I could be wrong, but that doesn't sound like a good idea to me. If the electrical wiring had metal conduit I suppose we could ground to that, but there isn't any.

 

[imagoon]

Just the patch panel for now, but yes. There's absolutely nothing in that room (or the rest of the building, as far as we can tell) that can be used for grounding except for the grounding wire/pin in the electrical sockets, which is the way it was set up by whoever arranged things before I took over (they had the wire rack that the server sits on grounded directly to the ground pin of the electrical socket). According to the owner, there is no metal in the building construction at all. All LAN and phone conduit is PVC and the electrical wiring has no conduit of any kind. The building frame is wood with drywall over it and it is all sitting on top of a solid concrete slab with no rebar inside. Other than earth grounding, what option would you use? Grounding directly to the wiring in the electrical wall socket? I could be wrong, but that doesn't sound like a good idea to me. If the electrical wiring had metal conduit I suppose we could ground to that, but there isn't any.

Normally you run a 4 or a 2 back to the service entrance ground.

You can get voltage readings between different points in the earth. That current will flow through the easiest path which could be the grounding paths.

 

[westom]

Other than earth grounding, what option would you use? Grounding directly to the wiring in the electrical wall socket? I could be wrong, but that doesn't sound like a good idea to me.

That is exactly what you do - the best idea - because it is required by code, necessary for reliable operation, and essential to human safety.

How do telcos all over the world earth their equipment? Protectors and earth ground are up to 50 meters separated from electronics - so that superior protection exists. Yes, that separation between earth ground and electronics is necessary to properly earth your equipment. Only earth ground that must exist (so that your equipment is not damage) is an earth ground that must be single digit feet from the breaker box. That is the single point earth ground AND the only earth ground for the entire building.

Your new earth ground has made damage easier. And violated code. And does not properly ground the equipment.

Appreciate a concept called ground loop. If all audio equipment does not connect to a single point ground, then a hum exists - a ground loop. Also what you must do inside the server room to avoid ground loops and other reliability problems. Not earth ground. Your equipment must connect to the single point safety ground - which is also the third prong on every wall receptacle in that room. The fact that grounding is so badly performed (and is now worse) explains why surge damage would exist.

Did you go out and inspect the critical earth ground? That is the first task performed when someone has surge damage.

BTW, if you did not read my previously post at least three times, then you still did not get it. You will not start comprehending it until at least the second reading because it is obviously all that new to you. And because some contradicts myths that must first be untaught. Unteaching myths makes learning even harder. That post is chock full of information you need to know for solving your problem AND for averting future damage. And some 'poor man' solutions that are useful when the boss is a cheapskate.

Eliminate the sever room earth ground. That violates code and makes future surge problems easier.

 

[Fardringle]

Normally you run a 4 or a 2 back to the service entrance ground.

You can get voltage readings between different points in the earth. That current will flow through the easiest path which could be the grounding paths.

That sounds reasonable, but since there isn't any conduit for the electrical wiring, we would have to literally tear the walls apart from one end of the building to the other to run the wire or to run it in a phone/LAN conduit to the closest wall port to the electrical service entrance and then drill a hole through the outer wall from there to the electrical box. Either that or run it across the floor/ceiling out in the open and the boss said no way to both...

 

[Fardringle]

I read your first post multiple times, westom. I appreciate anyone that is trying to help but frankly your post contradicts itself multiple times (you say in some spots that ONLY earth ground can be used and then say elsewhere to never use earth ground), and you referred several times to things that either simply don't make sense (what is a "wall receptacle in a computer"?) or go against what people I have come to trust - Spidey and Imagoon - have said about the need to ground patch panels and racks. You also said that any device possibly in the path of the surge must be replaced, which implies that every single device in the entire office has to go since the problem appears to have gone through the network wiring so everything is suspect.

I honestly wasn't going to reply at all because I don't want to discourage any attempts to help figure this out, and I'm sure you believe that you know what you are doing, but there wasn't anything at all useful to me in that post except the suggestion to run diagnostics on all of the individual NICs in every computer. I have actually started doing that but there are multiple different brands and chipsets so I unfortunately can't just run one mass test between all of the machines at the same time.

Having said that, your newest post is much clearer and makes more sense. I understand that it would be best to connect the server room equipment to the outside earth ground but there's no possible way to do that without tearing the walls apart (not going to happen), which is why the electrician said that he put the ground rods in the server room. I did not inspect the outside wiring since I was not there at the time but he said that he did and it appears to be fine so it's actually possible that the surge originated inside the building - possibly from the server NIC that failed - and not from the electrical wiring coming into the building.

Since you said to do exactly what this electrician (and my reading) said not to do and ground the equipment directly to the ground pin in the electrical outlet, I got curious and called a few other electricians (random selections from the phone book) to do an impromptu survey. They all said that it is a violation of state building code to connect non-electrical equipment such as patch panels and metal server racks directly to the ground pin in a wall outlet and to only use building infrastructure (conduit or metal building frame) or a direct connection to the outside ground point (as Imagoon said), or a direct to earth ground in the physical location of the equipment if the other two are not available. I suppose it's entirely possible that the codes are different for your location but that's the information I was given by multiple different people who know the codes for this location.

 

[Fardringle]

Third post in a row, shame on me..


I was actually hoping that an extended scan with Wireshark would tell me something useful but the results don't seem very helpful at all. I ran it for about five hours (new file every 10 minutes so the files didn't get too big) yesterday and again today and got anywhere from a small handful to as many as 100 malformed packet errors in each file. The errors are scattered all over the network and don't seem to show any pattern at all. However, there was one very odd result that I think is worth mentioning. While I actually ran Wireshark on the server in the office, I monitored and controlled it from my own PC outside the office through a VPN connection on the server, and during one 10-minute period there were well over 100 malformed packet errors with my computer's IP address attached, which seems to indicate that those errors were either generated by the server, by the router that connects the ISP to the office network, or were corrupted by the network switches.

I thought that I had eliminated the switches as a possible cause early in my troubleshooting by connecting a few computers directly to a new switch isolated from the rest of the network where they still had connectivity problems, but I ran the Intel Proset diagnostic tests on the NICs in the router and on the new NIC in the server and they all passed with no errors. The boss said that he is willing to spend the money to buy some new switches (HP Procurve, maybe?) if I can prove for sure that the switches are the problem. My initial test with the separate switch actually pointed away from the switches being the problem, but that was before I replaced the known bad wiring so it could have been an unrelated issue. I guess my next step is to finish running diagnostics on the individual NIC cards to make sure they all pass. I really wish I could just talk the owner into letting me rent a Fluke (or hire someone to bring one to his office) to test things properly, but it's going to be hard enough to get him to cough up the money to pay for new good switches if it turns out that they are needed.

 

[imagoon]

That sounds reasonable, but since there isn't any conduit for the electrical wiring, we would have to literally tear the walls apart from one end of the building to the other to run the wire or to run it in a phone/LAN conduit to the closest wall port to the electrical service entrance and then drill a hole through the outer wall from there to the electrical box. Either that or run it across the floor/ceiling out in the open and the boss said no way to both...

Honestly just be careful. A near by lightning strike could light your network up like the Fourth of July. I dunno, I would need to ask my electrician at the office to give a good response.

--EDIT--

In response to your switch post. If you have the ability, grab a laptop and connect it to the server. Run one of those network benchmarking programs that simply sends packets (name escapes me at the moment). Set wireshark to record and see if the malformed packets go away. You can install wireshark on the laptop and use it as a poor man's tester for the NICs. It won't do the job as elegantly as a real tester but it might help. It also takes the switches out of the loop. If the issues vanish during the machine <-> laptop tests it may indicate the switches have zombied.

 

[Fardringle]

That's a good idea. I've been trying to figure out how to do that without interrupting the people who are trying to work but I may just have to kick them off and let them grumble for a while or make arrangements to go in when the office is closed so nobody will be there.

 

[Fardringle]

Following imagoon's suggestion, I connected a laptop directly to the server's NIC using a Cat 6 patch cable with nothing else attached to either machine and ran Wireshark on the laptop while using LAN Bench to saturate the connection with packets. In the 600K+ packets generated during a 30 second test, Wireshark reported a few out of order and retransmitted packet warnings, but absolutely zero true errors of any kind. I ran the same test several more times for 30 seconds each (Wireshark seems to get mad with more than 600K packets so I limited to that size on each test) and still got zero errors on every test.

I then ran the same test with the server connected to its Gigabit port on the switch (cheap D-Link switch with 2 GB ports and 24 10/100 ports) and the laptop connected directly to the other Gigabit port on the switch. The test actually ran successfully twice but on the third run all network traffic stopped completely in the middle of the test. It seemed like the LAN Bench program had simply locked up so I restarted both machines and tried again and again all network activity stopped in the middle of the test.

Workstations on the network lost their access to the server at this time but were still able to access the Internet through the Smoothwall router plugged into one of the other switch ports so I connected the server and the test laptop to two empty 10/100 ports on the switch and tried the Wireshark/LAN Bench test again. It was significantly slower since they were no longer connected at Gigabit speeds, but the test ran multiple times with no errors. To be thorough, I plugged the server back in the Gigabit port on the switch and it initially connected but lost connectivity again very quickly just with normal LAN traffic so it appears that on top of everything else (failed server NIC and failed wiring to multiple stations) the Gigabit controller in the switch also bit the dust, but was only failing intermittently before I overloaded it with the LAN Bench test and killed it completely. Since it was the server's port that was failing, this also explains why my previous Wireshark test showed what seemed to be random errors from every computer on the network.

Now that I have confirmed that at least one of the switches needs to be replaced (they have a second 24-port switch with just 10/100 ports) I'm going to run similar tests on the second switch to see if it is also failing.

I know that I want to buy a new Gigabit switch so that the computers that have Gigabit cards can take advantage of it. I'm leaning towards the HP Procurve 1810G-24 based on multiple recommendations here in the past. So far, the best price I can find is $325 from Buy.com through Amazon. If I need to replace both switches then I'll probably just get two 24-port switches and link them together since the cheapest 48-port Procurve I can find is over $1200 and that's definitely beyond the budget that the boss said that he can handle. He already cringed at the idea of $650 for two good 1810g's but said that we can do that if we need to but can't go much higher. They don't actually need the full 48 ports, but do use close to 40 so I either need a good 48-port switch in that budget range or the two 1810g's. They are working temporarily with the server plugged in to a spare 100mb port but I'd like to get that switch replaced as soon as possible in case it starts to fail even more than just the Gigabit controller for the two "high speed" ports.

 

[imagoon]

Did the network wide "weirdness" vanish with the server @ 100mbps?

 

[Fardringle]

There are only a handful of people here right now so I'm not sure about all of the computers, but everyone here says that they are working normally now with the server connected to a 100mpbs port instead of the gigabit port. I ran a 10 minute "general" Wireshark capture on the server with this connection and it did not report any malformed packets. I also ran the 30 second LAN Bench test on a couple of the computers that are connected to the secondary switch and with the laptop plugged directly into that switch and all passed with no errors. I can't know for certain without more testing, but it does seem like all 100mpbs ports on both switches are working OK with the server on a 100mbps switch port as well which would seem to indicate that I only need to replace the one switch. I might try to talk the boss into replacing both just to get them on real business switches instead of their cheap D-Link stuff, but I'll have to talk to him and see what he says.


edit: It's strange to have a pretty solid confirmation of a switch problem when I tested with a separate switch originally and had the same problems, but I suspect that might have been due to the bad wiring in the walls that I replaced since I did have two of those really bad stations connected to the spare switch when I did that testing.

 

[westom]

I read your first post multiple times, westom. I appreciate anyone that is trying to help but frankly your post contradicts itself multiple times (you say in some spots that ONLY earth ground can be used and then say elsewhere to never use earth ground), and you referred several times to things that either simply don't make sense (what is a "wall receptacle in a computer"?) or go against what people I have come to trust

You are trusting what scams and myths have told you. If what you have hearsd contradicts what is posted, then either 1) you were told a lies, 2) you did not understand what they said, 3) you completely misread what I posted, or 4) you still don't understand why your 'contradictions' are really saying the same thing.

For example, did you see the expression "single point earth ground"? It is an almost biblical expression for your solution. You MUST have a single point earth ground at the breaker box. And no earth ground in the computer room.

BTW, electricians typically do not understand this. Electricians are taught what must connect to what for human safety. Learning that can take years. Only the few and better electricians also know why code required these things. And rarely understand additional requirements for surge protection. For example, radio engineers and ham radio operators would understand this because impedance (that electricians need not learn) is relevant to exceeding what is required by code. The electrician typically knows what is done to meet code. Surge protection is about exceeding code requirements.

Another example of why you might be confused. That wall receptacle safety ground is not earth ground. A third prong on an AC receptacle safety grounds the rack.

Server room is earthed via the wall receptacle and AC power safety ground. Not to earth anywhere in the computer (server) room. Everything in the server room is or is not earth depending on what exists at the service entrance (breaker box). If directly earthing anything in the computer room, then future damage is encouraged &#8211;a single point earth ground no longer exists anywhere in the building. To be earthed, computer room wires must be distant from the earth ground (located 'less than 10 feet' from the breaker box). Do no wiring anywhere inside the building (assuming your server room is powered from a three wire circuit as required by code). See again the part about how telcos have no surge damage? Protector and earth ground are up to 50 meters distant from their 'server room'. You must do same. Probably not 50 meters. But the point - to be earthed, the server room must be distant from the single point earth ground.

Any wire that enters your building must enter within feet of that earth ground so that it can connect to that earth ground. Every wire inside every cable must connect to earth ground before entering the building. Connect short (ie 'less than 10 feet') either directly (cable TV) or make that short connection via a protector (AC electric, telephone).

The figure in this application note demonstrates same. Even underground wires must be earthed before entering. In this app note are two structures. Each must have its own single point earth ground. Any wire that enters or leaves either structure must first connect to single point earth ground - directly or via a protector:
http://www.erico.com/public/library/fep/technotes/tncr002.pdf

Things like "safety ground vs earth ground", single point earth ground, 'less than 10 feet', up to 50 meters, "every incoming wire inside every cable", etc - these are critical important details. If you did not understand the significance of those details, then you will see contradictions.

What do telcos do? Where the wire connects to earth and where the wire connects to your server room electronics .... up to 50 meters separation. That distance increases protection. Expanding earth ground (and again, the only earth ground must be the single point earth ground) means protection inside the server room is enhanced. Yes, what is done farther away is important.

Appreciated a concept. You must view the entire building to have protection in that server room. Anything you do only inside the server room can even make damage easier.

Also inspect your primary surge protection. It this critical component in the primary protection layer is missing, then your server room is at risk:
http://www.tvtower.com/fpl.html
Another example of how far away solutions are best located - and you don't rewire anything inside the building. That was averting future damage.

Back to solving the current problem. Based upon what you have posted, well, the problem can be computer hardware layer, computer link layer (yes there is a difference), cable to wall, cable from wall through conduit, cable from conduit to router, hardware layer inside the router, and link layer inside the router. (Ground is not part of this analysis. Ground is about averting future problems.) For example, using the NIC manufacturer's diagnostics to talk between two computers (that must have the same NIC to have the same diagnostic) can be a powerful tool.

If nothing else, buy two NICs from the same (major) manufacturer that also provides those comprehensive diagnostics. (Some less responsible NIC manufacturers do not provide comprehensive diagnostics.) NICs installed in two computers (as even a second NIC port), now creates a diagnostic tool. (Only better NIC manufacturers provide those diagnostics).

Now view the suspect list. Execute diagnostics by simply connecting the two computers directly with a 'cross-over' ethernet cable (also get two RJ-45 female to female adaptors - another useful tool). Cable and female RJ-45 adaptors available and any better computer store that can also provide those NICs. IOW the cable (with wires inside the cable crossed) connects directly from one computer network port to another - no router or anything else. Run the diagnostic. Now you are only testing one thing - NIC hardware to NIC hardware. Simple test creates a benchmark. A test that works because two computers talk to one another using worst cable data packets, only using hardware layer and not even using link layer.

Next connect those same computers via conduit ethernet cables. IOW, in the server room, jumper two cables with the female RJ-45 ethernet adaptors and cross over cable. Now let the two computers talk directly to each other as before - without any router or anything else connected.

This second test adds only one suspect the test - cables in conduit. That is the principle. In each test, only add one more suspect. For example, the next test replaces the crossover cable with only a router that connects to nothing else but those two computers. Then next test only connects the network cable to a third computer - because one cable can act bad when the problem is created by another cable attached to the same router. (This sentence relevant to your router failure.) Step by step until something becomes unstable. If you do not do this, then your only useful solution must do something similar - a step by step addition or subtraction of only one suspect until things start working or stop working. Each step tested by that worst case data pattern from the NIC diagnostic (or something that also creates worst case data patterns such as a serious Ethernet cable tester). There is no better way to find and eliminate an intermittent.

Windows masks, works around, or compensates for failures - as it should. Windows as a diagnostic tool is inferior; cannot even differentiate between network hardware interface and network link interface. If you do not know the difference, then you are only that much more confused trying to eliminate the problem - which is why diagnostics are so critically important to solve problems faster and the first time.

Without a serious ethernet cable tester, oscilloscope, or something equivalent, then your only choice is shotgunning. Keep replacing good part until something works. Your Wireshark test seems to report useful facts. But again, follow the procedure of only adding one suspect at a time. The router test must first be with no other cables (as noted earlier). Then only add one or a few cables at a time. Under diagnostics, no router should reorder the packets unless the packet is being resent due to hardware failure (what Windows and other hardware do to work around problem so that you do not know that problem exists).


An AC three prong plug into the wall receptacle means the chassis / rack connects directly to wall safety ground. Nothing I posted says you wire that connection directly. Or, if the rack is hardwired, then safety ground wire inside the conduit does the same grounding. Conduit also does grounding BUT is not considered reliable. If the rack is powered, then that rack MUST be connected to a safety ground wire that enters the room with other AC power wires. That is necessary for human safety and required by code. No, you do not connect the rack with a wire to the wall receptacle screw. That third prong connects the rack to the wall receptacle safety ground.

Good reasons why it is called safety ground. Code calls it equipment ground. Equipment ground and earth ground are different. All equipment must connect to safety (equipment) ground. Equipment (safety) ground connects to earth ground only at the breaker box.

This post written and then modified before and after new information posted.

 

[spidey07]

Westom, that goes against EIA/TIA 568 (or is it 569? The communication grounding one) standards and best practices for data centers (I work with a few RCDDs). Everything in the data center should be grounded to a single large copper bus bar that goes to the main earth ground including patch panels/racks as well as your anti-noise mesh in the floor.

 

[westom]

Westom, that goes against EIA/TIA 568 (or is it 569? The communication grounding one) standards and best practices for data centers

Correct if the room has been constructed so that the room is literally an isolated entity within the building. The OP does not have that kind of building.

Every protection layer is defined by it earth ground. Defined was the primary protection layer. Then then all wires enter the building - connect to a secondary protection layer at that single point ground, service entrance, and breaker box. In your scenario, the comptuer room is literally an isolated building inside the building. AC is wired differently so that the room has its own breaker box (wired by code so that it can be earthed). Every overhead light and wall receptacle carefully wired only to that breaker box. Even the floor is isolated from the building; grounded to a separated computer room ground that is isolated from the rest of the building - except at one point. Even air ducts and plumbing inside walls must be integrated into the computer room earth ground. Every wire enters the room to be first earthed to that computer room's single point ground - even the bus bar which creates one big single point ground in that room.

His ducts, wires, floor, lights, conduit, plumbing, etc is not constructed to meet those stiff requirements. In his case, every telephone wire in every telephone cable can only be earthed where that wire enters the building. Only place the plumbing can be earthed is to the same breaker box and earth ground. His only earth ground must be at the building's service entrance.

BTW Qwest discusses this further:
> Oftentimes, outside cabling enters through conduit. Where possible, this conduit
> should be non-metallic and fire-retardant and comply with the requirements of NEC
> Articles 362 and 800. Metallic conduit offers a path for transients to enter the building,
> and we do not want this, even if the conduit is properly grounded.

Essential to protection is the earth ground that defines each layer. If anything enters a protected space without first connecting to that earth ground, then the entire protection layer has been compromised. Worse, sometimes that mistake can make damage easier.

So that direct lightning strikes do not cause damage, essential is single point earth ground for each layer. This necessary so that even a nearby struck tree does not become a direct lightning strike to bulding electronics.

Much of what is discussed is also defined by Polyphaser's highly regarded application notes. How a nearby strike becomes a direct lightning strike to building electronics and why single point earthing must be carefully implemented is describes in this app note:
http://tinyurl.com/38v2dv
> Lightning strikes somewhere across the street close to the below grade West cable
> vault. ...
> The first line of defense is the telco protection panel, but the panel must be connected
> to a low resistance / inductance ground. There was no adequate ground available
> in the telephone room.

 

[RebateMonger]

Much of what is discussed is also defined by Polyphaser's highly regarded application notes. How a nearby strike becomes a direct lightning strike to building electronics and why single point earthing must be carefully implemented is describes in this app note:
http://tinyurl.com/38v2dv

That link goes to a place on PolyPhaser's site that isn't accessible without a username/password.

 

[westom]

That link goes to a place on PolyPhaser's site that isn't accessible without a username/password.

Some IT droid must learn that constantly changing URLs does not attract customers. Polyphaser's app notes have long been highly regarded – despite their IT guy.

New URL is:
http://www.polyphaser.com/techdocs/telco and dataline.pdf

Or find "Shielded Data Cables and Protectors" at:
http://www.polyphaser.com/technical_notes.aspx

This app note demonstrates why nearby lightning strikes are actually direct lightning strikes to electronics if a single point ground is not properly implemented. GPR also explains why that computer room ground could even make surge damage easier.

 

[Fardringle]

I have done some more searching and found the HP Procurve 2510-48 and 2610-48 (not 2510g or 2610g) switches for prices that fit the budget but I can't find any reviews on either of those models. Would I be better off sticking with two of the 1810g-24 switches? I don't expect that they will ever really need the features of the managed switches but I definitely want to get something stable and reliable.

 

[Emulex]

i probably have a used 48 port 10/100 with a couple of gigabit uplinks like a 2824

i use the 2510G-48 myself - it is a great workhorse. highly recommended. I keep the 1810G-24 for a backup.

Keep in mind there are fans on the larger models - and lack of redundant power - so you may want two switches anyways.

 

[Fardringle]

I looked at the 2510G-48 as an option but the cheapest price I can find for that one is over $1200 and the owner says that $700 is pretty much the limit that I can spend, which is why I was looking at two 1810G-24 switches ($330 each) or the 2510-48 ($430) or 2610-48 ($680). About half of the computers in this office actually have gigabit NICs so I'd like to get a switch that supports the faster speed for those machines so I'm leaning toward the 1810G-24 in that regard. Low power usage and no fans in the 1810G are nice bonuses as well. However, if the 2510-48 or 2610-48 would be a more reliable unit then that would be a good choice even though they don't support gigabit speeds. I just can't find any reviews for either of those models so I'd like to know if anyone has any experience with them.