- Oct 23, 2000
- 9,200
- 765
- 126
I apologize in advance. This will be long, but it has me completely stumped and I want to include as much information as possible in hopes of getting some good suggestions.
The power went out for a short time at one of my client's offices. When the power came back on, nobody was able to access any network resources at all. I determined that the NIC in their domain controller (also acting as DNS and file server) had failed. I replaced that NIC and most of the stations started working again.
This is where it gets odd...
Of the roughly 20 stations that started working after I replaced the server NIC, about half seem to be perfect in that they have no issues and continuous PING spam of 2000+ iterations didn't lose a single packet. I know that perfect PING results don't necessarily mean a truly perfect connection, but for my purposes it was a step in the process of elimination since the other half are losing between 1 and 3% of their packets. This is not a huge problem, but still a concern since they used to be perfect and even a single lost packet can make their office management database program choke. I'll get back to them later since they are working even if they aren't at 100%.
The big problem is that there are 6 workstations that range anywhere from not working at all (can't even get an IP address from the DHCP server and have nearly 100% packet loss with a static IP) to working but only very poorly (20%+ PING packet loss). I used a working laptop (perfect connections at other locations) at all of the bad drops and it failed to connect at all of those drops, and one of the "bad" workstations worked normally at one of the "good" locations, so it doesn't look like an issue with the workstations themselves.
To eliminate the possibility of a bad switch, I took all of the "bad" stations and a few of the "perfect" stations and the server and wired them directly into a new switch isolated from the rest of the network and the bad computers stayed bad and the good ones stayed good.
I thought that somehow one of the modules on the AMP NetConnect 48-port patch panel had gone bad since all six of these stations are wired into the same 6-port module on the panel. However, I rewired two of the stations to a different module where some of the "perfect" stations are connected and the bad stations still had the same connectivity problems. I also wired two of the perfect stations into the presumed bad patch module and those stations still continued to work perfectly.
I also replaced one of the wall ports at one of the bad stations just to make sure the port wasn't bad. It didn't have any effect on the connection.
To me, the only thing left is the cable drops inside the walls. Normally I wouldn't consider that as a possibility with six different connections failing at the same time, but I can't think of anything else that it could be since it's not the server, not the workstations, not the patch panel, not the switches, not the wall ports, and not the patch cables between the workstations and the wall ports. Sadly, I don't have access to a cable tester/verifier so the only way I can eliminate the wall cabling is to replace it. I don't have any spare solid core wire at the moment. I have already ordered a spool of new cable from Monoprice but it won't be delivered until Monday so I wanted to get some ideas from greater minds over the weekend in hopes of resolving the issue in case the new cable doesn't fix the problem.
I wasn't involved when the office was built so I don't know how things are arranged inside the walls, but I do know that each cable drop is enclosed in its own 1 inch conduit with nothing else in the conduit except the standard twine for additional/new pulls. Since I know that each workstation is wired separately to the server room in its own conduit, I can't imagine any way that six different drops could fail at the same time unless the contractors who wired the place made a MAJOR mistake and ran the network conduit directly in contact with exposed or shorted electrical wiring and that electrical wiring overheated and damaged the LAN cables during the power surge/outage. The main flaw in this theory is that even though the bad stations are wired on the same module of the patch panel they aren't particularly close together in the building (they are separate offices along one long wall of the building) and the stations that are getting small amounts of packet loss are scattered all over the office.
I haven't pulled any of the cables out of the conduit yet to check for physical damage since I'm waiting on the delivery of new cable from Monoprice. I'm personally hoping that there isn't any apparent damage since that would mean having a contractor come in and tear the walls apart to get to and correct the problem (the walls are solid wood and plaster and not easily accessible), but I can't think of anything else that could have caused a failure in so many drops at the same time. Also, the fact that many of the still working stations are now getting small numbers of PING packet losses where they were all perfect before the power outage makes me wonder if there is something even more significant wrong with the wiring. Either that or I missed something in my troubleshooting and I'm hoping you can point that out for me..
If you have any ideas, no matter how unlikely, as to what might have caused all of the drops to fail at the same time, or if you think it might be something else causing the problem, I'd appreciate the input since this one has me baffled.
The power went out for a short time at one of my client's offices. When the power came back on, nobody was able to access any network resources at all. I determined that the NIC in their domain controller (also acting as DNS and file server) had failed. I replaced that NIC and most of the stations started working again.
This is where it gets odd...
Of the roughly 20 stations that started working after I replaced the server NIC, about half seem to be perfect in that they have no issues and continuous PING spam of 2000+ iterations didn't lose a single packet. I know that perfect PING results don't necessarily mean a truly perfect connection, but for my purposes it was a step in the process of elimination since the other half are losing between 1 and 3% of their packets. This is not a huge problem, but still a concern since they used to be perfect and even a single lost packet can make their office management database program choke. I'll get back to them later since they are working even if they aren't at 100%.
The big problem is that there are 6 workstations that range anywhere from not working at all (can't even get an IP address from the DHCP server and have nearly 100% packet loss with a static IP) to working but only very poorly (20%+ PING packet loss). I used a working laptop (perfect connections at other locations) at all of the bad drops and it failed to connect at all of those drops, and one of the "bad" workstations worked normally at one of the "good" locations, so it doesn't look like an issue with the workstations themselves.
To eliminate the possibility of a bad switch, I took all of the "bad" stations and a few of the "perfect" stations and the server and wired them directly into a new switch isolated from the rest of the network and the bad computers stayed bad and the good ones stayed good.
I thought that somehow one of the modules on the AMP NetConnect 48-port patch panel had gone bad since all six of these stations are wired into the same 6-port module on the panel. However, I rewired two of the stations to a different module where some of the "perfect" stations are connected and the bad stations still had the same connectivity problems. I also wired two of the perfect stations into the presumed bad patch module and those stations still continued to work perfectly.
I also replaced one of the wall ports at one of the bad stations just to make sure the port wasn't bad. It didn't have any effect on the connection.
To me, the only thing left is the cable drops inside the walls. Normally I wouldn't consider that as a possibility with six different connections failing at the same time, but I can't think of anything else that it could be since it's not the server, not the workstations, not the patch panel, not the switches, not the wall ports, and not the patch cables between the workstations and the wall ports. Sadly, I don't have access to a cable tester/verifier so the only way I can eliminate the wall cabling is to replace it. I don't have any spare solid core wire at the moment. I have already ordered a spool of new cable from Monoprice but it won't be delivered until Monday so I wanted to get some ideas from greater minds over the weekend in hopes of resolving the issue in case the new cable doesn't fix the problem.
I wasn't involved when the office was built so I don't know how things are arranged inside the walls, but I do know that each cable drop is enclosed in its own 1 inch conduit with nothing else in the conduit except the standard twine for additional/new pulls. Since I know that each workstation is wired separately to the server room in its own conduit, I can't imagine any way that six different drops could fail at the same time unless the contractors who wired the place made a MAJOR mistake and ran the network conduit directly in contact with exposed or shorted electrical wiring and that electrical wiring overheated and damaged the LAN cables during the power surge/outage. The main flaw in this theory is that even though the bad stations are wired on the same module of the patch panel they aren't particularly close together in the building (they are separate offices along one long wall of the building) and the stations that are getting small amounts of packet loss are scattered all over the office.
I haven't pulled any of the cables out of the conduit yet to check for physical damage since I'm waiting on the delivery of new cable from Monoprice. I'm personally hoping that there isn't any apparent damage since that would mean having a contractor come in and tear the walls apart to get to and correct the problem (the walls are solid wood and plaster and not easily accessible), but I can't think of anything else that could have caused a failure in so many drops at the same time. Also, the fact that many of the still working stations are now getting small numbers of PING packet losses where they were all perfect before the power outage makes me wonder if there is something even more significant wrong with the wiring. Either that or I missed something in my troubleshooting and I'm hoping you can point that out for me..
If you have any ideas, no matter how unlikely, as to what might have caused all of the drops to fail at the same time, or if you think it might be something else causing the problem, I'd appreciate the input since this one has me baffled.
Facebook
Twitter