Glitchy switches

[Mark R]

Just wondering how you guys would handle this.

We've been having some probs at work with the LAN going down, leaving mission-critical servers isolated, critical workstations without connectivity and VOIP phones failing.

Apparently, the problem has been traced to a network switch, as the LAN recovers after the switch gets rebooted. The problem has been that over the last couple of weeks, the switch has been needing rebooting every couple of days, and more recently, it seems to have been causing some neighboring switches to freak out, as occasionally some workstations which are not directly connected to it, seem to have weird connectivity issues (but this last issue is highly speculative).

I'm just wondering whether it is normal practice just to keep rebooting a switch, because it is 'glitchy'. Or, how many times would you reboot it before replacing it?

 

[ScottMac]

That is abnormal behavior. The switch could be bad, it could be stressed (too hot, no air flow, bad / noisy power), or may have taken a hit from power, static through the UTP ... any number of possibilities.

It may also be that there is/are a/some conflicting devices on the LAN, or multiple parallel runs (some user trying to get twice as much throughput by connecting to two ports), or someone "needing" more than on connection has added their own SOHO router to the line, and it's supplying DHCP pointing back to itself ... making a loop, or just shooting data to a black hole (this, of course, is grounds for killing the (L)user and feeding his worthless flesh to the river carp ... optionally, you could just fire them and kill 'em later).

IMHO, you need to put a monitor (ala WireShark) on the line to your most-shared link and see what the traffic looks like, and/or, quick & easy, replace the switch and see if the nature of the problem changes.

Good Luck

 

[imagoon]

My personal opinion is a switch should never need a reboot because a software crash. After a couple of consistent crashes I would replace it, justifying the replacement based on wasted cash on diagnosing it and employee lost time.

 

[mammador]

How is your network designed in the area near the switch? Is it a large broadcast domain?

 

[seepy83]

I'd pretty much echo the replies so far. You should try to find out a cause for the problem. You just need to make sure that the problem isn't being caused by a device that's plugged into the switch, or someone's sloppy cabling creating a loop. If it's really the switch, then it needs to be replaced...infrastructure hardware (switches, servers, firewalls, routers, etc) should never require regular reboots. It's just not acceptable as far as I'm concerned.

 

[RadiclDreamer]

My opinion is if you need to reboot a network device, upgrade the code. If it does it again, replace it.

 

[Specop 007]

I understand the desire for diags and troubleshooting however my feelings are sometimes its just not worth the effort.

I would replace the switch and also turn up diags once the new switch was in place with port mirroring and wireshark.

If the problem does not reoccur you know the switch was indeed bad. RMA the switch and recover your diag gear, case closed.

If the problem continues you should have the diags needed to analyze the problem and you also have a spare switch on hand.

Obviously if its an expensive switch I would collect diags first before just replacing it. So to a degree the cost of the equipment dictates the next steps.

 

[spidey07]

My guess is the processor is getting overloaded and it's not playing spanning-tree very nice and a loop is being created.

But as others have said, you should NEVER have to reboot network equipment. It's built to be up and stay up.

 

[VirtualLarry]

you should NEVER have to reboot network equipment. It's built to be up and stay up.

Amen to that! My experiences are more with SOHO gear than enterprise gear, but that's one thing I like about my DD-WRT routers, they stay running for months at a time.

I once had a Netgear gigabit switch, a 5-port in their silver-and-white plastic case. It was flaky. I would leave the computer connected to it, for months at a time, running DC projects, and I would come back to check on it every few months, and find out it had no connectivity, because the switch had gone dead and needed a reboot.

I replaced them when I moved with some Trendnet gigabit 8-port switches in metal cases, and haven't had any further problems.

 

[imagoon]

But as others have said, you should NEVER have to reboot network equipment. It's built to be up and stay up.

Oh come now... even Cisco needs a reboot once a decade or so.

 

[Cooky]

Oh come now... even Cisco needs a reboot once a decade or so.

Not sure about a decade, but we have Cisco switches that have been up for many years.
Of course now that I just jinxed myself, we'll probably get calls on Monday...

 

[spidey07]

Oh come now... even Cisco needs a reboot once a decade or so.

Or when they crash! And it DOES happen. Especially MSFCs on 6500s

 

[Stargun]

The short answer has already been stated. A network switch should really never need to be rebooted.

As for how to proceed is another question entirely. In order to answer that I would need to know first off is this a managed or un-managed switch and who makes it.

The idea of just simple replacing it some what baffles me. I work for a large enterprise where I have a decent budget and if I went to my boss and said this switch needs rebooting and my first trouble shooting step is to replace it he would laugh at me. Even for one of our relatively cheap un-managed switches. If it was a SOHO 8 porter than that might be viable. But I cant imagine just tossing money at something.

 

[cmetz]

Agree with others on the switch. However, just because rebooting the switch fixes the problem does not mean that the switch is the problem. Flaky cables and flaky power come to mind as other possible culprits too.

Just be prepared for the possibility that there are other causes, and work carefully to confirm or deny your theories when debugging. Prematurely blaming one component can waste you a lot of time when debugging intermittent issues. Do science.

 

[Emulex]

you sure you don't have a bad revision or firmware? I'd check that first. never had to reboot switches unless it was to reroute their power

 

[spidey07]

Agree with others on the switch. However, just because rebooting the switch fixes the problem does not mean that the switch is the problem. Flaky cables and flaky power come to mind as other possible culprits too.

Just be prepared for the possibility that there are other causes, and work carefully to confirm or deny your theories when debugging. Prematurely blaming one component can waste you a lot of time when debugging intermittent issues. Do science.

This is why network troubleshooting is different. You don't just replace stuff because something going wrong in one place can merely be a symptom of a problem somewhere else. You MUST find out what is actually causing the problem, the real root cause, and understand it/address it.

It's a very different kind of troubleshooting that most aren't accustomed to and why networking is "voodoo".

 

[Gryz]

People at Anandtech seem to think that everything is hardware. While in fact, a network device is more software than hardware. So my first step would be: do an upgrade of the software to the most recent maintance release. If it still causes problems, then start troubleshooting.

 

[imagoon]

People at Anandtech seem to think that everything is hardware. While in fact, a network device is more software than hardware. So my first step would be: do an upgrade of the software to the most recent maintance release. If it still causes problems, then start troubleshooting.

Switch gear is not like computers. The firmware is installed and runs exactly the same, there is never a new application installed etc. Generally if it has worked for years, then starts rebooting, it indicates there are issues with the hardware. Switch issues caused by software are often revealed early in the gears life. Example would be doing any thing advanced with certain versions of Dell or Extreme switches.

 

[Stargun]

We really cant help this guy much farther unless he tells us the make and model of the switch. Where to start would be different depending on if it is a managed or un-managed switch.

So Mark if you want the most accurate help to your problem we need to know the make and model of your switch.

 

[Mark R]

We really cant help this guy much farther unless he tells us the make and model of the switch. Where to start would be different depending on if it is a managed or un-managed switch.

So Mark if you want the most accurate help to your problem we need to know the make and model of your switch.

I can't tell you. I'm not the LAN guy, so I don't know anything except what I'm told. Which is that a switch has been acting up and been needing rebooting.

I was just wondering if it was part of the normal diagnostic procedures just to keep rebooting an apparently crashing switch. It just struck me that this was markedly abnormal for LAN infrastructure, but not knowing the subtleties of network debugging, I wasn't sure.

Anyway, thanks for the insight. It was interesting.

 

[Specop 007]

People at Anandtech seem to think that everything is hardware. While in fact, a network device is more software than hardware. So my first step would be: do an upgrade of the software to the most recent maintance release. If it still causes problems, then start troubleshooting.

Depends on our environment.

I finally have the green light that every truck roll out to a certain market is a complete hardware upgrade. We have some pretty old plant and we need to get off it and migrated to new equipment.

On our new stuff? We run our asses ragged gathering diags and sorting out issues. So half my day is diags and half my day is swap broke shit.

 

[Stargun]

I can't tell you. I'm not the LAN guy, so I don't know anything except what I'm told. Which is that a switch has been acting up and been needing rebooting.

I was just wondering if it was part of the normal diagnostic procedures just to keep rebooting an apparently crashing switch. It just struck me that this was markedly abnormal for LAN infrastructure, but not knowing the subtleties of network debugging, I wasn't sure.

Anyway, thanks for the insight. It was interesting.

Well in that case Mark you got your answer at the beginning. No it is not normal for the switch to continually need to be rebooted.

 

[Emulex]

my case for intermittent hardware: return it - hp procurve (most) and hp 3c0m have lifetime warranty. some newer SMB products (2960) may also have similar warranties - if software doesn't fix the situation and obvious configuration issues are not killing it - then return it - intermittent is worse than failed imo.