Network Mentality...

[jlazzaro]

Curious how many of you network guys ran into these issues with upper management/old timers. I think it?s a disservice to the network and the company you?re supporting to have the mentality of, "If it ain?t broke, don?t fix it".

For instance, we have over 30 remote wan sites around the country as well as 1500 local users. When I was first hired I began examining the topology of the network and noticed they used 100% STATIC routes for everything. When I asked the head network guy why they chose to go that route instead of something like OSPF he responded:

"Its worked this way for 10 Years, why change it?"

This statement kind of blew me away. I guess this is why the government is in the process of changing all of its IT to contractors. Instead of getting new young blood in there that can adapt the network to new technologies and find better ways to do things, they would rather use old, outdated technologies because it?s what they know and it?s safe. I guess they just fear change.

This is also true with the equipment we use. Yes, our main WAN and LAN routers are not that old. But our layer 2 closets still use 10 year old alcatel switches. These have no warranty or service on them, and we have very little spares. Again, I asked the head network guy why they haven?t upgraded these in the past 10 years; he gave me the response as the routing. I?m sure budget comes into play, but they could at least be upgrading what they can?

Comments?

 

[spidey07]

heh, you're not gonna like my answer.

On the WAN thing - sure I'm against static routes, but if you don't have multiple paths then static is just fine...especially in a hub and spoke WAN. If it isn't causing any trouble, why mess with it?

On the equipment side - are all the current needs being met from a performance and feature perspective? Hows the reliability? If it's reliable, works and there aren't any pressing needs for more features/performance then don't mess with it...the equipment is already fully depreciated and essentially free right now from an accounting perspective.

In short, if it isn't broke, don't fix it. Just like how the new network guys want to start upgrading IOS all over the place and I just ask them "why?" They can't come up with a reasonable answer other than "newer is better" Well that certainly isn't the case. New guys in IT/networking always want to change things but they don't have a good reason to do so and lack the experience to make good decisions.

There's another great rule in networking as well - Keep it simple stupid (KISS)

 

[jamesbond007]

I'm no uber-network-pro by any means, but here's my opinion...

It sounds like you work for a pretty big or at least fairly large-sized company. If you guys start making big changes and there's some kind of problem or error arising, perhaps the lead guy/people just doesn't want to take the heat? Judging by the amount people you guys are providing service to within your company, it sounds like a small failure could turn into a big catastrophy if a something bad happened. (Don't ask what - maybe a new router installation with a port left open or something and data gets stolen or whatever)

Just out of curiosity, is there any kind of shortage of bandwidth or high-latency issues where you could see improvements for? Would replacing the Alcatel switches improve or fix some bandwidth issues internally?

It really depends on the situation, but I don't really see a problem with the "If it ain't broken, don't fix it!" methodology.

 

[networkman]

Figure I'll chirp in with my two cents..

As the others have pointed out, rightly so, fixing things that aren't broken makes no sense, more so if you can't come up with a bonafide reason. LANs and WANs especially are all about RELIABILITY and UP-TIME, so you've got to come up with a REALLY good reason to risk messing with that.

Now, that said, what I would suggest for the younger/in-experienced guys is to do alot of reading, do alot of research, and build a case for your suggestion of upgrading. An IT manager is much more likely to seriously consider your suggestions if they can see that you've done your homework as it were, and are serious about your suggestion. I'd even suggest taking it a step further, and build a lab setup where you can demonstrate why the upgrade is needed.

Just because something works well doesn't mean it can't be made to work better - just do the research and be prepared to justify the expenditure of time and resources that it will require.

 

[blemoine]

My 2 cents.

"if it ain't broke don't fix it" really depends on what isn't broken. for example we have our workstations on a 3 year rotation. after 3 years 90% of them are still working just fine but we replace them because they are not under warranty and usually the software we are using has become more demanding. we can't afford to have a workstation go down for an extended amount of time. so it is worth it to replace them on a schedule.

on the other hand we have been using the same router for our T1 for about 7 years now. no reason to replace it because if the T1 is down all of our branches can operate independently with no problem. so it is not cost efficent for us to replace the router just to say we have a new router.

when making changes on a big network you must have your ducks in a row. you need to find out why they are setup like they are. there maybe a good explanation why they are using static routes. are why they are using old switches. once you cover all of your bases you will be in a better position to make a change

 

[ArchAngel777]

As with everything, you need to look at the bigger picture before ever making a change. If something works and the users are not complaining, do not change it.

Anyone who has family or themselves worked for a large company can attest to many of the idiot IT directors. They decide to go to a "new" system that is hyped only to make the employees dread the POS they installed. They were used to having something work and work well, then some asshat thinks they can make things better and pisses all over the companies employees... Then, in order to save their job, they try and find a way to blame the users for the system not working.

Happens all the time, both small and medium companies alike. Bottom line: Don't eff with a companies network unless it is broke.

 

[spikespiegal]

When I run into these kind of scenarios, I default to aking myself 'how will it help the end user perform their job', and if I can't justify it immediatley, I might as well not waste money on it because I won't be able to justify it to management.

I was working at this bank, when some 'young-un' consultant with his brand new Cisco certification decided our mixed infrastructure of 10/100 SMC hubs and a smattering of 3Com switches needed to be pulled out and replaced with Cisco Catalysts series switches. I wasn't opposed to the idea, but considered it a donkey before cart project because our servers were crap at the time and weren't very fast anyways. However, 'boy wonder' succeeded in talking management into the project. Cisco switches being better than all other types of switches because you can 'bridge and switch out of the same port' (mandatory for housewives everywhere), and you could manage them, and we all know switches need managing because we have so much time left over from not manageing our servers.

$75,000 later we had all our infrastructure upgraded, and not a single end user noticed a difference because there was no problem to begin with. Of course everytime we had a thunderstorm we'd lose a few switches because Cisco uses power supplies yanked from recyled Packard Bells in the Catalyst series. We also needed to train our staff to 'manage' the switches because god forbid a Network card malfunction would flood our entire network.

I also don't get the fuss over WAN switching. While not an expert in such things, the vast majority of WAN issues I run into are because you're trying to run an inappropriate applications across the pipe, or management is too cheap to pay for full T-1 services. I could care less what hardware is in the closet.

 

[Smilin]

I used to specialize in "dead" servers and getting them running again. I generally found two camps:

1. The "ain't broke don't fix it" crowd.
2. The "Always keep it updated" crowd.

Both of these groups had few if any problems. The first group for the obvious reason (no changes) and the second group because if a problem arose from an update they knew exactly which one.

The problem is that there is an invisible third group. These were the guys that were f!@#$ed. Bad.

The third group comes up when the guys in group #1 find something that "is broke, needs fixing". They would then need to get themselves patched and into group #2. They would apply a whole batch of updates, something would break and they would have no idea what happened.

The third group would also arrise when group #1 found their butts had been hacked.

 

[cmetz]

jlazzaro, you are not a business person. You need to learn to be, before trying to tell business people what to do.

You have described several technical things you think could be done better. You have not described the cost / benefit analysis of your proposed solutions. You are criticizing the business people for doing what they're supposed to do - not go spending money on things without a business case.

"If it ain't broke don't fix it" is often the viewpoint of the technical folk on the ground. "Don't spend money on stupid %@$#% we get no benefit from" is the viewpoint of the same situation from the executives up top.

Remember, they need to have money around to pay for expenses, like your salary. You really DON'T want executives who throw money at every little thing. You DO want executives who will sometimes tell you that you have a great idea, and they appreciate the input, but that there's not enough benefit to justify the cost. You also DO want executives to whom you can come with a thoughfully done cost/benefit analsys and get budget to make your changes.

Now, sometimes there is an institutional conservatism that is truly stifling and goes beyond the cost/benefit, even the most obviously justified change just won't get done. If you encounter this, LEAVE. Such corporations or organizations simply have a broken culture, and they're going nowhere - though many of those folks will take a long time to get there.

But more often than not, the fundamental problem is that you the technical guy are looking at technical things from a technical perspective, and the business guys who control the purse are looking at business cases from a business perspective. You gotta speak their language. Their language is $$, and their language is cost/benefit. Remember, the network is a cost center, which in MBA speak translates to "spend as little money as you have to on this stuff." The network is a means to an end. You need to show them that spending X amount of money on changes you want will either bring in 10*X more revenue, or save them 10*X money elsewhere (for example, increasing efficiency per employee in a *tangible way*). That's a business case. That will get you budget and approval to make changes.

 

[shortylickens]

Lots of good responses here. But I wanted to make my own little input, since I can.

I whole-heartedly agree with the motto, but inthe Navy we learned that you can maintain stuff before it breaks. Ever here of planned maintanence or preventive maintenence?
In the USN we had a very large, thourough system for just that. It keeps things from breaking more often and less severely when it does.
Do you guys have a preventive maintenence system?
I'd bet a dollar or two you have a lot more corrective maintenence than you need to be doing.
When I got out I went to a government contractor and they have a little program. Not quite as complex as the Navy's, but it gets the job done.

The other issue here is common in all technical fields (I learned it as a Navy Electronics Technician).
Its difficult to detect when something is actually "broken". More often than not, the device or system in question functions, just at reduced capability.
If you have a gigabit switch for 20 employees, and they are only transferring files to each other at 1Mbit, thats reduced capability (from a tech's viewpoint). Some folks would say its broken, other folks would say its less than acceptable.
The users will be mad. The bean-counters will say "oh well, it still works right?".

As was mentioned above, each department will have their own view-point on the matter. As was also mentioned, you have to find a happy medium. Is one 1Mbit costing us money? If they company was set up to provide digital video and 3D redering services, probably so. That reduces the work output of all your coders and artists considerably. If its a law firm, they will probably suck it up for a while.
Until one of the partners gets hit. The day his personal connection goes from 5 megs to 3 megs there will be hell to pay in the IT department, (many of you know exactly what I'm talking about).

Sadly we had the same issue in the Navy. The captain had about 1Meg just for himself. The department heads all had to share a meg. The chiefs and lower officers had a half meg. And the rest of us (all 350) had about one quarter meg to share. What happened if the poor old captain couldnt browse the web fast enough? Take a wild guess.
Many was the time I had to sit and wait for 5 minutes while a two line email went out to my folks.

When I got out I was happy because at least I didnt have to see all the spoiled executives and hear their complaints.

(Incidentally, I just used the bandwidth example because this is the Networking forum. My experience is in RADAR and Comm systems, HF, VHF, UHF, EHF, some microwave.)

 

[ArchAngel777]

Originally posted by: shortylickens
(Incidentally, I just used the bandwidth example because this is the Networking forum. My experience is in RADAR and Comm systems, HF, VHF, UHF, EHF, some microwave.)

I have some experience with microwaves too. They tend to dry out food though, so I always add a bit of water to any dish I put in there for over a 2 minutes.

 

[RebateMonger]

Originally posted by: ArchAngel777

Originally posted by: shortylickens(Incidentally, I just used the bandwidth example because this is the Networking forum. My experience is in RADAR and Comm systems, HF, VHF, UHF, EHF, some microwave.)

I have some experience with microwaves too. They tend to dry out food though, so I always add a bit of water to any dish I put in there for over a 2 minutes.

Lots of microwave experience in this Forum.