Ever inherit a shop that's so F'd up you want to throw everything out and start over?

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Got hired on to be a sys admin, shortly thereafter the guy that hired me, who was the only one running this place from the get go, quit. This place is an organizational nightmare. Random shares everywhere like the domain controller (which is also being used as an app server, what a brilliant idea!) and exchange server. AD has never been cleaned up so there's user accounts, that are still active mind you, from employees that left years ago, same with mailboxes. All the servers are still running 2K server, some of which haven't been patched in 6+ months (a couple of which I can't even patch now because the sys volumes were created with 2-3gig partitions and they're completely out of space). Very little policy set on group policy, pretty much any user has full reign of their PC with no oversight. No management of anti-virus or spyware, I don't find out about problems until someone complains. Horribly outdated antispam, people were getting dozens, in some cases hundreds, of spam emails direct to their inbox on a daily basis.

The only documentation on anything is what he threw together in the 2 weeks between him putting in his notice and leaving. He didn't keep track of the licensing on a lot of stuff either. Luckily he hired another guy, right after he brought me in, to manage all of the Cisco gear and VOIP stuff (and he's having the same amount of fun with all of that as I am with the servers).

Hopefully we'll get all the money we need from the higher ups. The hardware is 4-5 years old so I'm hoping to buy all new stuff, start from scratch and then migrate everything that's needed over to the new gear.

Let the good times roll. /rant
 

nakedfrog

No Lifer
Apr 3, 2001
61,775
17,492
136
Good luck on getting the money ;)
"Everything is working, why do we need new stuff?"
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: nakedfrog
Good luck on getting the money ;)
"Everything is working, why do we need new stuff?"

Yeah, and quick. Without going into too much detail, the industry this business operates in has many regulations regarding how stuff is handled, and being accountable for that by showing proof of your practices. If we were audited today, it wouldn't be fun.
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: FoBoT
Originally posted by: jtusa
Let the good times roll. /rant

sounds like job security to me ;)

I hope so. Between me and the other ugy, we know how to do just about everything that needs to be done, just can't do it for free. If I can get the resources, the difference will be night and day when we're done.
 

iamwiz82

Lifer
Jan 10, 2001
30,772
13
81
Sounds like when I started here. My boss had just been hired and he really straightened things out.

When we started there were 2 servers. One linux box that had out manufacturing database, and one windows box that did EVERYTHING else.
 

nakedfrog

No Lifer
Apr 3, 2001
61,775
17,492
136
Originally posted by: jtusa
Originally posted by: nakedfrog
Good luck on getting the money ;)
"Everything is working, why do we need new stuff?"

Yeah, and quick. Without going into too much detail, the industry this business operates in has many regulations regarding how stuff is handled, and being accountable for that by showing proof of your practices. If we were audited today, it wouldn't be fun.

I guess as long as they understand that you'll be alright.
 

Queasy

Moderator<br>Console Gaming
Aug 24, 2001
31,796
2
0
So why did the other guy quit? Is it for any reason that will make your life a living hell beyond what you've learned already?
 

pontifex

Lifer
Dec 5, 2000
43,804
46
91
Originally posted by: jtusa
Got hired on to be a sys admin, shortly thereafter the guy that hired me, who was the only one running this place from the get go, quit. This place is an organizational nightmare. Random shares everywhere like the domain controller (which is also being used as an app server, what a brilliant idea!) and exchange server. AD has never been cleaned up so there's user accounts, that are still active mind you, from employees that left years ago, same with mailboxes. All the servers are still running 2K server, some of which haven't been patched in 6+ months (a couple of which I can't even patch now because the sys volumes were created with 2-3gig partitions and they're completely out of space). Very little policy set on group policy, pretty much any user has full reign of their PC with no oversight. No management of anti-virus or spyware, I don't find out about problems until someone complains. Horribly outdated antispam, people were getting dozens, in some cases hundreds, of spam emails direct to their inbox on a daily basis.

The only documentation on anything is what he threw together in the 2 weeks between him putting in his notice and leaving. He didn't keep track of the licensing on a lot of stuff either. Luckily he hired another guy, right after he brought me in, to manage all of the Cisco gear and VOIP stuff (and he's having the same amount of fun with all of that as I am with the servers).

Hopefully we'll get all the money we need from the higher ups. The hardware is 4-5 years old so I'm hoping to buy all new stuff, start from scratch and then migrate everything that's needed over to the new gear.

Let the good times roll. /rant

Sounds like my job.
my boss was the only IT guy here before I came and is still here. everything just seems random and not set up really well. anytihgn i suggest to make it better, he turns down. i think its because he doesn't want to be told that he's wrong or doesn't want to lose complete control.

it could seriously use a complete start over.

now, we're in the process of being acquired by another company so we're going to have to comply with their policies and stuff. it's going to be a bit of a change around here, probably a much needed one, though.
 

FoBoT

No Lifer
Apr 30, 2001
63,084
15
81
fobot.com
Originally posted by: Queasy
So why did the other guy quit? Is it for any reason that will make your life a living hell beyond what you've learned already?

like disgruntled worker logic bombS?
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: Queasy
So why did the other guy quit? Is it for any reason that will make your life a living hell beyond what you've learned already?

You know I'm not 100% sure, but I hope not. I think it was a combination of him getting a better offer elsewhere, wanting to try a different place, and butting heads with the administration/directors here about all kinds of stuff (not all IT related).
 

MovingTarget

Diamond Member
Jun 22, 2003
9,002
115
106
Always beware layers 8 and 9 of the OSI model: politics and money. Hopefully you will get your funds, but most likely these things end up with having to do some rather creative work with existing equipment until the next fiscal year. Gotta love that.
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: FoBoT
Originally posted by: Queasy
So why did the other guy quit? Is it for any reason that will make your life a living hell beyond what you've learned already?

like disgruntled worker logic bombS?

Lol, I don't think that'll be a problem, at least I hope not. :Q
 

Skacer

Banned
Jun 4, 2007
727
0
0
Those kinds of challenges can be a lot of fun, if you get the money you can basically design shit the way you want to see it.

And if you don't get all the money you were hoping for, that is just another challenge.
 

alkemyst

No Lifer
Feb 13, 2001
83,769
19
81
how big of a shop is it? We have 30 something servers for 150 users here and about another 100 remotely....but we also attach to a Fortune 500 national network.

A small shop of even 50 associates will probably be sharing server duties and using older equipment.

Nothing is wrong with older machines. Heck a PIII-S 1.4 dual makes a nice web box still.
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: alkemyst
how big of a shop is it? We have 30 something servers for 150 users here and about another 100 remotely....but we also attach to a Fortune 500 national network.

A small shop of even 50 associates will probably be sharing server duties and using older equipment.

Nothing is wrong with older machines. Heck a PIII-S 1.4 dual makes a nice web box still.

Well it's not a huge shop, roughly 150 seats, some of those shared. The problem with the hardware is that some of it is end-of-life/out of warranty and while parts are available, they aren't worth the cost. There's also zero redundancy with most of it. Plus it's hard to rebuild an exchange server on the server that's currently in production. ;)

With the switching gear, we're just about at capacity with our ports, and some of the switches are almost end-of-life. Pretty soon parts and support will be very hard to come by if needed.
 

SoulAssassin

Diamond Member
Feb 1, 2001
6,135
2
0
Going through this right now in my new job. Backup environment is poorly designed, there's bottlenecks everywhere, no formal policies, nothing documented....we ran out of space on the disk we use for backup staging the past two weekends and everyone is looking at me and giving me shit. They have a single Oracle dbase that 4+TB plus logs going to a 4.7TB volume plus all of the other smaller databases which is probably a couple TB's. You can't fit 10 gallons of water in a 9 gallon container last I checked.
 

Apathetic

Platinum Member
Dec 23, 2002
2,587
6
81
Ummm... Please tell me your domain actually has a backup domain controller. If not, I think you know what your first step should be.

Dave
 

ViviTheMage

Lifer
Dec 12, 2002
36,189
87
91
madgenius.com
yup, but it was a messier version with novell 6.5...we just threw in 2 dc and we took down the original 13 shares from novell and made them 3 on AD .. life is easy now,
 

jtusa

Diamond Member
Aug 28, 2004
4,188
0
71
Originally posted by: Apathetic
Ummm... Please tell me your domain actually has a backup domain controller. If not, I think you know what your first step should be.

Dave

Well there's the primary domain controller, which also acts as an app server, and one of the actual app servers is the secondary. So technically we have two, but I don't like it like that at all.
 

middlehead

Diamond Member
Jul 11, 2004
4,573
2
81
I'm on my way out of such a situation. Walked in during early May without any support, the previous guys had quit in late March and early April, and have spent the last 5 months trying to drag this sinking ship above water. Came in with 30 servers and only 2 under active service contracts, convinced them to add one server in the late summer which is the only correctly built piece of hardware in that room, and spent the rest of the time trying to get them to approve new hardware for ridiculously over-tasked domain servers. One is a domain controller, DNS server, file server (for roaming profiles, no less), and the sole Exchange server. The other domain controller is also a partial file server, a print server, a DNS server, and hosts a 12 year old internally-created Delphi app for which there is 0 documentation. I'm quite sure I'm forgetting a few things on each of those boxes, and that's just two-of-thirty servers and doesn't begin to touch user level hardware.


My two week notice was effective today.