Damn!! Damn!! Damn!! Damn!! I was trying to leave work early and KABOOM!!!

[shiner]

The main SQL server for the hospital blew up.

 

[DurocShark]

hah

teach you!

Now, what kind of blew up? Hardware or software? Something a quick reboot can't fix?

 

[Rob9874]

Yeeaaahhhh, I'm gonna need you to come in on Saturday.

 

[djheater]

Originally posted by: shinerburke
The main SQL server for the hospital blew up.

never fvcking fails.

Why can't they go in the middle of the week?

Luckily I don't have to deal with that anymore.

Good luck.

 

[T3C]

Originally posted by: shinerburke
The main SQL server for the hospital blew up.

hehe, now your stuck in the office late

good luck getting it fixed..


-CYBPNK

 

[shiner]

Hardware....one of the array controllers failed. Luckily it's an old Proliant 7000 with redundant array controllers so we have it back up....just slow....I have one of my guys working it, but since I'm the Server Engineering Manager I have to stick around until the part is replaced.....why? So I can rport to upper management when they call every 10 minutes wanting an update. Here's an update for ya....quit calling me and making me call my people....when that happens it just slows them down.

 

[vi edit]

(poor)visual aide

 

[MichaelD]

Originally posted by: vi_edit
(poor)visual aide

But oh so efficient!

 

[NogginBoink]

Originally posted by: shinerburke
Hardware....one of the array controllers failed. Luckily it's an old Proliant 7000 with redundant array controllers so we have it back up....just slow....I have one of my guys working it, but since I'm the Server Engineering Manager I have to stick around until the part is replaced.....why? So I can rport to upper management when they call every 10 minutes wanting an update. Here's an update for ya....quit calling me and making me call my people....when that happens it just slows them down.

I can empathize.

When I get a "critical situation" support call, everyone starts running in circles in a panic and I get all kinds of emails and voice mails wanting status updates.

I always want to tell them, "The customer and I are working on fixing the problem. It would be much faster and easier if you would just stop calling me."

Of course, I can't say that.

 

[Mark R]

At the hospital where I work, when the main Win2k domain server went down on a friday night it was just left until Monday morning.

It was absolute chaos that weekend, as nobody could log into any PC, let alone get lab results, book appointments, etc.

 

[shiner]

Originally posted by: vi_edit
(poor)visual aide

You know we actually had some servers smoke once. Christmas Eve last year an asshat security guard went into our Comm Room for some reason. Why? Nobody knows. Well anyway he couldn't figure out how to get back out of the room, doorknobs must confuse him, so he saw a big read button. The big red button. Now this big red button is sealed in a plastic see through box on the wall next to the door. The box has a sign on the front of it that says "Warning: Power Off Entire Room!" The box is also locked. The key is taped on the side of the box, you know the box with the warning sign. Anyway his ass takes the key off, unlocks the box, and then hits the button. Guess what. The entire server farm and all the networking equipment went down. I was at home, relaxing when my pager and cell phone both started having seizures. I answer the phone and it's my boss, who was in the position I have now at the time, in a panic saying "All the servers are down. How fast can you be at the hospital?" I told him it would take me 9 minutes to get there. I get there and the power is still off, luckily. We, another server guy had made it in, are in the process of going around and unplugging all the equipment so that there would not be a surge when the button was pulled back out. We had unplugged about 12-14 servers when the security supervisor shows up and starts asking the guard what happened and why he pushed the button. Well in the process of his explanation the dumbass pulled the button back out which turns the power back on. Oh it was interesting.....bright lights come back on and I think "oh sh1t!" but it was too late....the resulting surge in power smoked several of the power supplies in older servers, a few hard drives, and a motherboard in one of our mail servers. It also took out some network gear.... Needless to say that dumbass guard lost his job and the server team got to spend Christmas Eve in the Comm Room. Our SQL guy, same poor guy that is dealing with it today, had to spend all night and most of Christmas day getting his SQL server back.

 

[shiner]

Originally posted by: Mark R
At the hospital where I work, when the main Win2k domain server went down on a friday night it was just left until Monday morning.

It was absolute chaos that weekend, as nobody could log into any PC, let alone get lab results, book appointments, etc.

Why do you only have one DC? Lemme guess....you have multiples but it was the one with all the FSMO roles.

 

[Encryptic]

Originally posted by: Rob9874
Yeeaaahhhh, I'm gonna need you to come in on Saturday.

"Oh, I almost forgot. I'm also going to need you to come in on Sunday. Yeeeeeaaaahh..."

 

[Ime]

That sucks shinerburke, esp. since I saw the picture you posted about who is waiting for you at home!

I hope it gets fixed quick so you can have a good weekend!

 

[Beau]

Never would have happened if you were still using paper and pens. Damn technology. It's making us lazy.

 

[NogginBoink]

Originally posted by: shinerburke

Originally posted by: Mark R
At the hospital where I work, when the main Win2k domain server went down on a friday night it was just left until Monday morning.

It was absolute chaos that weekend, as nobody could log into any PC, let alone get lab results, book appointments, etc.

Why do you only have one DC? Lemme guess....you have multiples but it was the one with all the FSMO roles.

No, you can still authenticate with none of the FSMO roles online.

 

[vi edit]

The only time I have actually seen a computer smoke, and I mean heavy smoke, was when a power plug inside the case got pinned between the rackmount and the case itself. It somehow managed to short itself out and burnt off all of the insulation on the power cord and melted down the plastic plug on the end of the cable. All that was left was a scrawny little charred pair of wires.

Ooops!

 

[shiner]

Originally posted by: NogginBoink

Originally posted by: shinerburke

Originally posted by: Mark R
At the hospital where I work, when the main Win2k domain server went down on a friday night it was just left until Monday morning.

It was absolute chaos that weekend, as nobody could log into any PC, let alone get lab results, book appointments, etc.

Why do you only have one DC? Lemme guess....you have multiples but it was the one with all the FSMO roles.

No, you can still authenticate with none of the FSMO roles online.

That's true...supposedly. We did have another customer that their server with the roles on it died and they couldn't get in. Hell...you never know with Windows what the hell is going to happen

 

[RayH]

If I recall correctly, in our server room hitting the shiny red button means you have a seconds before you suffocate to death.

 

[shiner]

Originally posted by: RayH
If I recall correctly, in our server room hitting the shiny red button means you have a seconds before you suffocate to death.

Our mainframe room is like that. They haven't switched out the Halon yet.

 

Originally posted by: shinerburke

Originally posted by: vi_edit
(poor)visual aide

You know we actually had some servers smoke once. Christmas Eve last year an asshat security guard went into our Comm Room for some reason. Why? Nobody knows. Well anyway he couldn't figure out how to get back out of the room, doorknobs must confuse him, so he saw a big read button. The big red button. Now this big red button is sealed in a plastic see through box on the wall next to the door. The box has a sign on the front of it that was "Warning: Power Off Entire Room!" The box is also locked. The key is taped on the side of the box, you know the box with the warning sign. Anyway his ass takes the key off, unlocks the box, and then hits the button. Guess what. The entire server farm and all the networking equipment went down. I was at home, relaxing when my pager and cell phone both started having seizures. I answer the phone and it's my boss, who was in the position I have now at the time, in a panic saying "All the servers are down. How fast can you be at the hospital?" I told him it would take me 9 minutes to get there. I get there and the power is still off, luckily. We, another server guy had made it in, are in the process of going around and unplugging all the equipment so that there would not be a surge when the button was pulled back out. We had unplugged about 12-14 servers when the security supervisor shows up and starts asking the guard what happened and why he pushed the button. Well in the process of his explanation the dumbass pulled the button back out which turns the power back on. Oh it was interesting.....bright lights come back on and I think "oh sh1t!" but it was too late....the resulting surge in power smoked several of the power supplies in older servers, a few hard drives, and a motherboard in one of our mail servers. It also took out some network gear.... Needless to say that dumbass guard lost his job and the server team got to spend Christmas Eve in the Comm Room. Our SQL guy, same poor guy that is dealing with it today, had to spend all night and most of Christmas day getting his SQL server back.

Note to self: Stay out of server room

 

[Mark R]

Why do you only have one DC? Lemme guess....you have multiples but it was the one with all the FSMO roles.

I don't know exactly what happened. I don't work in IT.

All I know is that some server crashed, and that I couldn't find a working PC anywhere in the building, and that IT don't have an out-of-hours contact number (or no one thought to ask switchboard for it, myself included).

I also suspect that it can't have been a major problem, because by 8:05 on Monday morning everything was up and running.

 

[DurocShark]

Hate dumba$$ phb's. "Is it fixed yet?" 5 min later "Is it fixed yet?" 5 min later "Is it fixed yet?"... and on and on...

Then there's the other side: My job is a help desk lead supporting a brand new custom developed app. Of course, the company doing this have an "in" with the pres of our company, so they feel like they can do whatever. They have STUPID bugs in the app (like error messages with an invalid phone number/e-mail addy for the user to contact) and odd calculation problems, javascript coding issues, etc. Stuff that should have been found during test and val, but...Oh, we don't NEED test and val... But I digress...

We at the he11desk call them when a major failure occurs, they say "ok" and we NEVER hear from them again. No idea what the problem is, no kind of ETA, nothing. So our thousands of users call us screaming, and all we can say is, "the developers are working on it... Sorry, they haven't provided us with an ETA as yet, but as soon as it's fixed we'll send out an alert.."

There *is* a middle ground!!! If it's a coding problem, that will probably take an hour, SAY SO. If that ETA changes, let us know! Our job is to field the calls so you can get the job done, but if you don't give us anything to work with, all you're gonna get is us calling your friggin cells constantly. We'd rather have the back end fixing the problem than answering our calls, but we've gotta keep calling until we get some kind of information.

 

[Mr N8]

Originally posted by: Mark R

Why do you only have one DC? Lemme guess....you have multiples but it was the one with all the FSMO roles.

I don't know exactly what happened. I don't work in IT.

All I know is that some server crashed, and that I couldn't find a working PC anywhere in the building, and that IT don't have an out-of-hours contact number (or no one thought to ask switchboard for it, myself included).

I also suspect that it can't have been a major problem, because by 8:05 on Monday morning everything was up and running.

You understimate the pressure that management puts on IT when these things happen. We have to be there until the newtwork is running better than before the crash, and it doesn't matter that your family is all at home eating Thanksgiving dinner.

 

[Mark R]

You understimate the pressure that management puts on IT when these things happen.

You're probably right. However, I was very inconvenienced by this - espeically when the cause of the crash was an expected power outage (scheduled generator load testing).