A night from hell in the ER last night

[TraumaRN]

So most people here know that I work in an urban ER setting as an ER nurse. And I'll occasionally post interesting stories about some of my patients and whatnot. But last night was hell and it wasn't because of one patient...

First a little background, and interesting note. That is my hospital system is one of the most wired hospital systems in the entire US. We've been awarded numerous times for our IT infrastructure(giant server racks with HP blades in all 9 hospitals) and actually were named by Forbes.com as one of the 'Most Wired Hospital Systems' Blah blah...we have totally computerized charting, Only 3-4% of a patients chart is on paper now...well thats all well and great, makes life so much easier.

Except last night it failed.

The server room lost power in my hospital, all the servers kicked the bucket and shut down. So all our nice computerized systems at my hospital suddenly FAILED.

And of course our server is also our internet backbone too. So we didnt even have internet access and only our landline phones were working.

It was utter mass chaos for over 6 hours until they could restore power and reboot all the computer systems. We had to handwrite all patient wristbands, stickers, everything. It had to be so dangerous for patients too, handwriting stuff like that. Especially where medications are concerned. We had to call in extra doctors, nurses everything. FOr 6 hours it was pure hell.

And i have to give massive props to our IT people. I here on ATOT and elsewhere it's easy to be ridiculed for working in IT but only 6 hours to reboot all those systems and in a server room that large(pushing around a hundred terabytes or more a day) to get everything working again is awesome...I have new respect for IT people...

On the other hand I want to kill the person who didn't install some sort of redundancy system...or if they did it needs to work better than it did last night.

 

[skyking]

That was an unforseen and totally unacceptable failure mode for a hospital, IMO. Someone may get fired, and huge changes will be made because of that.

 

[DaveSimmons]

I'd be wondering why the servers had no UPS systems with alarms, and why there wasn't a working backup generator.

 

[2Xtreme21]

What would cause that all to fail like that? Someone didn't place correct safeguards in, if you ask me.

 

[bwatson283]

A server room "should" never lose power! Please tell me the generators kicked the bucket.

 

[Quasmo]

No kidding, the first thing I would install would be a backup generator.

 

[TraumaRN]

Supposively the server's power supply was cut and the backup generator didnt and/or couldnt fire. But i'm just a nurse they dont tell me everything

Originally posted by: skyking
That was an unforseen and totally unacceptable failure mode for a hospital, IMO. Someone may get fired, and huge changes will be made because of that.

Yea trust me I found that out first hand. I've sure someone is in some deep sh!t this morning.

That old saying about having all your eggs in one basket...

 

[LordMorpheus]

I worked at a consulting engineering company a few years back, we were re-doing some systems in a VA hospital.

Part of the project was focused on the server room - they had a battery UPS good for a fair amount of time and a huge diesel generator in the basement that would kick in (the UPS was for the transistion) to power the servers and the other critical systems in the hospital. An outside power failure wouldn't kill the servers. A problem in the server room might take them down, though.

Could be the case here - it might not be an outside power failure causing problem, something could have shorted inside the server room cause some breakers to flip.

 

[TraumaRN]

Originally posted by: LordMorpheus
I worked at a consulting engineering company a few years back, we were re-doing some systems in a VA hospital.

Part of the project was focused on the server room - they had a battery UPS good for a fair amount of time and a huge diesel generator in the basement that would kick in (the UPS was for the transistion) to power the servers and the other critical systems in the hospital. An outside power failure wouldn't kill the servers. A problem in the server room might take them down, though.

Could be the case here - it might not be an outside power failure causing problem, something could have shorted inside the server room cause some breakers to flip.

It had to be in the server room to cause the backup UPS and generators to not fire. Because the rest of the hospital never even had a hiccup in power.

 

[SarcasticDwarf]

Originally posted by: DeathBUA
Supposively the server's power supply was cut and the backup generator didnt and/or couldnt fire. But i'm just a nurse they dont tell me everything

Originally posted by: skyking
That was an unforseen and totally unacceptable failure mode for a hospital, IMO. Someone may get fired, and huge changes will be made because of that.

Yea trust me I found that out first hand. I've sure someone is in some deep sh!t this morning.

That old saying about having all your eggs in one basket...

I find it interesting how many failures of datacenters occur when the backup generators fail to start.

 

[Homerboy]

Sounds like there's going to be an opening at your ER for a SysAdmin.
Simply put, that should NEVER EVER happen, especially in a mission critical application such as yours. Hell Ebay would never go down like that. No reason an ER should.
Sadly I think all the awards and accolades your facitly has received don't seem to be warranted if there is no adequate fail-safe in place.

Sucks for you guys in the trenches I'm sure. Sucks more for the soon-ti-be-unemployed Admin or consulting firm.

 

[TraumaRN]

Originally posted by: Homerboy
Sounds like there's going to be an opening at your ER for a SysAdmin.
Simply put, that should NEVER EVER happen, especially in a mission critical application such as yours. Hell Ebay would never go down like that. No reason an ER should.
Sadly I think all the awards and accolades your facitly has received don't seem to be warranted if there is no adequate fail-safe in place.

Sucks for you guys in the trenches I'm sure. Sucks more for the soon-ti-be-unemployed Admin or consulting firm.

It wasn't just the ER the whole hospital is computerized like that. So everywhere went down, ER, floors, OR, ICUs, clinics...etc...

 

[Pliablemoose]

Ouch, my current hospital is not computerized, we only have lab & xray online.

It's lame, but nice, I rmember those days when everything would go down at other places I've worked, it's pure hell.

 

[xanis]

Your entire infrastructure was fucked because the sever room lost power? That seems pretty absurd. Shouldn't the server room have been set up with UPSs and generators?

 

[bwatson283]

Originally posted by: Xanis
Your entire infrastructure was fucked because the sever room lost power? That seems pretty absurd. Shouldn't the server room have been set up with UPSs and generators?

learn to read

 

[xanis]

Originally posted by: bwatson283

Originally posted by: Xanis
Your entire infrastructure was fucked because the sever room lost power? That seems pretty absurd. Shouldn't the server room have been set up with UPSs and generators?

learn to read

FYI, I can read very well. What exactly are you talking about?

 

[Greenman]

The more complicated the plumbing, the easier it is to stop up the drain. It's a rule they don't seem to teach in engineering school anymore.

A critical system in a hospital should be fail safe, even if the fault was in the server room, there is no excuse for that system to go down. It's no different than an elevator that just plummets into the basement every now and then.

 

[TraumaRN]

Originally posted by: Xanis

Originally posted by: bwatson283

Originally posted by: Xanis
Your entire infrastructure was fucked because the sever room lost power? That seems pretty absurd. Shouldn't the server room have been set up with UPSs and generators?

learn to read

FYI, I can read very well. What exactly are you talking about?

I'll type it again...our entire hospital infrastructure is computerized. EMR, or electronic medical records. We are about 97% paperless, meaning nearly all charting and record keeping for a patient is kept on a computer(an ER visit before EMR generated about 100-150 pieces of paper charting, now with computerized charting an ER visit will generate about 5-8 sheets of paper). Now last night the server room lost power and according to an e-mail I got the backup generator didn't fire. Probably means the problem was within the server room itself since the hospital didn't lose power. When the servers went down so did our infrastructure since the servers are literally the beating heart of the hospital.

 

[homercles337]

Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

 

[xanis]

Originally posted by: homercles337
Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

You're generalizing. Not all IT people suck. It seems like the problem the OP's hospital had was all due to poor and ill-considered design.

 

[BrunoPuntzJones]

Originally posted by: Xanis

Originally posted by: homercles337
Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

You're generalizing. Not all IT people suck. It seems like the problem the OP's hospital had was all due to poor and ill-considered design.

Probably not the case, but it could be due to cost constraints at the top.

Some yahoo could have thought it crazy to authorize a million bucks for a back-up system, and IT had to make due with what they were given.

 

[smack Down]

Originally posted by: Xanis

Originally posted by: homercles337
Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

You're generalizing. Not all IT people suck. It seems like the problem the OP's hospital had was all due to poor and ill-considered design.

He is generalizing and he is right all IT people do suck

 

[SoundTheSurrender]

It could be possibly that Executives didn't want a backup solution due to costs which now they realized that they really need to assess the situation.

 

[middlehead]

Originally posted by: homercles337
Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

Judging from this quote, all scientists are assholes.

 

[iamwiz82]

Originally posted by: BrunoPuntzJones

Originally posted by: Xanis

Originally posted by: homercles337
Look, IT people SUCK. You would not have had a "night of hell" if IT did their fucking job. Sounds like you have a bunch of fuck ups in IT if you ask me. Im a scientist so this may mean nothing.

You're generalizing. Not all IT people suck. It seems like the problem the OP's hospital had was all due to poor and ill-considered design.

Probably not the case, but it could be due to cost constraints at the top.

Some yahoo could have thought it crazy to authorize a million bucks for a back-up system, and IT had to make due with what they were given.

Story of my life. Whenever someone bitches about things like that I say "I can do it, bring money!" If management doesn't approve the cost, it does not get done until it catastrophically fails, and then it's implemented the next week.