Power failure practice at work

[Mark R]

So, we had a routine 'power failure' drill at work. The main idea was to test the effects of a mains power failure, and ensure that all critical systems were correctly supported by the backup generators.

All well and good, but a few things struck me as being a bit odd:

Drill was scheduled to start at 0900. Shortly after 0800, instructions went round to cease all computer work, save all data and shut down. By 0830, the core servers started being taken down and powered off prior to the test. Instructions were given not to use any computer until the test was complete and the all clear was given.

All normal work was cancelled, and only 'emergencies' were to be handled using contingency systems (pen and paper). Any appointemnts and meetings were cancelled.

Elevators were all grounded and switched off (20 storey tower blocks FTW)

Once all preparatory work was completed, the generator was started and power switched over. This kept some lights and receptacles in 'critical' areas running. (but not IT systems or elevators)

After 2 hours, the test was stopped after being hailed a resounding success, and IT systems started being restored and elevators were switched back on.

So now we all know we're ready for anything. Yay!

--

Still, better than last time. They tried this a few years ago - same thing. Except that time, the domain servers and other critical servers didn't come back up (apparently some sort of hardware failure - whether it was related to the generator test, or just bad luck when 24/7 critical servers were power cycled). However, that was on a Friday. It took until Monday before the new hardware could be installed. Boy, working that weekend was a real PITA - everything had to be done manually or by phone to departments that had working IT systems.

 

[spidey07]

That's retarded. You do the test with EVERYTHING running to simulate the outage. If the power was properly designed and installed you wouldn't even notice it except for the alarms/alerts that were automatically sent and the generators running.

 

[herm0016]

Originally posted by: spidey07
That's retarded. You do the test with EVERYTHING running to simulate the outage. If the power was properly designed and installed you wouldn't even notice it except for the alarms/alerts that were automatically sent and the generators running.

this. that was a crappy test of running a few things in the office. the school here runs tests, but they just send out an email with a time, and nobody notices any difference. we have 3 big caterpillar generators that can run most things on campus when the grid is down.

 

[Fritzo]

Heheh...I do this about once a year. I have everything designed to run non-crit systems for 2 hours, crit systems for 4 hours, and our phone system for 8 hours on battery power. I've never done a test past 60 minutes though, so this is all in theory of course

 

[MmmSkyscraper]

2 bloody hours? WTF were they doing?

"Yep, power's still off."

 

[DrPizza]

For what it's worth, we've done something similar at the school I teach at - we get about 1 day's notice, and then the next day, we have to shut power down to as many things as possible. In your situation, I find it difficult to believe that they can't figure out if "critical systems" are supported by the back-up generators without having to actually shut down the power. It seems completely illogical. (I'd love a logical explanation) The reason I find it illogical is I'm thinking about a hospital - suppose you have 10 patients on life support. (i.e. critical). You don't shut the power off for 2 hours to test whether or not their life support is backed up! You don't crash a car into a tree to see if you have the proper tools & parts in your garage to repair it. Even when emergency preparedness teams practice, they don't actually blow up a dam to see how well they can respond.

Anyway, I mentioned that my school does the same thing. However, the financial aspect is made very transparent to us: on a certain day, announced only 24 hours in advance, the power company has the big users in the area cut their consumption by as much as possible. (Something to do with their own preparedness for fighting brownouts, or something like that)

Anyway, the reason our school cooperates is that there are financial kickbacks for doing so. The greater the percent of energy use we decrease by for the hour, the bigger the kickback.

But again, your company's reason seems very illogical. Assuming there are some critical systems that aren't backed up by the generators, then ultimately by testing for 2 hours a year, in the long run, your systems are most likely down for more time than if you had just waited for an actual outage & just made any necessary fixes during the outage.

 

[I Saw OJ]

Originally posted by: spidey07
That's retarded. You do the test with EVERYTHING running to simulate the outage. If the power was properly designed and installed you wouldn't even notice it except for the alarms/alerts that were automatically sent and the generators running.

Agreed. At least you guys know that you are ready for a power outage if given 2 hours advance warning...

 

[SparkyJJO]

Originally posted by: MmmSkyscraper
2 bloody hours? WTF were they doing?

"Yep, power's still off."

Possible they were testing the power at the servers and making sure all the circuits that were to remain up were still up.

 

[Train]

Originally posted by: MmmSkyscraper
2 bloody hours? WTF were they doing?

"Yep, power's still off."

<walkie talkie voices>

is the power off in the 7th floor womens room?

roger.

is it off in the 9th floor cleaning closet?

pant pant, whew... uhh roger.

is it off on that cutesy aquarium outside the CEO's office on the 10th floor?

gasp...., ya!

ok just a few more random spot checks...

</walkie talkie voices>

 

[DrPizza]

Actually, the more I read your OP, the more I wonder if the reason given to you guys is bogus, or if the people running the place are really that stupid. How the hell do you not know what the effect of a power outage will be on elevators that you actually have to test it??! You even pointed out that a "test" took systems out for a weekend before. When's the last time there was a real power outage over there that lasted that long?

Here's an example of a similar program in another state: here Ours was slightly different in that we were given 24 hours advanced notice. If my post has made you suspicious at all, simply search your power provider's online site to see if they have a similar program. Maybe someone in your company is getting a nice "bonus" for decreasing your power, while costing the company more money in lost work.

 

[MmmSkyscraper]

Originally posted by: SparkyJJO
Possible they were testing the power at the servers and making sure all the circuits that were to remain up were still up.

LOL, props for your username in this thread.

 

[nerp]

It does seem lame that you prepare for a power outage by powering everything down first. Unfortunatley, that's not how unexpected power outages are handled. And generators need to be tested with the whole place up and running. Creating a dreamy-easy scenario to test equipment is hardly stress testing.

 

[DrPizza]

I did some more searching; it appears the programs I mentioned are pretty common, with the kickbacks ranging from a few thousand dollars to (*gasp*) a couple hundred thousand Although, in that link, I'm not certain that what they're referring to is exactly the same thing.

 

[Mark R]

My understanding is that:
1. The generators need to be tested under 'typical' load conditions to ensure that they maintain output, don't overheat, and that the fuel supply systems don't run dry. And that 2 hours should be sufficient to test this.
2. 2 hours allows all 'critical' systems to be tested, even if they aren't in use at the time. In some departments, I know the staff went around with a portable lamp, testing every single 'essential' receptacle, to make sure that it was powered, to identify problems where receptacles had been incorrectly labelled/wired.
3. There was a 2 minute 'dead' period after they disconnected mains power and before they started the generator - supposedly so that any critical equipment on UPSs (which couldn't be powered down) could be tested (in a relatively controlled manner, as considerable notice had been given) to ensure that the UPSs operated correctly. [I'll congratulate DrPizza for correctly deducing that this is a hospital - so there is plenty of equipment that is *critical* which doesn't necessarily mean IT].
4. I don't really understand the reasoning for shutting down all the servers. This is particularly so, as we have migrated to partial electronic records (e.g. all X-ray is digital and electronically viewed, as are all blood test results). However, it may be related to the chaos last time, where several servers were totally b0rked by the test. However, it's not clear whether these systems remained at all accessible during the outage (I'm guessing that as several core servers were taken down, they might not remain available - even though we have redundant off-site application servers) - instructions were not to use any computers for the duration of the test, so even if IT had managed to keep some of their systems up whether those systems would remain accessible.
5. Even so, the generators are woefully underpowered - hence only 1 elevator (reserved for 'code' calls). I was grateful to have a new office this time. Last time I worked on both ground and the 20th floor. That was fun. The elevators strike me as a real problem. 'Hi. Is that the X-ray department? Dr Doe here from 20 C - Mr X is very unwell, can you bring a portable X-ray machine to do a Chest X-ray?'. 'No. The elevators are all grounded, we can't bring the machine up.' No doubt that some manager somewhere thought this was an acceptable solution.

 

[randay]

they were installing dynamite in the steel suppor structures. a "power failure" test was needed to insure that no suspicions were aroused.