Fire in server room !

[iamgenius]

Unfortunately, we had a fire in a big server room. The AC was the culprit. There were like 20 servers in there. The servers didn't actually get burned, but was only affected by the thick black smoke(Soot). Parts of the servers were already starting to rust because of oxidization.
Internal parts are all covered with a thick black layer of soot.

How would you assess the situation? There servers are no longer usable??

Thick smoke I think will cause a short-circuit, right? If this is the case, then all electronics inside will be just bad. It will be just like dipping the boards in water. right?


Thanks.

 

[Red Squirrel]

I'm not an expert other than the fact that I have a small server room at home and have worked in professional (small) ones so someone may have better advice but here's my thoughts:

If cleaned I think they will be fine, but personally I would take the entire server room offline, which on it's own is a huge risk as there WILL be hardware failures simply from the fact that they've been on so long and now got to cool off. Even at home I do everything to NOT have to shut down my servers for this reason. But in this case it has to be done. While they are off I would then clean one at a time in a separate room with an air compressor, maybe even alcohol? Someone may be able to give a better method than me. Not sure what is the best way to deal with soot/smoke damage.

Once all cleaned I'd probably want the actual room to be cleaned as much as possible from stray soot. New AC unit installed, run it for a while and change filter and repeat till it comes clean. Maybe even get air quality test done before and after just to compare. Once things check out start to move the servers back in and rack and connect them, and turn them up one by one and hope for the best. Have spare hard drives on site as it's inevitable that some will fail.

This would also be a good time to have a separate team working on any required UPS upgrade or maintenance, or anything else power related as you mitigate the risk by having all servers already offline.


I work at a noc and we have a server room at one of our sites and twice in a row one of us got a fire alarm + AC fail alarm at same time. This is a scary scenario because just a fire alarm, 99% of the time it's false, so while you do treat it as real and have people go on site, you know it's probably going to be false. But when you have equipment failure at the same time it makes you figure there more than likely is something very wrong. Turns out the AC unit had dumped it's glycol load in the server room and it caused a plume of smoke which activated the fire detector. A week later I'm on night shift and get the exact same incident happen, but to the other AC unit. What are the odds. But on that note if there is another AC unit in there of same brand/model/vintage I would get it replaced too just to be on the safe side. This sounds like a decent sized disaster that will require lot of downtime so may as well do all the big work now. If you can't afford the downtime I'd look at setting up a temp server room by stacking the servers in another room with a portable AC unit or something while the work is being done.

 

[mindless1]

Thick smoke won't by itself cause a short circuit but possibly carbon left behind. The more likely problem is fouling of electrical contacts.

It won't be like dipping the boards in water. First, water is practically non-conductive at the low voltages in a PC except the PSU high voltage side. Second, the more likely cause of problems from water exposure is they typically use a water soluble flux and traces of it remain behind so when a component gets wet the flux is activated and leeches away at the metal creating conductive pools of liquid. Dipping the whole thing in water instead, you won't have small high concentration pools of flux until it's removed and starts drying in which case a water rise agent or rinse in alcohol, etc. can be used.

However, none of the above really matters if you had them insured. Insurance isn't likely to want to go the path of paying an existing employee to strip, clean, reassemble and test them all.