Dirty Power - What to do and what gets affected?

[momeNt]

So I am being told at my office that we experience what essentially amounts to dirty power. We experience major fluctuations in voltage and our UPS kicks on very often and it registers significant power events at least once a week, with minor power events in the hundreds a week.

Apparently this has caused many of our hard drives which were in raid 5 to go bad, we have replaced 4 drives so far this year and now we are replacing our server. We have not addressed the dirty power issue though. What I'm being told is that there are so many power events that our UPS sometimes becomes drained and ends up passing through this dirty power and isn't able to clean it.

Question 1: Is a hard drive really the first in line to be blown from dirty power? I would think the PSU would manage to clean the incoming voltages enough that the first in the line to blow would actually be the PSU and not any component.

Question 2: Is there something we can do in our UPS that makes it not pass through dirty power in the event that it is drained from multiple events not allowing it to recharge in time to be delivering clean steady volts.

Is what my IT telling me legitimate? I don't understand why a new server should be purchased if the power problem isn't addressed, if this is a legitimate threat to our server, shouldn't this be addressed with either a bigger UPS or possible a natural gas generator that kicks on in the event of bad voltages?

 

[Zap]

1) If the computer's PSU passes through the fluctuations, then the HDD may be affected. If you look at the motherboard you will see a bunch of VRMs and capacitors to make sure the motherboard components have the exact voltages they need. HDDs don't. IDK this for sure, but it is plausible. The solution of course is to make the incoming power cleaner, and use a PSU that does a better job of regulation. Of course there could be other reasons why several HDDs dies in a short period of time.

2) Use a UPS that does line conditioning, sometimes marketed as "line interactive" or "sine wave output" (usually a simulated sine wave) or "AVR" (Automatic Voltage Regulator). Just make sure the one you guys get regulates all the time, not just when it is on battery. Read the specs carefully.

 

[momeNt]

1) If the computer's PSU passes through the fluctuations, then the HDD may be affected. If you look at the motherboard you will see a bunch of VRMs and capacitors to make sure the motherboard components have the exact voltages they need. HDDs don't. IDK this for sure, but it is plausible. The solution of course is to make the incoming power cleaner, and use a PSU that does a better job of regulation. Of course there could be other reasons why several HDDs dies in a short period of time.

2) Use a UPS that does line conditioning, sometimes marketed as "line interactive" or "sine wave output" (usually a simulated sine wave) or "AVR" (Automatic Voltage Regulator). Just make sure the one you guys get regulates all the time, not just when it is on battery. Read the specs carefully.

Thanks. #2 I will definitely inspect our current UPS and make sure that it does that and if it doesn't I will recommend we purchase one immediately. I see no reason to crap out $6k server by not fixing the root cause of why we had to replace old one.

With how expensive hard drives are currently the fact that we had to purchase a new server with 4 drives after just replacing 4 drives that now are bad is making me ready to explode. Thanks very much for the response.

 

[Concillian]

We have had issues with this at work with a specific circuit that had sensitive testing equipment on it. It ended up being a transformer was tapped for 220V instead of 240V (110 vs. 120v on the single leg circuits) and the equipment would do okay until it pulled about 7A or so, then line voltage would sag enough that voltage would get pulled down and stuff would start resetting.

We were able to temporarily fix the issue with a line filter that also had a voltage boosting transformer in it. It was made for home audio, and would boost voltage some % when it dropped below a certain threshold. We had tried other UPS and line filter options and that was the only one that worked.

Your situation may be different than ours, just sharing what worked for us until we were finally able to find the root of the issue and tell the facilities group what they had done wrong and how to fix it, since they couldn't figure it out themselves.

 

[momeNt]

Hmm. We have a APC Smart UPS 2200, it has AVR, so I suppose I'm still at a loss for why dirty power has anything to do with us losing our drives or server.

Perhaps the number of events make the UPS unable to regulate or we are trying to make the UPS regulate too much of a spread without going into power down off battery.

 

[Zap]

With how expensive hard drives are currently the fact that we had to purchase a new server with 4 drives after just replacing 4 drives that now are bad is making me ready to explode.

If the drives are old enough to be out of warranty, then it is time to replace them anyways. If not, then buy replacements to use for now, RMA the failed drives, then keep the warranty replacements around as spares in case more drives fail.

Perhaps the number of events make the UPS unable to regulate or we are trying to make the UPS regulate too much of a spread without going into power down off battery.

Does the UPS report these things? Some do keep track (with their software installed and the UPS hooked up to the PC through serial/USB) and log information such as incoming voltages. If they are seriously out of whack, then maybe it is indeed overwhelming the UPS.

Alternately... how old is the UPS? Electronics do degrade over time.

 

[momeNt]

If the drives are old enough to be out of warranty, then it is time to replace them anyways. If not, then buy replacements to use for now, RMA the failed drives, then keep the warranty replacements around as spares in case more drives fail.



Does the UPS report these things? Some do keep track (with their software installed and the UPS hooked up to the PC through serial/USB) and log information such as incoming voltages. If they are seriously out of whack, then maybe it is indeed overwhelming the UPS.

Alternately... how old is the UPS? Electronics do degrade over time.

It's highly possible the initial failed drive were out of warranty, fairly sure the server is older than 5 years and can't imaging most drives would have a longer than 5 year warranty, the new drives definitely are not. I will make sure that they are RMAd and we can have them as spares.

I'm going to have to investigate these things further. It is a small company but I am not directly connected to the situation, I took it upon myself because I noticed we replaced the server but not the supposed underlying cause and it's driving me nuts knowing we could be setting ourselves up for another fiasco.

I have one of the managers tracking down the issue Concillian mentioned with the transformer taps and I will try and get the log from the UPS as well as the age, I believe it may have been bought with the server in which case it would be about 6 years old. I will check for sure though. Not sure what the average replacement cycle is for a UPS.

Thanks so far for the help.

 

[VirtualLarry]

TBH, 5-6 years is usually time for both HDs and UPS batteries to fail. The way I see it, it may just have been their time.

 

[momeNt]

TBH, 5-6 years is usually time for both HDs and UPS batteries to fail. The way I see it, it may just have been their time.

Yes we are close to 5 years, and over its life this UPS has probably had to do more than most as far as shutdowns and line regulation. So I will recommend we investigate if any of the new drives were damaged from not replacing an aging UPS as well as bringing out our electrician to investigate the power issue to see if we can maybe give the next UPS an easier job going forward.

I realize now that maybe our IT dropped the ball by letting both the UPS and hard drives get to nearly 5 years old without a replacement plan as well as verifying that the UPS would be able to handle its normal workload.

 

[Zap]

I realize now that maybe our IT dropped the ball by letting both the UPS and hard drives get to nearly 5 years old without a replacement plan as well as verifying that the UPS would be able to handle its normal workload.

That actually sounds quite normal. Many businesses regard IT expenditures as purely extra cost, and avoid it as much as possible.

5 years old? I'd just chalk it up to age. If IT uses it as an "excuse" for a new server, make sure that "excuse" covers a new UPS too if that UPS is as old as the old server.

 

[momeNt]

Posting to close issue and let you guys know what we ended up doing.

I researched the UPS a little more, found out that it is a transfer UPS (APC smart model) not the Smart On-Line model. Zap told me to look for that, but I was mistaken in thinking the AVR mode offered what he mentioned. AVR only kicks in on this model if it goes outside of the range - generally 106-127. I think in the times that we were getting bad voltage, it was passing through inside that range without the AVR kicking in.

On-Line UPSs sized for our needs are somewhere around $1,400, but given that our UPSs still are functioning okay, we went with a Tripp-Lite line conditioner for ~$260. Gives 120 all the way from 89-147 or something. We also threw in some vertical PDUs because the rack was really messy looking with 3 UPSs (1 rack, 2 tower...) and wires going everywhere.

Thanks for the help.

 

[Modelworks]

You need to contact the utility. They are not responsible for the occasional outage but they are responsible to provide power at the utility level. Each area has its own rules but it is generally that the voltage has to stay between a set amount and any wild fluctuations have to be fixed by the utility. Contact them and tell them the problem and requests that a monitoring device be installed at the location. It will allow them to monitor the power and determine what the issue is. Many people are not aware that a utility has no way of knowing if the power at your meter is low or high or having issues, they only know the conditions of the last monitoring point, usually the substation. If local lines are degraded, there is bad transformer, etc the only way they know is if you complain.

If you are having brownouts or issues several times a week then something is wrong.