Power went out while formatting drives for Software RAID 5

[stufff]

I was formatting 4 2TB SATA drives (all brand new) for a software RAID 5 setup. I'm re-purposing an old Dell desktop to be a media center computer for the living room, and home media file storage for the other computers in the house. The Dell Motherboard only supports RAID 0 or 1, and it wasn't worth it for my purposes to buy a real raid controller, so I went with Software RAID 5 using a hack that allows XP Pro to do it by moving over some Windows Server files. My OS is on an IDE drive.

I went with a full format for some reason, I wasn't completely sure if I should or not, so figured I'd better be safe than sorry. Due to the slower computer and slower 5400 rpm drives, the formatting was going at a rate of about 20% every 12 hours. I went to bed last night and it was at about 77%, and woke up when I heard the power go out for a second this morning.

In Windows XP disk management I saw that all the drives said "Regenerating." The appeared to be taking just as long to regenerate as they did to format. I don't understand this, because they weren't even done formatting in the first place and there wasn't any data there, so I don't understand what data it would be regenerating.

I tried deleting the RAID, doing a quick format on each disk individually (at which point it showed them all as "healthy" instead of "regenerating"), and then creating a new RAID, but as soon as I created a new RAID it went back to slowly "regenerating."

My first question is, do I need to be worried about the disks being permanently damaged as a result of the power loss during formatting?

My second question is, how do I get it to stop trying to "regenerate" the RAID so I can just start using it. Is it necessary to do a full format or will a quick format be good enough?

 

[Nothinman]

The Dell Motherboard only supports RAID 0 or 1, and it wasn't worth it for my purposes to buy a real raid controller, so I went with Software RAID 5 using a hack that allows XP Pro to do it by moving over some Windows Server files. My OS is on an IDE drive.

That's a really bad idea, any update or patch from MS could at the very best stop you from accessing that volume and at the worst cause data corruption problems.

I tried deleting the RAID, doing a quick format on each disk individually (at which point it showed them all as "healthy" instead of "regenerating"), and then creating a new RAID, but as soon as I created a new RAID it went back to slowly "regenerating."

Of course it did. The parity stripe needs built regardless, all you did was restart the process.

My first question is, do I need to be worried about the disks being permanently damaged as a result of the power loss during formatting?

No more than you would any other drive from losing power unexpectedly.

My second question is, how do I get it to stop trying to "regenerate" the RAID so I can just start using it. Is it necessary to do a full format or will a quick format be good enough?

You don't. That's how RAID5 works. A quick format should be good enough as the OS can't see the real logical sectors anyway. If you're unsure as to the individual drives' health you should test them individually.

 

[Emulex]

raid-5 without a battery backup and flash back write cache (battery) is insane. its not meant for consumer use.

 

[stufff]

That's a really bad idea, any update or patch from MS could at the very best stop you from accessing that volume and at the worst cause data corruption problems.

Yeah, I've considered that. I've read that when XP SP3 came out everyone who had been doing this in SP2 found that their RAID was missing. But copying the server files back to the same place fixed the issue and all their data was still intact. As a precaution, I've disabled automatic updates on this machine after getting everything up to date.

I realize this isn't as good a solution as a dedicated RAID controller, but it's a lot cheaper, and I'm not storing any critical data. It's almost entirely going to be data I have in some other form on optical media, I'm just consolidating it for easy access. It will piss me off and annoy me for a couple days if I lose everything, but it won't be the end of the world.

Of course it did. The parity stripe needs built regardless, all you did was restart the process.

I'm not sure I understand this. Why is there parity data when there isn't data on the drives yet? Please forgive my ignorance on this, but I thought it wrote the parity data as actual data was written to the drives... are you telling me it keeps parity data for free space? Even if that were the case, I would assume that the calculations for creating parity data for free space would be a lot quicker.

I assume if I had done a quick format from the beginning it wouldn't have spent 2+ days doing something and I could have used the drives immediately, right?

So why does it "remember" that it was regenerating after I've dissolved the RAID, formatted each drive individually, and created a new RAID? Shouldn't it act as though I just created a raid for the first time with a quick format?

I would really like to start freeing up space on other computers, so the fact that it's going to take another 2+ days to finish regenerating non-existent data puts a real damper on my plans for the holiday weekend. If I started moving data over to it now, would that cause any problems (other than the write speed performance hit from the drives doing whatever they do while regenerating)?

No more than you would any other drive from losing power unexpectedly.



You don't. That's how RAID5 works. A quick format should be good enough as the OS can't see the real logical sectors anyway. If you're unsure as to the individual drives' health you should test them individually.

That's good to know. For some reason I was under the impression that power loss during formatting was more likely to cause serious permanent damage to a hard drive. I will give them each an individual test, but I'm reluctant to stop and have to completely restart the regenerating process.

Thank you for your reply!

 

[stufff]

raid-5 without a battery backup and flash back write cache (battery) is insane. its not meant for consumer use.

I fully intend to get a UPS before I start putting all my data on it, I was hoping to find a deal on one this weekend. I just didn't expect the formatting to take so long, and I haven't had the power flicker out in about 4 months, so I thought I would be safe.

I would just like to reiterate that I'm not storing any important data on this machine. Everything on it is something that I already have on optical media (DVDs, music CDs, etc), so the risk of data loss was not worth the cost of buying enterprise hardware.

 

[ch33zw1z]

I would just like to reiterate that I'm not storing any important data on this machine. Everything on it is something that I already have on optical media (DVDs, music CDs, etc), so the risk of data loss was not worth the cost of buying enterprise hardware.

Then RAID 5 is not for you, it's a waste of time and money. Get the biggest drive you can and dump the data onto it...save yourself the headache that is a.k.a RAID 5 for home use.

also, read this: http://en.wikipedia.org/wiki/RAID

 

[Nothinman]

Yeah, I've considered that. I've read that when XP SP3 came out everyone who had been doing this in SP2 found that their RAID was missing. But copying the server files back to the same place fixed the issue and all their data was still intact. As a precaution, I've disabled automatic updates on this machine after getting everything up to date.

I realize this isn't as good a solution as a dedicated RAID controller, but it's a lot cheaper, and I'm not storing any critical data. It's almost entirely going to be data I have in some other form on optical media, I'm just consolidating it for easy access. It will piss me off and annoy me for a couple days if I lose everything, but it won't be the end of the world.

Using XP in the first place is a bad idea since it's EOL'd, but using it in a hackish way like this is even worse. There are plenty of free alternatives that are actually maintained and will get you data redundancy with software RAID that's much more flexible and reliable.

I'm not sure I understand this. Why is there parity data when there isn't data on the drives yet? Please forgive my ignorance on this, but I thought it wrote the parity data as actual data was written to the drives... are you telling me it keeps parity data for free space? Even if that were the case, I would assume that the calculations for creating parity data for free space would be a lot quicker.

I'm not 100% sure how Windows software RAID does it, but in other systems the software RAID is below the filesystem so it doesn't know anything about the data. So yes, it builds parity data for everything there, even unallocated space containing random bits.

I assume if I had done a quick format from the beginning it wouldn't have spent 2+ days doing something and I could have used the drives immediately, right?

Pretty much, the rebuild would still happen so it would've been slow but it would've been usable.

So why does it "remember" that it was regenerating after I've dissolved the RAID, formatted each drive individually, and created a new RAID? Shouldn't it act as though I just created a raid for the first time with a quick format?

It doesn't remember, it's regenerating the parity stripe just as it would any new array.

I would really like to start freeing up space on other computers, so the fact that it's going to take another 2+ days to finish regenerating non-existent data puts a real damper on my plans for the holiday weekend. If I started moving data over to it now, would that cause any problems (other than the write speed performance hit from the drives doing whatever they do while regenerating)?

AFAIK it won't cause any more problems than you're creating for yourself by doing this with a hacked up XP.

 

[RebateMonger]

You might take a look at Windows Home Server. It allows you to use disks of any type and any capacity and combine them into a single storage pool. You can expand the size of the pool at any time with a few mouse clicks. You can enable folder-level disk redundancy (a folder-level RAID 1 equivalent) if you want. Personally, I prefer just making backups of important files.

If a disk fails, you'll only lose data on that particular disk. Also, the disks and files are standard NTFS format, so you can attach them to another PC for data recovery if you desire.

The software costs $100, but allows you to use all your old disks, doesn't require Enterprise-grade disks to build the storage pool, and doesn't require a RAID controller or the use of Dynamic disks. And it will make REALLY good automatic daily full system image backups of up to ten client PCs.

 

[stufff]

You might take a look at Windows Home Server. It allows you to use disks of any type and any capacity and combine them into a single storage pool. You can expand the size of the pool at any time with a few mouse clicks. You can enable folder-level disk redundancy (a folder-level RAID 1 equivalent) if you want. Personally, I prefer just making backups of important files.

If a disk fails, you'll only lose data on that particular disk. Also, the disks and files are standard NTFS format, so you can attach them to another PC for data recovery if you desire.

The software costs $100, but allows you to use all your old disks, doesn't require Enterprise-grade disks to build the storage pool, and doesn't require a RAID controller or the use of Dynamic disks. And it will make REALLY good automatic daily full system image backups of up to ten client PCs.

I'm reading about this and this "drive extender" sounds like exactly what I want. The only reason I went with RAID 5 is because I wanted all my storage to appear as one one drive to programs and networked computers/devices, but didn't want to lose everything in the event one drive failed. This would even free up an extra drive for use. This sounds absolutely amazing, thank you!

 

[RebateMonger]

Just as a warning, be aware that, as of yesterday, there's a ton of discussion about the (not yet released) WHS V2 (currently called "Vail"). For various reasons, MS announced it was dropping the Drive Extender technology in the new product. At this moment, there are about 1200 negative votes and hundreds of negative comments on MS' "Connect" web site about this development.

Regardless, the current WHS V1 is up to Service Pack 3 and is being used by a ton of people. It's still under full Microsoft support and uses Windows Server 2003 as its basis, a product that is still used by a huge number of businesses. It's not clear when WHS V2 will be released and its features are in flux. I'd concentrate on the features of WHS V1, which is real and shipping and works great for the vast majority of users.

I have around twelve WHS Servers installed at various homes and businesses and the owners are extremely happy with them. They are almost maintenance-free and they should continue to meet the needs of their owners for several more years.

 

[Nothinman]

I'm reading about this and this "drive extender" sounds like exactly what I want. The only reason I went with RAID 5 is because I wanted all my storage to appear as one one drive to programs and networked computers/devices, but didn't want to lose everything in the event one drive failed. This would even free up an extra drive for use. This sounds absolutely amazing, thank you!

Just be aware that MS axed the drive extender from WHS v2, so you'll be stuck with v1 until you replace it with something completely different.

 

[Emulex]

you can mount drives on a mount point then share that drive?

if you don't want to lose data - back it up. period end of story.

raid-5 is something i don't want in enterprise honestly. i've seen too many double drive failures in my lifetime to realize how lame it is.

 

[Nothinman]

you can mount drives on a mount point then share that drive?

Except that doesn't pool the space so you have the same data shuffling issues you would with multiple drive letters.

 

[stufff]

Except that doesn't pool the space so you have the same data shuffling issues you would with multiple drive letters.

Yes, this is exactly the problem I was hoping to avoid. I have multiple users in my home, many of whom would not understand or be annoyed at having to remember what is where, what should go where, etc.

Also, as, for example, our collection of DVD TV shows grows, even with 2TB allocated to TV shows alone we would eventually have to split it into two drives, which would defeat the seamlessness to the end user I am going for. The problem would be even worse if we ever get on the Bluray bandwagon.

 

[stufff]

Just be aware that MS axed the drive extender from WHS v2, so you'll be stuck with v1 until you replace it with something completely different.

Ouch. This feels like the time I discovered Firefly a week before it got canceled.

 

[Nothinman]

Yes, this is exactly the problem I was hoping to avoid. I have multiple users in my home, many of whom would not understand or be annoyed at having to remember what is where, what should go where, etc.

Also, as, for example, our collection of DVD TV shows grows, even with 2TB allocated to TV shows alone we would eventually have to split it into two drives, which would defeat the seamlessness to the end user I am going for. The problem would be even worse if we ever get on the Bluray bandwagon.

Are you married to Windows already? A Linux distro can do this easily with software RAID and/or LVM manually. Or if you don't want to do it manually you can use one of the NAS distributions like FreeNAS, which is technically FreeBSD and not Linux but that doesn't matter much since you'll be doing everything through it's web interface.

 

[stufff]

Are you married to Windows already? A Linux distro can do this easily with software RAID and/or LVM manually. Or if you don't want to do it manually you can use one of the NAS distributions like FreeNAS, which is technically FreeBSD and not Linux but that doesn't matter much since you'll be doing everything through it's web interface.

I don't really know enough about Linux to be comfortable with it, I've tried using Kubuntu as my main OS for my laptop and managed to break it to the point where I had to reinstall no less than 3 times before I gave up. My problem is I am never satisfied with things as they are by default and am compelled to tweak them to my liking, with Linux it just results in me breaking everything.

I would also need it to be easy and transparent for other network users. They would need to be able to use mapped network drives easily as they would with a Windows share, the Xbox running XBMC would need to be able to access it, I'm planning on running XBMC on the machine itself and I'm not sure if there is a Linux port, We're getting an Xbox 360 for another room and I would want it to play nice with that as well, and I would need it to run Orb for streaming to my mobile phone. I'd like to be able to remote desktop into it from my windows mobile phone as well, but I guess I could settle for VNC.

I guess I am a little married to Windows.

 

[Nothinman]

I don't really know enough about Linux to be comfortable with it, I've tried using Kubuntu as my main OS for my laptop and managed to break it to the point where I had to reinstall no less than 3 times before I gave up. My problem is I am never satisfied with things as they are by default and am compelled to tweak them to my liking, with Linux it just results in me breaking everything.

I would also need it to be easy and transparent for other network users. They would need to be able to use mapped network drives easily as they would with a Windows share, the Xbox running XBMC would need to be able to access it, I'm planning on running XBMC on the machine itself and I'm not sure if there is a Linux port, We're getting an Xbox 360 for another room and I would want it to play nice with that as well, and I would need it to run Orb for streaming to my mobile phone. I'd like to be able to remote desktop into it from my windows mobile phone as well, but I guess I could settle for VNC.

I guess I am a little married to Windows.

I don't know about the XBox or Orb stuff as I don't have an Xbox and my phone runs Android, but Samba will make it show up as just another server on the network and you can map drives just as well so the other users wouldn't notice a thing. I think I remember reading about XBMC stuff for Linux, but I haven't had a reason to look into it yet.

 

[exdeath]

The RAID5 and the file system are completely separate things. The RAID5 is just doing what it does regenerating, regardless if there is anything on the drives or not. If you pull a hot swap drive out of a brand new blank RAID5 right after creating it, it will do the same thing, that's what RAID5 does.

After it's done, better yet even now while it's regenerating, you still have an unusable file system with an interrupted full format. Just do a quick format and you're done. In fact you can quick format it and use it while it's regenerating, it will just be extremely slow.

 

[Emulex]

i would never do raid-5 without a hot spare. Better yet i'd prefer a leveling v-raid type system where the hot spare is in use but reserved space is kept for a double drive failure in which case all the drives shift data around to keep the 1 or 2 drive failure mode as long as you have drive space. (high end raid systems use virtual raid like this to level) so hot spares are actually in use and add to performance.

i'd bet 50% of raid-5 failures are during rebuild. which is why i only recommend striped raid-1 now (raid-1 pairs striped as raid-0 AKA raid-10).

 

[taltamir]

RAID6 is flat out superior in every way shape and form to RAID5 with a hot spare.

@OP: RAID5 is a really really bad idea for what you are doing. it gives terrible performance AND terrible data safety.
if your data in unimportant, then don't use raid at all, or use raid0.

 

[Nothinman]

RAID6 is flat out superior in every way shape and form to RAID5 with a hot spare.

@OP: RAID5 is a really really bad idea for what you are doing. it gives terrible performance AND terrible data safety.
if your data in unimportant, then don't use raid at all, or use raid0.

Sure there are better options, but RAID5 performance is far from terrible and the data safety is still pretty good. Millions of servers have been running RAID5 for years without issue. Just because better options have been created doesn't automatically mean it's utter shit.

 

[Emulex]

raid-10 is far superior to either. i'm talking raid-1 (2 drives) then striping all of them raid-10 style. massive IOPS on write. drives are alot cheaper now than ever.

 

[Nothinman]

raid-10 is far superior to either. i'm talking raid-1 (2 drives) then striping all of them raid-10 style. massive IOPS on write. drives are alot cheaper now than ever.

And for a home server IOPS are usually at the very bottom of the list of requirements.

 

[taltamir]

Sure there are better options, but RAID5 performance is far from terrible and the data safety is still pretty good. Millions of servers have been running RAID5 for years without issue. Just because better options have been created doesn't automatically mean it's utter shit.

the issue is that RAID6 and RAID5 + Hot spare have an equal number of drives giving an equal amount of usable space. Only raid6 is far more resistant to failure... Only reason to have a hot spare is if you are concerned about a double failure, and a raid6 array is much safer as it requires triple failure to lose data.

I am not saying RAID5 is a horrible choice for servers (since they use a quality controller card that can do it properly), but IF you are going to add a hot spare, you might as well just go ahead and make it raid6.

raid-10 is far superior to either.

why yes, yes it is. but i was specifically considering the hot spare issue.