Server Data, Raid 5 Initialization Time vs. Rebuild Time

[Anteaus]

I currently maintain a home file server which contains 15TB of storage using a spanned partition over 3 4TB Hdds and 3TB HDDs. I maintain two independent backups which I sync weekly so my data is "reasonably" secure. I am planning on upgrade the server soon to contain 20TB of storage, using as many 4TB drives as necessary depending on the method.

This isn't a "which RAID level should I use" thread, though it is related to RAID. I would like some level of cost effective redundancy purely to buy time to sync to my latest backup. Downtime isn't an issue once that is achieved.

My thinking is that I would like to use RAID 5 purely as a stop gap measure, which no intention of rebuilding should a drive fail. For example, lets say I use 6 X 4TB drives to build a 20TB RAID 5 array. At some point one of the HDDs dies and the array becomes degraded. All I'm looking for is time to update my backups. Afterward, I will replace the drive (probably days later), reinitialize, and restore from backup instead of going through the rebuild process. I'm thinking this will remove some of the risk of rebuild failure since I won't need to successfully read every sector of every drive. I will need to rely on my backups at that point, but that is why I have two.

In all, what I'm looking for are real world experiences of how long it takes to initialize a RAID 5 array versus rebuilding it. If it isn't noticeably faster to reinitialize instead of rebuilding, then its not really worth the hit in write performance or the time investment. I'll just stripe it and make sure I'm consistent with my backups.

I don't mind working without a net (RAID 0, JBOD), but running so many drives makes me wonder about my chances of failure. My other thought is to use something similar to flex raid to create a parity drive. Perhaps that is the better way to go regardless.

Opinions?

 

[dave_the_nerd]

In most cases, it's faster to wipe and restore than it is to rebuild an array.

Syncing data to and from a device over a network is easy math - 20*2^20 MB @ ~80 MB/sec = 262144 seconds. So about three days.

Rebuild time/speed just depends on your server and raid controller. For rebuild times, I've heard people cite anywhere from 1TB/day to... well, a lot more, or a lot less. Anandtech NAS reviews usually include a RAID rebuild benchmark. But the data is accessible that whole time.

So, as the good book says. "RAID is not a backup, RAID is for uptime." 3 days to sync back your data is three days that it's not accessible and 3 days that your dependent services are unavailable.

I'm thinking this will remove some of the risk of rebuild failure since I won't need to successfully read every sector of every drive. I will need to rely on my backups at that point, but that is why I have two.

I don't want to dissuade you from keeping two backups, but most software RAID implementations work around that rebuild failure problem.

What I would suggest is RAID-6. (RAIDZ2, double parity, whatever you want to call it.) Then your data is still parity protected and accessible while you rebuild the failed drive.

 

[Anteaus]

In most cases, it's faster to wipe and restore than it is to rebuild an array.

Syncing data to and from a device over a network is easy math - 20*2^20 MB @ ~80 MB/sec = 262144 seconds. So about three days.

Rebuild time/speed just depends on your server and raid controller. For rebuild times, I've heard people cite anywhere from 1TB/day to... well, a lot more, or a lot less. Anandtech NAS reviews usually include a RAID rebuild benchmark. But the data is accessible that whole time.

So, as the good book says. "RAID is not a backup, RAID is for uptime." 3 days to sync back your data is three days that it's not accessible and 3 days that your dependent services are unavailable.



I don't want to dissuade you from keeping two backups, but most software RAID implementations work around that rebuild failure problem.

What I would suggest is RAID-6. (RAIDZ2, double parity, whatever you want to call it.) Then your data is still parity protected and accessible while you rebuild the failed drive.

Thanks for the feedback. For me the question is whether the initialization/rebuild times are worth potentially losing a few files should i lose the volume between backup cycles. I'm basically in agreement that it is faster to just wipe and restore. I think I may just continue running without redundancy. Again, I'm fine with the downtime if I need to restore. It's the time up front for initialization that I was interested in. I don't want to wait for three days to initialize just to spend another 1-3 days copying data from backup. Some will think I'm nuts for running a 20TB Raid 0 array, but I guess it all depends on what you are trying to do. The write penalties and additional cost for RAID 6 aren't worth it for my usage. I'd go Raid 10 if money were no object.

I may implement some sort of dump drive, where all new files are copied in a secondary location as well as the array when between backup cycles.

 

[master_shake_]

no to raid 5 with that big of drives.

big no.

raid 10 or nothing

rebuild of a raid 5 with that big of drive is more than likely going to cause a failure.

are you using a hardware card?

raid cards are cheap these days.

http://www.ebay.com/itm/L3-25121-61...788479?hash=item2116768b7f:g:CtoAAOSwFV9Xw16i

i use one of those with 2 virtual drives.

one 22 2tb drive raid 6 and on 12 2tb drive raid 6.

 

[dave_the_nerd]

no to raid 5 with that big of drives.

big no.

raid 10 or nothing

Umm...

one 22 2tb drive raid 6 and on 12 2tb drive raid 6.

 

[master_shake_]

Umm...

yeah no to a 22 drive raid 10. or 12 drive raid 10.

my raid card supports 0,1,5,6,10,50,60 so i chose raid 6.

redundancy and maximum amount of space.

 

[Anteaus]

yeah no to a 22 drive raid 10. or 12 drive raid 10.

my raid card supports 0,1,5,6,10,50,60 so i chose raid 6.

redundancy and maximum amount of space.

That would be a reasonable way to go for me except for the heavy write penalties and tendency to destroy consumer HDDs. I have no intention on actually rebuilding an array with a failed disc. I'm looking for a solution that buys me enough time to update my backup.

My original thought was to use RAID 5 if/until a disc failed, backup the array, then nuke it and reinitialize. From what I understand, the wear and tear on the HDDs during the rebuild process is why RAID 5 isn't preferred. Again, I don't care if the array fails completely as long as my backups are up to date.

 

[dave_the_nerd]

The wear and tear of a rebuild isn't necessarily any worse than normal server-is-busy operations. (Seeks around the disks, reads everything once.) The potential problem is the unrecoverable error rate vs. the size of the disks and how the RAID controller / software handles those UREs. (Some don't deal with them well - they hit an error and just shut down. Some can try to recover and keep going, but the data in that sector is still lost.)

There's nothing really wrong with parity RAID (5 or 6) but conventional wisdom is to make sure that you use RAID6 if your drives are over 700GB in size or so, because of the URE/recovery behaviors.

What do you use this array for? That would actually determine whether or not the write penalty is meaningful, as well as what other options you may have regarding your backups / scheduling.

 

[Anteaus]

The wear and tear of a rebuild isn't necessarily any worse than normal server-is-busy operations. (Seeks around the disks, reads everything once.) The potential problem is the unrecoverable error rate vs. the size of the disks and how the RAID controller / software handles those UREs. (Some don't deal with them well - they hit an error and just shut down. Some can try to recover and keep going, but the data in that sector is still lost.)

There's nothing really wrong with parity RAID (5 or 6) but conventional wisdom is to make sure that you use RAID6 if your drives are over 700GB in size or so, because of the URE/recovery behaviors.

What do you use this array for? That would actually determine whether or not the write penalty is meaningful, as well as what other options you may have regarding your backups / scheduling.

The array is predominately for media use, so the write penalties aren't so problematic; however. the idea of waiting for many days for an init or rebuild to take place isn't good. All things being equal, I'd rather go with performance and simplicity if I have the option. The issue I have with Raid is that its generally seen as an either/or situation. Either you use raid with all of its caveats or you find another way to do it. If MS would build a parity option into Storage Spaces similar to how Unraid does it I think it would be killer.

I do see your point about Raid 6. If what your saying about wear and tear is true, then the virtue of Raid 6 is that, during a rebuild, you would need to have a URE happen on both parity copies at the same time for a rebuilt to fail, assuming a second, full drive failure isn't imminent? I hope I worded that right.

 

[master_shake_]

it took 17 hours to rebuild my 12 disk array when a drive failed

 

[dave_the_nerd]

The array is predominately for media use, so the write penalties aren't so problematic; however. the idea of waiting for many days for an init or rebuild to take place isn't good. All things being equal, I'd rather go with performance and simplicity if I have the option. The issue I have with Raid is that its generally seen as an either/or situation. Either you use raid with all of its caveats or you find another way to do it. If MS would build a parity option into Storage Spaces similar to how Unraid does it I think it would be killer.

I was actually thinking of unRAID. Might be more your speed, since you've already got a disparate batch of disks. Would be nice to just add a couple more disks one at a time to increase your storage pool size, rather than having to start from scratch with a new array just to go from 3TB to 4TB disks and 15 to 20TB storage.

But if you're a Windows guy, well.

I do see your point about Raid 6. If what your saying about wear and tear is true, then the virtue of Raid 6 is that, during a rebuild, you would need to have a URE happen on both parity copies at the same time for a rebuilt to fail, assuming a second, full drive failure isn't imminent? I hope I worded that right.

1

Basically you've got it.

Regarding wear & tear - I know people who swear up and down that a rebuild is the most likely time for a second drive to fail, but IME it seems like everybody knows a guy who knew a guy who had it happen, but nobody's ever had it happen themselves. Meanwhile, I've done plenty of RAID rebuilds (one of my hats is SAN admin. I see a LOT of failed hard drives.) and failures are pretty spaced out, in general. (With a population of a couple hundred drives per SAN, it's rare that I get two failed disks in the same SAN in the same week. In three years, I've never had a second dead drive in the same disk group during a rebuild. But we're using double-parity anyway, so it would take three to really ruin my day.)

Granted, if I did have that happen and it hosed an array, It'd be pretty traumatic and I might be scared of a repeat performance in spite of whatever the "real" odds were.

 

[Anteaus]

I was actually thinking of unRAID. Might be more your speed, since you've already got a disparate batch of disks. Would be nice to just add a couple more disks one at a time to increase your storage pool size, rather than having to start from scratch with a new array just to go from 3TB to 4TB disks and 15 to 20TB storage.

But if you're a Windows guy, well.

I'm very much a Linux guy also. I'd be 100% linux based if I didn't game on my main PC.

If I end up with a Raid 6 array it will likely be using Ubuntu or Debian server with mdadm. I just think Storage Spaces is pretty cool for what it does. With Unraid, I don't like that it is USB Flash based.

On another note, I wish Linux had something as good as RDM. Sure, there some decent options like Teamviewer but MS has the remote admin thing down cold.

I have a couple weeks to mull it over before I need to place my order for additional drives. Thanks for all the info up to this point I will take it all under consideration.

 

[nk215]

I also do the wipe, re-create and restore from backup is significantly faster than rebuild a fail drive. If i remember correctly, it took almost 3 days to rebuilt my 5-2TB RAID 5 but less than a day to re-create from scratch.

The only problem I have with re-create and restore is that data is not available during that time (I pointed my Plex to my the backup NAS so no real downtime for me, but not everyone has similar setup).

 

[bmuscotty88]

Just found this thread while searching about raid50. I see you are using raid5 with 6 drives but I noticed something convenient with raid50. I set up a raid50 with 6 drives (which is actually 2 raid5 with 3 drives each + striping) which shows up as a single volume. Nice thing about this setup is I had a drive fail, but since it was in ONE of the TWO sets, I could still read and write data just fine as the new drive rebuilt, AND ALSO while the degraded unit was initializing! (this was because the drive failed before the original init finished) I found this to be very convenient but not sure if something like this would work for you or if your controller support raid 50. With caching enabled write speeds on an older 3ware card averages about 110MB per second. And with six drives on raid 5, you increase your chance of losing data if more than one fails or during a rebuild, but with raid50 it splits these 6 drives into two 3drive raid 5 arrays, plus the benefit of striping data. Maybe consider using this option to possibly avoid having to blow away and start over.

 

[MalVeauX]

Just found this thread while searching about raid50. I see you are using raid5 with 6 drives but I noticed something convenient with raid50. I set up a raid50 with 6 drives (which is actually 2 raid5 with 3 drives each + striping) which shows up as a single volume. Nice thing about this setup is I had a drive fail, but since it was in ONE of the TWO sets, I could still read and write data just fine as the new drive rebuilt, AND ALSO while the degraded unit was initializing! (this was because the drive failed before the original init finished) I found this to be very convenient but not sure if something like this would work for you or if your controller support raid 50. With caching enabled write speeds on an older 3ware card averages about 110MB per second. And with six drives on raid 5, you increase your chance of losing data if more than one fails or during a rebuild, but with raid50 it splits these 6 drives into two 3drive raid 5 arrays, plus the benefit of striping data. Maybe consider using this option to possibly avoid having to blow away and start over.

Check the date... 2016.

Now you can buy a 14TB (+/- based on your needs) drive and not require buying multiple little drives just to get the capacity. You can get two 14TB drives (+/-) and do 1:1 mirror without all the special controllers, no special RAID enabled, no special RAID drivers, no parity, no slow rebuild time, with 1:1 redundancy and only use 2 SATA ports to do it. Or, if you really want 2 drive redundancy, setup TWO mirrors, each high capacity 1:1 and you have 2 drive fault tolerance, no parity, no controllers, and only 4 SATA ports and if a drive fails, there's no slow down in read/write like there is with parity. And the drives can be pulled and put in another box and read them directly without RAID or controllers. There's really not a compelling reason to use RAID levels these days if your goal is high capacity fault tolerance HDD.

Very best,

 

[Lopoetve]

Check the date... 2016.

Now you can buy a 14TB (+/- based on your needs) drive and not require buying multiple little drives just to get the capacity. You can get two 14TB drives (+/-) and do 1:1 mirror without all the special controllers, no special RAID enabled, no special RAID drivers, no parity, no slow rebuild time, with 1:1 redundancy and only use 2 SATA ports to do it. Or, if you really want 2 drive redundancy, setup TWO mirrors, each high capacity 1:1 and you have 2 drive fault tolerance, no parity, no controllers, and only 4 SATA ports and if a drive fails, there's no slow down in read/write like there is with parity. And the drives can be pulled and put in another box and read them directly without RAID or controllers. There's really not a compelling reason to use RAID levels these days if your goal is high capacity fault tolerance HDD.

Very best,

And for that matter, software raid or LVM-esque filesystems (ZFS, ReFS, etc) are just as good as hardware RAID, if not better (lots of time spent optimizing that in software due to the enterprise space), and also do better checksumming/data management.