Server Data, Raid 5 Initialization Time vs. Rebuild Time

Anteaus

Platinum Member
Oct 28, 2010
2,448
4
81
I currently maintain a home file server which contains 15TB of storage using a spanned partition over 3 4TB Hdds and 3TB HDDs. I maintain two independent backups which I sync weekly so my data is "reasonably" secure. I am planning on upgrade the server soon to contain 20TB of storage, using as many 4TB drives as necessary depending on the method.

This isn't a "which RAID level should I use" thread, though it is related to RAID. I would like some level of cost effective redundancy purely to buy time to sync to my latest backup. Downtime isn't an issue once that is achieved.

My thinking is that I would like to use RAID 5 purely as a stop gap measure, which no intention of rebuilding should a drive fail. For example, lets say I use 6 X 4TB drives to build a 20TB RAID 5 array. At some point one of the HDDs dies and the array becomes degraded. All I'm looking for is time to update my backups. Afterward, I will replace the drive (probably days later), reinitialize, and restore from backup instead of going through the rebuild process. I'm thinking this will remove some of the risk of rebuild failure since I won't need to successfully read every sector of every drive. I will need to rely on my backups at that point, but that is why I have two. :)

In all, what I'm looking for are real world experiences of how long it takes to initialize a RAID 5 array versus rebuilding it. If it isn't noticeably faster to reinitialize instead of rebuilding, then its not really worth the hit in write performance or the time investment. I'll just stripe it and make sure I'm consistent with my backups.

I don't mind working without a net (RAID 0, JBOD), but running so many drives makes me wonder about my chances of failure. My other thought is to use something similar to flex raid to create a parity drive. Perhaps that is the better way to go regardless.

Opinions?
 
Feb 25, 2011
16,527
1,307
126
In most cases, it's faster to wipe and restore than it is to rebuild an array.

Syncing data to and from a device over a network is easy math - 20*2^20 MB @ ~80 MB/sec = 262144 seconds. So about three days.

Rebuild time/speed just depends on your server and raid controller. For rebuild times, I've heard people cite anywhere from 1TB/day to... well, a lot more, or a lot less. Anandtech NAS reviews usually include a RAID rebuild benchmark. But the data is accessible that whole time.

So, as the good book says. "RAID is not a backup, RAID is for uptime." 3 days to sync back your data is three days that it's not accessible and 3 days that your dependent services are unavailable.

I'm thinking this will remove some of the risk of rebuild failure since I won't need to successfully read every sector of every drive. I will need to rely on my backups at that point, but that is why I have two. :)
I don't want to dissuade you from keeping two backups, but most software RAID implementations work around that rebuild failure problem.

What I would suggest is RAID-6. (RAIDZ2, double parity, whatever you want to call it.) Then your data is still parity protected and accessible while you rebuild the failed drive.
 

Anteaus

Platinum Member
Oct 28, 2010
2,448
4
81
In most cases, it's faster to wipe and restore than it is to rebuild an array.

Syncing data to and from a device over a network is easy math - 20*2^20 MB @ ~80 MB/sec = 262144 seconds. So about three days.

Rebuild time/speed just depends on your server and raid controller. For rebuild times, I've heard people cite anywhere from 1TB/day to... well, a lot more, or a lot less. Anandtech NAS reviews usually include a RAID rebuild benchmark. But the data is accessible that whole time.

So, as the good book says. "RAID is not a backup, RAID is for uptime." 3 days to sync back your data is three days that it's not accessible and 3 days that your dependent services are unavailable.



I don't want to dissuade you from keeping two backups, but most software RAID implementations work around that rebuild failure problem.

What I would suggest is RAID-6. (RAIDZ2, double parity, whatever you want to call it.) Then your data is still parity protected and accessible while you rebuild the failed drive.
Thanks for the feedback. For me the question is whether the initialization/rebuild times are worth potentially losing a few files should i lose the volume between backup cycles. I'm basically in agreement that it is faster to just wipe and restore. I think I may just continue running without redundancy. Again, I'm fine with the downtime if I need to restore. It's the time up front for initialization that I was interested in. I don't want to wait for three days to initialize just to spend another 1-3 days copying data from backup. Some will think I'm nuts for running a 20TB Raid 0 array, but I guess it all depends on what you are trying to do. The write penalties and additional cost for RAID 6 aren't worth it for my usage. I'd go Raid 10 if money were no object.

I may implement some sort of dump drive, where all new files are copied in a secondary location as well as the array when between backup cycles.
 

master_shake_

Diamond Member
May 22, 2012
6,430
291
121
Last edited:

Anteaus

Platinum Member
Oct 28, 2010
2,448
4
81
yeah no to a 22 drive raid 10. or 12 drive raid 10.

my raid card supports 0,1,5,6,10,50,60 so i chose raid 6.

redundancy and maximum amount of space.
That would be a reasonable way to go for me except for the heavy write penalties and tendency to destroy consumer HDDs. I have no intention on actually rebuilding an array with a failed disc. I'm looking for a solution that buys me enough time to update my backup.

My original thought was to use RAID 5 if/until a disc failed, backup the array, then nuke it and reinitialize. From what I understand, the wear and tear on the HDDs during the rebuild process is why RAID 5 isn't preferred. Again, I don't care if the array fails completely as long as my backups are up to date.
 
Feb 25, 2011
16,527
1,307
126
The wear and tear of a rebuild isn't necessarily any worse than normal server-is-busy operations. (Seeks around the disks, reads everything once.) The potential problem is the unrecoverable error rate vs. the size of the disks and how the RAID controller / software handles those UREs. (Some don't deal with them well - they hit an error and just shut down. Some can try to recover and keep going, but the data in that sector is still lost.)

There's nothing really wrong with parity RAID (5 or 6) but conventional wisdom is to make sure that you use RAID6 if your drives are over 700GB in size or so, because of the URE/recovery behaviors.

What do you use this array for? That would actually determine whether or not the write penalty is meaningful, as well as what other options you may have regarding your backups / scheduling.
 

Anteaus

Platinum Member
Oct 28, 2010
2,448
4
81
The wear and tear of a rebuild isn't necessarily any worse than normal server-is-busy operations. (Seeks around the disks, reads everything once.) The potential problem is the unrecoverable error rate vs. the size of the disks and how the RAID controller / software handles those UREs. (Some don't deal with them well - they hit an error and just shut down. Some can try to recover and keep going, but the data in that sector is still lost.)

There's nothing really wrong with parity RAID (5 or 6) but conventional wisdom is to make sure that you use RAID6 if your drives are over 700GB in size or so, because of the URE/recovery behaviors.

What do you use this array for? That would actually determine whether or not the write penalty is meaningful, as well as what other options you may have regarding your backups / scheduling.
The array is predominately for media use, so the write penalties aren't so problematic; however. the idea of waiting for many days for an init or rebuild to take place isn't good. All things being equal, I'd rather go with performance and simplicity if I have the option. The issue I have with Raid is that its generally seen as an either/or situation. Either you use raid with all of its caveats or you find another way to do it. If MS would build a parity option into Storage Spaces similar to how Unraid does it I think it would be killer.

I do see your point about Raid 6. If what your saying about wear and tear is true, then the virtue of Raid 6 is that, during a rebuild, you would need to have a URE happen on both parity copies at the same time for a rebuilt to fail, assuming a second, full drive failure isn't imminent? I hope I worded that right. :)
 
Feb 25, 2011
16,527
1,307
126
The array is predominately for media use, so the write penalties aren't so problematic; however. the idea of waiting for many days for an init or rebuild to take place isn't good. All things being equal, I'd rather go with performance and simplicity if I have the option. The issue I have with Raid is that its generally seen as an either/or situation. Either you use raid with all of its caveats or you find another way to do it. If MS would build a parity option into Storage Spaces similar to how Unraid does it I think it would be killer.
I was actually thinking of unRAID. Might be more your speed, since you've already got a disparate batch of disks. Would be nice to just add a couple more disks one at a time to increase your storage pool size, rather than having to start from scratch with a new array just to go from 3TB to 4TB disks and 15 to 20TB storage.

But if you're a Windows guy, well. :D

I do see your point about Raid 6. If what your saying about wear and tear is true, then the virtue of Raid 6 is that, during a rebuild, you would need to have a URE happen on both parity copies at the same time for a rebuilt to fail, assuming a second, full drive failure isn't imminent? I hope I worded that right. :)
1

Basically you've got it.

Regarding wear & tear - I know people who swear up and down that a rebuild is the most likely time for a second drive to fail, but IME it seems like everybody knows a guy who knew a guy who had it happen, but nobody's ever had it happen themselves. Meanwhile, I've done plenty of RAID rebuilds (one of my hats is SAN admin. I see a LOT of failed hard drives.) and failures are pretty spaced out, in general. (With a population of a couple hundred drives per SAN, it's rare that I get two failed disks in the same SAN in the same week. In three years, I've never had a second dead drive in the same disk group during a rebuild. But we're using double-parity anyway, so it would take three to really ruin my day.)

Granted, if I did have that happen and it hosed an array, It'd be pretty traumatic and I might be scared of a repeat performance in spite of whatever the "real" odds were.
 

Anteaus

Platinum Member
Oct 28, 2010
2,448
4
81
I was actually thinking of unRAID. Might be more your speed, since you've already got a disparate batch of disks. Would be nice to just add a couple more disks one at a time to increase your storage pool size, rather than having to start from scratch with a new array just to go from 3TB to 4TB disks and 15 to 20TB storage.

But if you're a Windows guy, well. :D
I'm very much a Linux guy also. I'd be 100% linux based if I didn't game on my main PC. :)

If I end up with a Raid 6 array it will likely be using Ubuntu or Debian server with mdadm. I just think Storage Spaces is pretty cool for what it does. With Unraid, I don't like that it is USB Flash based.

On another note, I wish Linux had something as good as RDM. Sure, there some decent options like Teamviewer but MS has the remote admin thing down cold.

I have a couple weeks to mull it over before I need to place my order for additional drives. Thanks for all the info up to this point I will take it all under consideration.
 

nk215

Senior member
Dec 4, 2008
403
2
81
I also do the wipe, re-create and restore from backup is significantly faster than rebuild a fail drive. If i remember correctly, it took almost 3 days to rebuilt my 5-2TB RAID 5 but less than a day to re-create from scratch.

The only problem I have with re-create and restore is that data is not available during that time (I pointed my Plex to my the backup NAS so no real downtime for me, but not everyone has similar setup).
 

ASK THE COMMUNITY