10.5TB + 4.5TB migration Strategy

[Evadman]

I currently have a bunch of data on different NAS's, but my FreeNAS has three 2 TB RAID5 arrays built on a adaptec 31205 controler. I have another 2 TB on another NAS. I need more space, and I want to combine all my NAS's together. all together, I have 16 750GB drives with just over 6 TB of actual data across all arrays and NAS's. I just bought a adaptec 52445 28 port SAS/SATA controler to put n my freenas box as part of this process.

I don't have a way to back up data to tape or something, it costs way too much. (disk is < tape, and disk is still expensive for this amount of data, and I don't have a drive). So this is what I am going to do.

1. Buy four 1.5 TB drives (6TB total space)
2. Move data from all systems to these 4 drives
3. Put all 16 750GB drives into freenas box on new controler.
4. Build RAID6 array of those 16 disks, total space 10.5TB.
5. Move data from 1.5 TB drives to new array
6. Turn 1.5 TB drives into a new 4.5TB RAID5 array

Good:
Cost for this solution (disk) would be $520, limited to the four 1.5TB drives.

Problems/concerns
Moving data will take forever, and during that time, the 1.5TB drives each become a single point of failure. if one fails, I lose 1.5TB of data. Possible solution: Getting 5 1.5TB drives and build an array on the new card first. Issues with that solution are that the mainboard I have has 2 PCIe 16x slots, but I don't know if both can have non-video cards installed in them (old and new controller). It also costs an extra ~$130 for the drive. Additionally, the case I have only has 20 hot swap trays, and that is a total of 21 drives. (16 750's, 5 1.5's)

Is there any reason I shouldn't create a 10.5TB array on 16 disks? It needs to be redundant since I don't do backups (calculated risk) and with 16 drives the possibility of one failing while rebuilding from a failed drive becomes larger. My current calculated risk that I will accept is at 4 disks in a raid5 array. That's why I am switching to RAID6 (dual striped parity).

Another solution I can think of would be this:
1. Get 10 1.5TB drives
2. Copy data from one of the 2TB arrays to every available free space portion on the other three 2 TB arrays.
3. Pull those 4 disks out of the server
4. Install 10 1.5TB drives (now 10+8 drives in the 20 bay server)
5. Build 12 TB RAID6 Array
6. Copy data from all other arrays and NAS's back to the new 12TB array.

Good:
Have new drives in the server, all the same size.

Problems/concerns:
Post for this solution is $1300 for 10 1.5TB disks. Possible Solution: Sell the 16 750GB drives to recoup some of the outlay. However, this will only be $40-$50 a disk after shipping expense and all that good stuff. It also takes up time I don't have. That would put the cost at around $600, plus 5-6 hours of time for selling the disks.

By the way, if you want to caluclate MTBF failure rates (full loss) of an array:
RAID0 - MTBF of a disk / total disks
RAID5 - MTBF of array / (Rebuild time/(MTBF of a disk / total disks-1))
RAID6 - Failure of 1 disk turns array into RAID 5, so it would be failure possibility of RAID5 / MTBF of single disk

So a 4 disk RAID5 of Seagate drives has a MTBF of the array of 750,000/4 = 7812 days to lose a disk from the array. Assuming rebuild time of 24 hours, the probability of total loss (assuming instant replacement) would be 0.0096%. That's high, but acceptable, especially since it will actually take 5-7 days to get a replacement disk if you order one online. The 24 hour rebuild time is best case. So total failure would have a 0.0768% chance for 8 days, with only 4 drives. 16 drives in array raises that to 0.384%, a 5 fold increase in possibility. 16 drives in RAID6 would have a probability of total failure of 0.0000512%. Back to safer levels, and 7500 times less likely to have total failure than 16 drives in RAID5. Of course, this is disk failure only, not controller, memory, CPU, or others which can all cause corruption, and only a backup can solve for.

Whitepaper on RAID failure calculations: http://www.google.com/url?sa=t...LAsQ4qkAgomcB6HkZ0eIdw

What do you think about the process I have planned? Any reason I shouldn't create an array that big?

 

[RebateMonger]

You might also consider just moving to MS Windows Home Server. It'd give you a lot more disk flexibility. You can add disks of any size in any combination at any time in the future. You can also easily remove single disks and replace them with little fanfare.

Since WHS' redundancy is similar to RAID 1, you'll lose some total capacity, but would really reduce the chances of of losing "everything". Even the horror of multiple simulataneous disk failures would only lose data if a particular file has its duplicate on one of the failed drives. And WHS stores files on standard Basic disks, so disk data recovery, if ever needed, would be much easier than trying to recover from a bunch of failed, striped disks.

 

[Evadman]

You can add disks of any size in any combination at any time in the future. You can also easily remove single disks and replace them with little fanfare.

Thanks for the feedback. I can do that with ZFS too, and ZFS has a higher redundancy with lower overhead than WHS. WHS costs money, and lost me as a customer when they screwed up the basic block level implementation and corrupted tons of files. I have had horrible experience with windows servers (NT, 2000, 2003 and WHS), and do not want to try them for a 5th time when they already failed.

WHS is a good idea if I didn't just spend a ton on a 28 disk hardware raid controller, but I believe ZFS is a better implementation that accomplishes the same task and more (like "free" file journaling and snapshots). However, ZFS is still in early beta in FreeBSD (and Linux I understand) since it was designed for solaris recently. ZFS (or any single disk implementation) would have hardware requirements of ports, so you can go with cheap as dirt SATA controllers ($20 for 4 ports probably). The way I am going, I can go for RAID6 and have an astronomically low (but non-zero) total failure rate. WHS can't accomplish one of the main tasks, I have which is consolidation. IIRC, WHS has a 2 TB logical volume limit, which I would hit several times over. I would also only have roughly 1/2 the space available, while the possibility of failure would be the same as RAID6, at 0.0000512%.

 

[RebateMonger]

Originally posted by: Evadman
WHS has a 2 TB logical volume limit, which I would hit several times over.

No, you can create a very large storage pool. WHS doesn't expose "Volumes" to the user in the traditional sense. There's just one big data store. That store appears to be limited to 16 TB at this time, probably related to the 32-bit OS. Now THAT might be a limit for you, since you are enabling redundancy. It'd depend on how much of your desired storage you are actually using at this time.

Edit: It appears that I'm wrong about the 16 TB Storage Pool limit. There are reports of 24 TB and 32 TB Windows Home Server systems out there. Discussion about very large WHS Storage Pools.

There IS a problem using disks larger than 2 TB each, but by the time those drives are affordable, the new 64-bit version of WHS will likely be available.

 

[pjkenned]

Or you could just be wasteful like me and run Raid 6 then WHS on top of it.

 

[Evadman]

RebateMonger, why wouldn't I want to use ZFS, AKA, why is WHS better than a ZFS partition on Unix/FreeBSD?

 

[RebateMonger]

Originally posted by: Evadman
RebateMonger, why wouldn't I want to use ZFS, AKA, why is WHS better than a ZFS partition on Unix/FreeBSD?

Evadman, I wouldn't recommend ZFS over WHS to MOST people because WHS is so much simpler to install and support. As far as I can tell, ZFS seems very cool. Many of the disk management features are similar to WHS.

In your OP, you asked for opinions.
PERSONAL OPINION
I worry about ANY important data being stored on a single device, single media, single array, etc. Whether it's on a CD, a USB stick, a single hard drive, a redundant RAID array, or a ZFS or WHS storage pool. The chances of multiple drive failures, drive-destoying power glitches, or human error seem to be higher than estimated by combining MTBF numbers for single hard drives. If I could only afford two hard drives, I'd use the second drive for backups and not to make a RAID 1 array. And I'd be very concerned about striping my important data across sixteen hard drives without backups, redundancy enabled or not.
PERSONAL OPINION

With data stores that large, there aren't any "best" answers. The better answers are always more expensive and making backups of such large restores isn't trivial and isn't cheap.

 

[Evadman]

Oh gotcha. I agree, WHS (like most windows products) is a ton easier to use and configure for most people than a system that can utilize ZFS. Currently ZFS requires that the user learn Unix or BSD. I am actually using FreeNAS (which runs on freebsd) which has a very simple to use web interface which is kind of close to WHS. Unfortunately though, the version of FreeNAS which supports ZFS is in alpha now and is not expected to be released stable for 6 to 12 months. That means right now, WHS (or a Netgear ReadyNAS, which is just as simple as WHS IMHO) would be my top recommendations for anyone who want to get "one of those NAS things" that they are hearing about

<edit>
Oh, I should also say I am not arguing, I am generally curious to your thoughts on WHS RebateMonger. I used WHS for less than a week IIRC before I figured out it was fubaring files, so my experience is limited to that only.

Yes, backups cost a boatload of money with large data stores. Since I am only using ~6TB now, I was thinking of using the "spare array" of the 1.5 TB disks as backup space by running backups every once and a while. in a kind of poor man's RAID 50/60. But that still makes the entire computer a single point of failure. Meaning bad memory, CPU, controller or something can still kill all my data.

I would be lying if I said I wasn't concerned about that loss, since that server basically has every bit of data for my life on it. Once the economy gets better and disk space gets cheaper, I may revisit a backup solution. Perhaps throwing some high capacity drives into the NV+ that I won't be using after this upgrade and using that as a backup. But right now, I don't have the cash for that

 

[RebateMonger]

Originally posted by: Evadman
But right now, I don't have the cash for that

Trust me, I know the feeling...

I am generally curious to your thoughts on WHS RebateMonger. I used WHS for less than a week IIRC before I figured out it was fubaring files, so my experience is limited to that only.

Since the PP1 fix for the data corruption issues, I haven't heard of any data loss issues with WHS. Considering how sensitive many of us were about the file corruption bug, I'd expect any more problems to be freely reported.

I've migrated my WHS data and backup database several times across multiple WHS servers, I've virtualized it twice, and I've done multiple full PC restores. WHS has been flawless for me and for my clients.

I bounce back on forth rather to use the Folder Redundancy on WHS or not. It's SO easy, it's tempting. I had Redundancy enabled for my music and videos for a while when I had five hard drives in my WHS box and a bit more free space. But right now I only have a single Hitachi 1 TB drive. I think I'll stick with backups of my data to my Seagate 1.5 TB drive for the time being. In my case, it doesn't make sense to use BOTH redundancy AND backups.

I used to worry about WHS not having a built-in way to back up the client PC backups. But it's easy to do if you really want those backups. Yeah, if your WHS catches fire, you'll lose the ability to go back to those really old backups, but it's not the end of the world. 90% of the time, you'll want to restore your PC to YESTERDAY's state, not to its state a year ago.

Similary, WHS' inability to make full backups of itself (full OS backups) was disappointing, but an OS reinstall isn't that big a deal unless you've installed and configured a bunch of add-ins like web sites. And if you want to RAID 1 the OS disk, that makes it very unlikely you'd ever have to re-install the system from scratch.

 

[Net]

i have to ask, what do you use all that space for?

 

[Evadman]

Originally posted by: net
i have to ask, what do you use all that space for?

Mostly pictures of Doug777.

 

[taltamir]

a raid6 disk failure does not turn it to a raid5 array, it turns it to a raid6 array with one faulty disk which still retains data and can survive another disk failure without dataloss.

Also... I recommend you go ahead and sell the 750GB drives, replacing them with 1.5GB drives. make a raid6 array... so buy enough 1.5tb drives to make new raid6 arrays to contain all the data. migrate data off of .75tb drives, sell those used.

Also, i recommend you look into zfs, it is much better, it actually protects against random errors, bit rot, and bit flip from solar radiation (it does happen... on a 5x750GB drive raid6 zfs array i get about 1 error in 2 months, which is corrected because zfs actually hashes all files individually so it knows which is the good data and which isn't)
it also has a bunch of other benefits.

 

[RebateMonger]

Originally posted by: Evadman

Originally posted by: net
i have to ask, what do you use all that space for?

Mostly pictures of Doug777.

Send me some?

 

[Evadman]

Originally posted by: taltamir
Also, i recommend you look into zfs, it is much better, it actually protects against random errors, bit rot, and bit flip from solar radiation (it does happen... on a 5x750GB drive raid6 zfs array i get about 1 error in 2 months, which is corrected because zfs actually hashes all files individually so it knows which is the good data and which isn't)
it also has a bunch of other benefits.

Absolutely. I really like ZFS from what I know about it. Free snapshots and everything. But in FreeBSD/FreeNAS it is still in alpha

 

[taltamir]

than use opensolaris?
I thought the FreeBSD implementation was pretty advanced already... i use osol over it because i like being on the cutting edge.
genunix.org is the place to get it

anyways, using open solaris intead of freenas is a small "price" to pay for preventing file corruption

 

[Evadman]

Thanks for the feedback everyone. I decided to get ten 1.5TB drives and build a RAID 6 array for 12TB of storage or so. Here, have some drive pr0n.

 

[n7]

:Q

 

[cliftonite]

Originally posted by: Evadman
Thanks for the feedback everyone. I decided to get ten 1.5TB drives and build a RAID 6 array for 12TB of storage or so. Here, have some drive pr0n.

Nice.

 

[specialk90]

Side note: for backup, have you looked into online backup? I know 6TB would take forever, but some places will send you hard drives to store your data on and send back to them. Also, most sites have an unlimited plan so you won't need to worry about ridiculous costs. Heck, Carbonite is only $55/yr unlimited but I don't know whether they send drives.

Since you never actually stated what your data consists of, I can't say whether compression would help a little or a lot for backup.

If I need to start a new thread, just tell me but hopefully, I can get a few quick answers.

Maybe some of these great minds in this thread can tell me how risky using XP Pro and a 3ware 9650SE-8 with a 4-drive Raid 5 using 7200.11s 500GB. Is Vista x64/32 any better? Sever 2008? I do video editing & compositing so I still need a regular OS for Adobe. I ask because I had a partition corruption a few yrs ago with 8-250GB 7200.10s in Raid 5 causing the entire partition to be missing, and I spent over 50 hours fixing it. I just got an idea: I have been considering consolidating my 2 PCs into 1; main PC has 4-150GB Raptors(1 is dead, to be RMA'd) in Raid 10 on Intel board, the 3ware Raid 5, and my 2nd PC has a 4-drive Raid 10 using 4-74GB Raptors on Intel board(both PCs use same boards & ICH8R).

So here is my idea: move my 3ware Raid 5 to the other PC and convert that PC to ZFS and network them to each other(would ethernet directly connected be better or should I use a router). I don't have a serious backup plan at the moment due to giving my 3rd PC to a family member(w/ 1TB storage). I have an external drive with absolutely important data backed up on it. I would need to be able to store video on the Raid 5 array and access it on my main PC. I would also install Vista x64 on my main PC because I need more ram for Adobe. I will most likely be adding more drives within the next 6 months because I started video editing as a career recently and I have some jobs lined up which will fill up my drives quickly.

I don't need very detailed answers(would be appreciated though). If this is feasible and safer, just point me in the right direction and I can do the needed research.

Thanks

 

[Evadman]

I figured out that this was a bad idea, a very bad one. The seagate 1.5 TB drives are not supported on the upgraded controller I got unless the firmware on the disks is updated. The hitch is that you can't upgrade the firmware on seagate drives that have firmware CC* because the hardware is physically different. Guess which firmware version I have on all 10 drives? If you guessed CC, you win a cookie. I didn't catch it until after I had built the array on the old 31205 card, which actually supports the drives. But when I tried moving all the arrays to the single card, I dropped about 9 disks from the different arrays because the new card freaked out. I got 2 of the 3 arrays rebuilt on the new adaptec card, and I will rebuild the last one (hopefully, I may need some help from adaptec) once I get all the data off the 2 current arrays and can put the other disks back in the enclosure. On top of it, I can't even use both cards in the same box, my motherboard only initializes to whichever card is first in the motherboard, and ignores the other one.

http://ask.adaptec.com/scripts...ZT0x&p_li=&p_topview=1]Adaptec KB Article[/url]. Adaptec also has no plans to update their firmware to work with the 1.5 TB CC* firmware drives. Woo.

So now I have 10 1.5TB drives that are unboxed and used. Go me. However, I am looking on the bright side (if there is one to wasting a bunch of cash). Since I have a 4 disk readynas NV+, I threw 4 of the 1.5TB disks in that, for about 4 TB of space. I am going to use that as a backup device, since the configuration is stored entirely on the disks. So you can put 4 disks in, build an array, copy data to them, then pull the disks out and put 4 more disks in. To "recover" data, all you need to do is put the same 4 disks back in the NAS, and boom, instant recovery. So with 8 of the 1.5 TB disks, I have ~8 TB of backup space, that itself is backed up in a raid 5 software array.

I would like to take this time to point out that I spent a bunch that I didn't need to; if only I had figured this out before hand. Darn you people, you were all supposed to tell me this before I bought the disks. The internet has failed me. Or I failed the internet, one of the 2.

Originally posted by: specialk90
I can't say whether compression would help a little or a lot for backup.

It would help very little for my data. Maybe 10%.

 

[Syntax Error]

I've had experience with those drives with Adaptec controllers; one thing led to another and I decided to ditch my Adaptec (31605 for a 3Ware 9650-16ML.

Check this thread for some info that may be pertinent to your situation:

http://www.hardforum.com/showthread.php?t=1380766

Long story short, Adaptec's support for the ST31500341 is "flaky" because of a combination of Seagate's firmware issues as well as Adaptec's unwillingness or slow response time to updating drivers to fully support the drives in a RAID environment, especially in ASM.

I mainly ditched the 31605 because of its ludicrous OCE times; I can't afford to buy enough drives + 1 to expand when I need to, thus mitigating the need for OCE.