encrypted fault tolerant expandable NAS?

[TXFI]

I've been looking into building a NAS for myself and while there are several options out there, none of them can do all the things I'd want it to do.

Could you guys help me find out whether it would be possible to obtain a NAS server that is:

1) fault tolerant - must be able to sustain a minimum of one failed drive, preferably two failed drives
2) expandable - must be able to dynamically add new drives that are (semi)automatically added to the pool and made available to the total storage capacity
3) encrypted - must be able to encrypt the data and still handle adding new drives


As far as I know, FreeNAS etc. can't do all of these things and ZFS encryption is still in beta. Any ideas whether there is a solution out there that could do all three things? I'd prefer something I could build myself but commercial solutions aren't out of the question, either.


thanks!

TX FI

 

[Nothinman]

Any Linux installation will let you do that with dm-crypt on top of software RAID but you'll probably have to configure it by hand.

 

[taltamir]

that expandability seems like the biggest problem. Any sort of raid is really not practical to expand. The data is stored in a completely different manner depending on your raid flavor and the amount of drives. If you want to expand it, you have to create a new blank array and transfer data to it. (so, two sets of hardware there). Unless you completely eschew fault tolerance that is, and just have each drive contain its own data.

So fault tolerance and expandability are mutually exclusive.

Read this: http://en.wikipedia.org/wiki/RAID

 

[TXFI]

Thanks for the responses.

I've read that RAID page many times. If you read this page: http://en.wikipedia.org/wiki/Non-standard_RAID_levels you'll see that expanding can be done (RAID-K) so all I'd need on top of that would be an encryption mechanism that allows increasing volume size/automatically grows with the available disk space.

AFAIK ZFS also allows adding disks to RAID pools without recreating the array.

Any useful information you might have would be greatly appreciated.

 

[taltamir]

I forgot about those... those type of raids are a whole different beast. very nice stuff. But aren't they prohibitively expensive?

What kind kind of budget are you dealing with? and what is the general purpose of your array?

Actually aren't they just fancy name for regular raid array created by a controller that contain an OS with a variety of tools and subroutines for changing the type of raid? They are all proprietary and they are all controller based. I see no reason why an array couldn't be migrated from raid 1 to raid 5 or from 5 to 6, or from different amounts of drives on the same RAID5 without loosing the data thanks to an intelligent controller. But those controllers are generally prohibitively expensive.

I am actually interesting in creating my own highly redundant multi terabyte storage system, but my budget is very limited. So I am very interested to know if there is some open source NAS software that can make such a flexible "RAID"

 

[Nothinman]

So fault tolerance and expandability are mutually exclusive.

Not true, Linux software RAID has had online reshaping capability for a while now.

 

[taltamir]

how does that OS based RAID handle failure of the controller (for RAID-K and the like "hardware" RAID) / OS upgrade (for linux software raid)? (This isn't rhetorical, I really want to know)

I can and I HAVE taken RAID1 arrays between vastly different systems. At WORST it simply detected them as two regular drives... so I created a new RAID1 array using one of the drives as a source (maintaining all my data)..
This gives me fault tolerance in both the drive, AND in the controller. If the controller (motherboard) fails, I can just move it to any system and the data is all there perfectly accessable.
It doesn't matter what controller I use, it doesn't matter OS I use (since NTFS is now supported by linux). The point of redundancy isn't just to eliminate the harddrive failure, but to eliminate ALL single points of failure. transferring the single point of failure from the HDD to a controller is not doing anyone any good. (or allowing the controller to be replaced by the exact same model, creating a situation where an expensive, obsolete, replacement part is needed...)

Motherboard based SATA RAID5 on the other hand explodes if you reset the bios (intentionally or through battery replacement). It CAN be restored, in a dangerous process... you have to delete the array, recreate it by adding the drives in the EXACT same order, with the exact same stripe size, and then tell it NOT to clear the array. if you click the wrong prompt at any of these stages you just lost all your data. It is MORE risky to use software RAID5 then it is to use individual drives.

Oh one more thing. ALWAYS break in a new drive. Do a full format, Let it spin for a day. Fill it up, let it spin for two more days. Format again. If it is still alive then all is good. But there is a decent chance it will die on you from that procedure (and good thing you found out now while it was clear of data). Drives that survive that last years.

 

[Nothinman]

how does that OS based RAID handle failure of the controller

Depends on what symptoms the failure causes. But that's like asking how it deals with bad memory, at that point it's not the RAID software's job to fix the problem.

I can and I HAVE taken RAID1 arrays between vastly different systems. At WORST it simply detected them as two regular drives... so I created a new RAID1 array using one of the drives as a source (maintaining all my data)..

Those were probably all onboard fake RAID arrays where the actual RAID is all done in software in the driver. And that's only RAID1, the OP will most likely want RAID5 or RAID6 so that he doesn't have to keep adding drives in pairs to mirror them.

This gives me fault tolerance in both the drive, AND in the controller. If the controller (motherboard) fails, I can just move it to any system and the data is all there perfectly accessable.

With real hardware RAID the situation is pretty much the opposite, go ahead and take an array from a CCISS controller and drop it in a PERC and see if it can use it.

 

[taltamir]

So you are saying I can't move controllers? then this is my point.. I DON'T CARE if it is the software fault or santa claus' fault. My data is GONE! that is NOT acceptable.

The whole point is to have a system with no single point of failure. as far as I can tell RAID5 isn't it.

So far multiple RAID1 arrays have been the holy grail of data safety for me.
And I only wish that there was a way for me to combine them with some sort of parity system that protects me from file corruption due to other issues. Like a random bad sector or a memory/cpu hickup during copy operations. A write-compare system and a file system that has parity (preventing loss of specific data due to bad sectors) would make it ideal, making data loss highly improbable (double drive failure)

 

[Nothinman]

The whole point is to have a system with no single point of failure.

Actually since you're not the OP you're not the one who gets to set the point of the thread. But even if that was the point it's pretty much impossible on a home budget.

 

[TXFI]

I'm not so much interested in fault tolerant systems as I am in fault tolerant data. If my motherboard/memory/system drive/processor/mouse dies, chances are I can buy a new one. If I didn't want encryption and the ability to add more disks without re-creating storage/encryption volume, I'd buy 3Ware 9650SE-24M8 and live happily ever after with my RAID6+FreeNAS (or maybe NexentaStor?).

But then I wouldn't be able to add new drives as I go and while e.g. TrueCrypt can allegedly create dynamic volumes on NTFS partitions I haven't ever tried it in a RAID setup (not to mention the encryption concerns related to dynamic volumes). I'm currently running about 5 TB of encrypted data on multiple disks and while I can survive the death of 100% of the drives due to multiple backups, I'd rather not have the system unavailable during data restoration.

So all in all, this is still what I want:

1) fault tolerant - must be able to sustain a minimum of one failed drive, preferably two failed drives
2) encrypted - must be able to encrypt the data
3) expandable - must be able to dynamically add new drives that are (semi)automatically added to the pool, encrypted and made available to the total encrypted storage capacity

and so far I am unaware of a solution that can do it. I need to look into dm-crypt and Linux SW RAID more, maybe there's something there that could work. Or maybe I should wait for ZFS encryption to become stable?

And just so that I wouldn't limit anyones creativity, let's pretend that money is not an issue, that I am an extremely talented Linux/FreeBSD/Windows/OpenSolaris/Malbolge developer and that crypto routines are what I read with my breakfast cereal. (none of these are true) In other words, _all_ ideas are welcome.

Thanks for your help so far, let me know if you can think of anything else.

 

[taltamir]

i meant the whole point of fault tolerance, not the whole point of the thread. while you can easily buy "motherboard/memory/system drive/processor/mouse" if there is a very real risk of loosing your data when replacing the controller with an identical one, then its not exactly an ideal situation. Case in point is RAID5...

As for the OP... so you want to have 5TB of data or so encrypted and expandable and double drive fault tolerant... I can't help you there.

 

[Nothinman]

and so far I am unaware of a solution that can do it. I need to look into dm-crypt and Linux SW RAID more, maybe there's something there that could work.

I just told you that you can do it with that, it's a bit of work but with Linux software RAID6 underneath a dm-crypt volume you'll be able to survive 2 drive failures at once and expand it whenever you want.

And just so that I wouldn't limit anyones creativity, let's pretend that money is not an issue, that I am an extremely talented Linux/FreeBSD/Windows/OpenSolaris/Malbolge developer and that crypto routines are what I read with my breakfast cereal. (none of these are true) In other words, _all_ ideas are welcome.

Money doesn't even come into play except for buying the hardware which is mostly disks and controllers to hold them.

 

[taltamir]

ok. You should really REALLY consider ZFS with RAIDZ (raidz is a special raid5 implementation that works with ZFS file system...). It is amazing. It will provide protection against silent data corrupt and is also very cheap to implement. It adheres to all your requirements and then some with the exception being encryption.

ZFS works best on opensolaris (the original system it was built for). there are incomplete ports of it to other systems since it has gone open source. But solaris is still the most complete and solid implementation.

I suggest that you build a ZFS/RAIDZ server (or RAIDZ2 which is like raid6 for ZFS, ie, two drive failure tolerant) and then put some sort of file encryption on top of it. Independent of the actual file system. Anything else just doesn't even come close to the glory that is ZFS+RAIDZ

http://www.opensolaris.org/os/community/zfs/

 

[rtyle]

With Linux you can do it all:

1. Create a RAID array for fault-tolerance (mdadm --create). I would suggest creating a LVM physical volume from the array (pvcreate) and using it as the basis for a volume group (vgcreate) with one or more logical volumes (lvcreate) with the e2fs filesystem (mke2fs) . This way, you will be able to extend the logical volumes and the filesystems in them when you extend the array.

2. Expand the RAID array (see -add and --grow options to mdadm), expand logical volumes on the array (lvresize) and expand the filesystems in each expanded logical volume (resize2fs).

3. Encrypt the block devices (losetup) that underly a filesystem. For example, you can do this to any of the logical volumes above before creating your e2fs filesystem on it, by way of the losetup device.

 

[taltamir]

except it does not protect the data the way ZFS would. And allows silent corruption.

 

[TXFI]

taltamir, as you might have noticed, I mentioned in my initial post that ZFS encryption is still in alpha/beta stage and hence ZFS on any OS is currently not an option/doesn't meet the requirements. rtyle and Nothinman have the most viable solution at the moment although I don't particularly care for SW RAID.

 

[Nothinman]

although I don't particularly care for SW RAID.

I don't see why, the performance difference is virtually nonexistant for a low use server and the flexibiilty is pretty much infinite compared to a hardware solution.

 

[TXFI]

Nothinman, mostly because it's difficult to find a way to attach 16+ SATA drives to a single system without purchasing a multi-port RAID card. Most of the (inexpensive) SATA controllers seem to have a limit of 4 drives and 2 controllers/system and I'd rather not use port multipliers.

Otherwise your solution is fine so if you can help me figure out how I could add e.g. 17 drives + 2 mirrored system drives to a single PC (without paying tens of thousands) I'd be all set.

 

[Owls]

Did someone shrink Department of Homeland Security's budget? Or is Dick Cheney tossing his man sized safe for encrypted HDs?

If you are going to do SW Raid then your best bet would be PCI-E SATA cards with a 3x SLI motherboard. I don't know what kind of case you are going to be using or what power supply to house 17 drives but at that point you are heading into SAS territory where you should be.

Otherwise it's looking like a house of cards.

edit: Looking at newegg, there's nothing other than 4 port PCI-E cards, I suppose if you fill up the PCI slots with 4 Port SATA cards then you should be fine, if not you're looking at actual raid cards with SAS for that type of need.

 

[DivideBYZero]

Text
Enjoy.

 

[taltamir]

10 watts per drive. 5 watts if its a GP drive. 17 drives will take no more then 170 watts. Less then some video cards. IF coupled with on board graphics then power is not an issue. (before you start about the value of watts in measuring psu output. if you check the amps then it isn't an issue either.).

My understanding is:
1. ZFS encryption is beta.
2. Linux software raid doesn't do compression, doesn't hash, and doesn't protect against bit rot.
3. Any other solution is gonna cost a ton of money. In fact, I am not even SURE there is another solution. GFS (google file system) was developed by google simply because nobody is SELLING the kind of high quality file systems they want. My bet is that the department of defense uses a custom file system as well. (that, or its running NTFS on a windows NT4 server, wouldn't surprise me at all).

I seriously doubt you actually NEED encryption for whatever you intend this database.

 

[TXFI]

Like I said, I'm yet to find a PCI SATA card that would be compatible with more than two of the same cards in the same system. So buying those cheap cards from Newegg won't work as 3+ cards can't co-exist. Hence the only viable choice to add 17+ drives to the same system (that I know of) is to use a hardware RAID card, such as 3Ware 9650SE-24M8.

So the only solution so far that meets the requirements:

1) fault tolerant - must be able to sustain a minimum of one failed drive, preferably two failed drives
2) expandable - must be able to dynamically add new drives that are (semi)automatically added to the pool and made available to the total storage capacity
3) encrypted - must be able to encrypt the data and still handle adding new drives

is to set up a Linux box, buy a 3Ware 9650SE-24M8 and use it in Single Drive mode, attach 17+ SATA drives and use Linux SW RAID + encryption. Thanks to Nothinman and rtyle for your help.

 

[RaiderJ]

Originally posted by: taltamir
ok. You should really REALLY consider ZFS with RAIDZ (raidz is a special raid5 implementation that works with ZFS file system...). It is amazing. It will provide protection against silent data corrupt and is also very cheap to implement. It adheres to all your requirements and then some with the exception being encryption.

ZFS works best on opensolaris (the original system it was built for). there are incomplete ports of it to other systems since it has gone open source. But solaris is still the most complete and solid implementation.

I suggest that you build a ZFS/RAIDZ server (or RAIDZ2 which is like raid6 for ZFS, ie, two drive failure tolerant) and then put some sort of file encryption on top of it. Independent of the actual file system. Anything else just doesn't even come close to the glory that is ZFS+RAIDZ

http://www.opensolaris.org/os/community/zfs/

Completely agree - ZFS is far and away the best way to go. Granted, encryption isn't yet there natively, but there are ways you can encrypt things easy enough outside of the filesystem itself.

And a HUGE benefit to ZFS that I think is oddly overlooked at times is the end-to-end data integrity. Nothing is good without that.

And you can always add disks to the pool - just not to a RAIDZ(2). But if you think about it, does that *really* matter? Three more drives and you have another fault tolerance array to add, or two if you do a mirror.

 

[RaiderJ]

ALL RAID is software RAID. Very important to understand that point and why ZFS was designed to work with direct access to drives. Unless you don't care about data integrity.

As for your multiple drives issue, you CAN use "hardware" RAID controllers in conjunction with ZFS, and actually get benefit from it. Just enable the onboard cache and use JBOD. Allows ZFS full access to the drives and lets it take advantage of the RAM.

So, just enable encryption above the ZFS filesystem and you're set. I'd argue if you are REALLY concerned about data security, you'd offload encryption to a separate module anyway in hardware.