NAS/Data Server - How would you do it?

[NickOlsen8390]

Sorry if this gets asked alot, I don't lurk much on the data side of the house here at AT.

In the market for a new NAS/Dataserver.
Currently, I'm running 5x 1tb drives in a software raid 5 array under CentOS 5.x
This is fine and all. But I'm wanting to build something new, With more space. Be pretty reliable, Not super expensive.

Now, If the solution you guys recommend is PC based, I'd reuse my current hardware (maybe not the drives, they are seagates and are starting to fail lately :S).
I'm aiming for 10TB or so. Raid 5/6/50 (what do you guys think?)
What hardware would you use? (drives, raid cards (software raid?)) My past experiences with mdadm have me a bit shy of using it again. So I'm leaning toward hardware based raid.
What FS would you use?

This device/machine runs in the rack with the few other machines I have, And is "epicenter" of all data. My desktop rig just has an install of windows, All music/movies...etc is stored on my current data box. I access it via samba or NFS.
Any of the other machines that need any type of storage just NFS mount to it. As the other machines just have small disks for the OS on board.
All devices connect to my 24 port Gig Switch.

Has to be relatively fast. I get anywhere from 30 to 100MB/s off my current array over GigE depending on what I'm moving.
Also, Ease of use is a plus. Something like how the Drobo devices will auto rebuild and resize when you swap disks would be nice. (I think the drobo PRO stuff would be perfect if it wasn't only iSCSI)
All the while being very reliable and redundant. Data loss isn't really an option. And thats a lot of data to backup off site.

Thanks for the input guys.

 

[Fallen Kell]

Personally, I would check/test your system with the OpenSolaris hardware test tool: http://www.sun.com/bigadmin/hcl/hcts/device_detect.jsp

If everything is detected or if there is a third-party driver, I would install OpenSolaris or Solaris11 express x86, and setup your disks in a ZFS filesystem. You will need a separate install disk (or 2 if you want to mirror them), but otherwise, use ZFS to setup all the other drives into their own zpool using raidz or raidz2 (depending on how much redundancy you want for disk failures). You do need 4GB of RAM or more for really good performance.

I ran a crap ton of iozone benchmarks on different setups/configurations as well as comparing EXT3, UFS, and ZFS performance using both the hardware raid controller, and the software raid, as well as combinations there in, on the same exact hardware. In all cases, ZFS consistently beat the other solutions in just about every metric (except system usage). However, if this is a dedicated box, the only time you care about that is if it is maxed out at 100% CPU (ZFS will use up all available memory by default, as it uses it both as a cache and buffer for the read/write operations, you can configure it to not buffer writes to the disk for more data integrity in case of power failure).

You really only need to know a few commands for zfs, "zpool" and "zfs", and their options. Zpool deals with creating, destroying, and managing the pools of disks. Zfs deals with creating/destroying/managing the ZFS filesystems that you have on your disk pool. There are some web based management capability, but the reality of it is the command line is much easier and concise.

For instance:
zpool create data_pool raidz c0t0d0 c0t0d1 c0t0d2;

zfs set sharenfs=yes data_pool;

zfs set sharesmb=on data_pool;

You just created a ZFS equivelant to a software raid5 which is mounted as "/data_pool" and shared out via NFS and Samba. You can specify NFS options in there instead of just saying "yes", so you could do "zfs set sharenfs=root=myserver" or whatever else you want to do in terms of restricting the share. If you don't want the disks mounted in a directory with the same as the pool name, you can change it with "zfs mountpoint=/new_mount_point data_pool".

 

[Nothinman]

If you're considering ZFS, I'd say go with FreeBSD instead. It's future is a lot less questionable. I have a vague recollection of ZFS having some odd expansion-related limitations that weren't present in other things like Linux software RAID but I can't remember the specifics right now.

For me, I'd probably just go with Debian, mdadm and XFS. Probably with LVM between the last 2 if I was thinking about breaking it up into multiple volumes. ZFS checksumming is nice, but it's not worth the tradeoff of having to use and maintain an OpenSolaris or FreeBSD box, IMO.

 

[NickOlsen8390]

Fallen Kell, That looks like a great idea.
Only thing holding me back on that is how unfamiliar I am with Sun*. You know what they say about teaching an old dog new tricks. I'll play around with it in a vm for a bit and see what I figure out. Something CentOS or windows based would be best for me, As I know that best. That or some 3rd party device that does everything in the background autonomously.

 

[NickOlsen8390]

If you're considering ZFS, I'd say go with FreeBSD instead. It's future is a lot less questionable. I have a vague recollection of ZFS having some odd expansion-related limitations that weren't present in other things like Linux software RAID but I can't remember the specifics right now.

For me, I'd probably just go with Debian, mdadm and XFS. Probably with LVM between the last 2 if I was thinking about breaking it up into multiple volumes. ZFS checksumming is nice, but it's not worth the tradeoff of having to use and maintain an OpenSolaris or FreeBSD box, IMO.

I've heard good things about XFS, Specially on larger arrays.

 

[Nothinman]

I've heard good things about XFS, Specially on larger arrays.

It's not a layer-violating, all-in-one fs like ZFS or BTRFS but I've been using it since it's stable release on Linux around 10 years ago now with virtually no problems. But I have no idea how it compares performance-wise with ext4 since I haven't had any reasons to bench the two.

 

[pjkenned]

If you're considering ZFS, I'd say go with FreeBSD instead. It's future is a lot less questionable. I have a vague recollection of ZFS having some odd expansion-related limitations that weren't present in other things like Linux software RAID but I can't remember the specifics right now.

For me, I'd probably just go with Debian, mdadm and XFS. Probably with LVM between the last 2 if I was thinking about breaking it up into multiple volumes. ZFS checksumming is nice, but it's not worth the tradeoff of having to use and maintain an OpenSolaris or FreeBSD box, IMO.

Odd expansion related things are basically that ZFS doesn't have standard online capacity expansion. It kinda stinks.

Good news is that you can get cheap SAS 1068E or SAS 2008 based cards and set up the system inexpensively. Here's a LSI->OEM vendor cross matrix that we are trying to setup to source cheap LSI based cards that do disk pass-through for software RAID implementations.

Depending on how big you are going, NexentaStor CE is a pretty easy way to setup a ZFS system.

 

[Nothinman]

Odd expansion related things are basically that ZFS doesn't have standard online capacity expansion. It kinda stinks.

Good news is that you can get cheap SAS 1068E or SAS 2008 based cards and set up the system inexpensively. Here's a LSI->OEM vendor cross matrix that we are trying to setup to source cheap LSI based cards that do disk pass-through for software RAID implementations.

Depending on how big you are going, NexentaStor CE is a pretty easy way to setup a ZFS system.

I knew it wasn't able to do on-line expansion of some part, but couldn't remember which layer since I don't use ZFS. Wouldn't you think this would be a high priority feature since virtually every other hardware and software RAID implementation out there does OCE now?

And I wouldn't call a hardware bandaid to work around that limitation good news. Not having your array tied to one brand, model, etc of hardware controller is one of the main benefits of software RAID, IMO.

 

[pjkenned]

I knew it wasn't able to do on-line expansion of some part, but couldn't remember which layer since I don't use ZFS. Wouldn't you think this would be a high priority feature since virtually every other hardware and software RAID implementation out there does OCE now?

And I wouldn't call a hardware bandaid to work around that limitation good news. Not having your array tied to one brand, model, etc of hardware controller is one of the main benefits of software RAID, IMO.

Totally agreed. I would be using ZFS a lot more if it had it. You can add disks, but it isn't pretty.

The reason one uses LSI controllers that can operate in initiator-target (IT) or pass-through mode is so you do not get tied into one vendor. LSI and Areca do not write RAID metadata on pass-through disks from what I have seen (Adaptec did last time I checked). I have taken drives from an Areca 1680LP, put them on a LSI 1068e without issue because of this.

The other big reason you do this is because when you have software RAID you basically have your static disk controller and your software RAID platform trying to do things like error correction on the disks. Adding hardware RAID in the middle means software RAID is not talking to the disk controller directly anymore.

 

[sxr7171]

Nexentastor. That's what I used. Great performance no headaches. The hardware it supports is listed. Get an LSI HBA, set it up and use.