Build Log: 12TB ZFS File Server (6x2TB)

[sygyzy]

Sure thing. Also added in some similar products that I thought might be comparable, although it's really tough to find commercial products that I'd actually consider comparable.

Wow, that is above and beyond what I expected. I was going to do the legwork myself; I didn't expect you to. I know that took some time so truly, thank you! I just saw the new sticky - this forum is looking for a moderator. My vote is for you.

 

[RaiderJ]

Wow, that is above and beyond what I expected. I was going to do the legwork myself; I didn't expect you to. I know that took some time so truly, thank you! I just saw the new sticky - this forum is looking for a moderator. My vote is for you.

Hope it's helpful - it was good for me to put down all of the parts & prices. Gives me a way to see what I've spent on everything and check if it's really a good deal or not. Personally I'm happy with it.

Thanks for your support to take on a moderator position! Unfortunately with my current travel schedule I just don't have the time to spare. Enjoying just sharing my build notes with people

Saw a new Thunderbolt based RAID system here on AT today. 6x2TB drives run $1,999! Double what my system runs!

 

[Dadofamunky]

Man, NICE upgrade. What IOPS is that thing throwing out (not that it's a terribly meaningful benchmark). Love that SATA power cable mod. Go all 2 TB drives and max that thing out!

 

[Suspicious-Teach8788]

Not to threadjack but I built something similar in April-ish.

Core i3 2100
Asus P8H67M-EVO
6x2TB Samsung F4EG
Lian Li V354-B (slightly larger than the Q08 you have in that it supports 7 drives)
8GB DDR3

I ran OpenIndiana with Napp-it. VERY nice. I ultimately scrapped it and gave my mom the i3 processor. I wanted to go for a 10 drive setup instead of 6. 6 drives in Raidz2 seems like a waste because 1/3 of it is used for redundancy. 8TB is nice, but it's not that big of a boost from my main desktop which already has 3TB of storage.

 

[RaiderJ]

Man, NICE upgrade. What IOPS is that thing throwing out (not that it's a terribly meaningful benchmark). Love that SATA power cable mod. Go all 2 TB drives and max that thing out!

Not sure on the IOPS - do you have a good benchmark in mind that I can run to find out? Might be a bit before I can get back to it, I'm on business travel at the moment. Once I get some data on the array I'll swap out the last 750GB drive too - verify the array will rebuild and expand on its own.

 

[RaiderJ]

Not to threadjack but I built something similar in April-ish.

Core i3 2100
Asus P8H67M-EVO
6x2TB Samsung F4EG
Lian Li V354-B (slightly larger than the Q08 you have in that it supports 7 drives)
8GB DDR3

I ran OpenIndiana with Napp-it. VERY nice. I ultimately scrapped it and gave my mom the i3 processor. I wanted to go for a 10 drive setup instead of 6. 6 drives in Raidz2 seems like a waste because 1/3 of it is used for redundancy. 8TB is nice, but it's not that big of a boost from my main desktop which already has 3TB of storage.

That is a very similar build! I decided to go with a RAIDz1 exactly because losing 1/3 of the drives didn't seem worth it. When benchmarking, the RAIDz1 came out faster too - guess the "ideal" number of drives doesn't always make a big difference. Plus, my usage doesn't require tons of performance anyway.

I didn't really want to go bigger than what I had for my old server, but the Lian-Li case was really cool looking. What case did you end up going with once you rebuilt?

 

[bjamm2]

Nice thread. Interesting read as I build my 24bay zfs similar setup. Will be doing two 12drive raidz2 pools. Now if newegg would lift the dumb 5 drive limit on those F4 drives so I can get going

 

[themillak]

they don't have a 20 pack for sale?

 

[RaiderJ]

Update time! I've been out of state on business travel and had some configuration issues with the server. Hopefully resolved now.

Ordered some left-angle SATA data cables to connect all my drives. Cleans up the case a bit and gets rid of more clutter. Will update with a picture when I get a moment.

The configuration issues I ran into were mostly network related - OpenIndiana is flaky with the onboard wireless. It would sometimes connect, then randomly disconnect. I had planned to use a wired connection for the long-term, but during setup it would have been nice to not have to mess with the cable. Nothing I changed in the network settings seemed to help. Once I manually set an IP and used a wired connection things worked fine.

Once up and running with napp-it, I found setting SMB permissions difficult. According to the napp-it guide, you should just have to connect via Windows networking to set ACL's. That sort of works, but what I can't do is add/remove SMB users created in napp-it on Windows. After some playing with the extensions in napp-it, I was able to get it to work.

Now I'm copying over files from 'My Documents' in Win7 to a share on the server (\\server\raiderj). Able to only allow my user account and of course root access to this folder. I've also set up a \\server\media share that everyone has access to, hopefully to make it easy for my HTPC and other devices to stream media without issue.

 

[RaiderJ]

First up, pictures!

Inside Shot w/ Left-Angle SATA Data Cables

Close Up of Drives & Cables

New Server & Old Server Comparison

Yesterday I copied over ~ 1.2TB of data from both my main computer and my old file server. I wanted to test transfer speeds and get some data on the new server before I did a drive swap. Transferring from the old server went at ~10MB/s, limited by the 100Mb Ethernet connection. The new server didn't seem to have any issues handling this transfer and running downloads from newsgroups at the same time.

Today, I powered down the system and swapped out the single 750GB for the 2TB, giving me a total of 12TB of raw space. Booted up into napp-it, and it showed a new drive and a degraded array, as expected. However, it didn't auto-replace and auto-expand the array, complaining of a bad/missing drive label on the new drive.

My guess? New drives should be clear of any existing partitions before adding them to ZFS. Could be useful to help prevent accidental disk overwrites. Either way, I just had to run "# zpool replace server c2t5d0p0" from the command line to start the rebuild process. Altogether it took 51 minutes to resilver the array. No errors and no data issues afterwards. Array size expanded as well!

ZFS RAIDz1 Online Expansion - 51 Minutes (1.2TB data)
Old Array: 5x2.0TB, 1x750GB = ~3.31TB (usable space)
New Array: 6x2.0TB = ~9.70TB (usable space)


I also looked at the rates that the internal disks ran during a rebuild on both servers - the old server ran at ~4.5MB/s, while the new server runs at ~64MB/s. My guess is that the old server's bottleneck is the RAID controller CPU, it simply cannot recalculate the new parity data any faster.

On the new server, I'm guessing 64MB/s is the write limitation of the 2TB drives during a rebuild. CPU stats showed ~82% idle, so that means I'm not hitting any CPU bottlenecks. Overall, 51 minutes to resilver 1.2TB of data is pretty quick! More data would mean longer rebuild times - a full array would require ~7 hours to rebuild a new drive is my guess.



Finally, here are new benchmarks with the full 6x2TB RAIDz1 array! Percentage increases are over the previous benchmark with 5x2TB + 1x750GB.

Test - Speed - %CPU (Increase over previous benchmark)

RAIDz1, 6x2TB Samsung F4EG

• Seq-Wr-Chr - 79 MB/s - 98 (-7%)
• Seq-Write - 293 MB/s - 34 (-5%)
  
• Seq-Rewr - 175 MB/s - 26 (-3%)
• Seq-Rd-Chr - 93 MB/s - 99 (0%)
• Seq-Read - 567 MB/s - 26 (32%)
  
• Rnd Seeks - 561.8 - 1 (-11%)

I expected all my benchmarks to go up! However, since the 2TB drives were previously limited to 750GB, they most likely only used the outer part of the platters, which is the fastest part. Sequential speed however went WAY up, new drives definitely make a difference!

Summary

So - hard to complain about my new setup! It's anywhere from 10-20x faster (and 5x capacity) than my old server.

I'd recommend a ZFS based storage system to anyone that doesn't mind the extra effort and research that's required. None of the hardware I picked up I would consider enterprise or performance grade - in fact I'd say it's all relatively slow. But, compare this server to commercial products in cost or performance, and it's a landslide victory IMO!

 

[postaled]

I liked following your progress throughout the build.

A month or so ago I did something similar myself... but I ended up going raid 5 and ext4 for the fs. Not sure if that will end up kicking me in the butt later or not.

I got a lot of parts pretty cheap for the build...

Phenom X4 945BE
8GB ram
Some open box asus? mobo.. can't even remember now.
5x2TB hitachi drives

Spent just over $500 for the overall build after all costs(not including time)

Not sure on my write speeds with this setup but my read speeds are ~340MB/s and I max out the 1gig ethernet connection when doing file transfers to and from.

Anyone think I should try switching over to ZFS?

 

[RaiderJ]

I liked following your progress throughout the build.

A month or so ago I did something similar myself... but I ended up going raid 5 and ext4 for the fs. Not sure if that will end up kicking me in the butt later or not.

I got a lot of parts pretty cheap for the build...

Phenom X4 945BE
8GB ram
Some open box asus? mobo.. can't even remember now.
5x2TB hitachi drives

Spent just over $500 for the overall build after all costs(not including time)

Not sure on my write speeds with this setup but my read speeds are ~340MB/s and I max out the 1gig ethernet connection when doing file transfers to and from.

Anyone think I should try switching over to ZFS?

I don't think you'll see much of a practical benefit to switching over - you're already getting great speeds and I believe ext4 is pretty robust.

Personally, I've found ZFS/OpenIndiana to be pretty frustrating on the software end of things. Lots of little tweaks needed, very limited support, and other such things. Some of that is my fault since I used hardware of my own choosing.

That said, it certainly can't hurt to test out ZFS. Run it on a separate drive or partition, maybe throw a small data disk at it too. See how it works for you - you certainly have space available in your case for another drive or two.

 

[postaled]

I don't think you'll see much of a practical benefit to switching over - you're already getting great speeds and I believe ext4 is pretty robust.

Personally, I've found ZFS/OpenIndiana to be pretty frustrating on the software end of things. Lots of little tweaks needed, very limited support, and other such things. Some of that is my fault since I used hardware of my own choosing.

That said, it certainly can't hurt to test out ZFS. Run it on a separate drive or partition, maybe throw a small data disk at it too. See how it works for you - you certainly have space available in your case for another drive or two.

Yeah, transfer wise this thing has been great to me.

Maybe once I fill up the thing I'll add more drives and try out zfs. Actually I think we have an old san at work that I may try zfs out on if I can

I am just running debian on this pc for my file server needs and I was suprised at not needing to tweak samba configuration files at all to get the speeds I am at.

Its been an amazing media server/subsonic server as well(it seems to browse folders on subsonic much faster than my win7 desktop did)

 

[Dahak]

Just curious, for the sata power connectors is there any limitation on how many can be put on one cable aside from psu output?

I had always thought there was some electrical reason that there is not more than usually 4 connectors ( the max ive seen) on one cable. I see you have 6 and I have seen some builds that have had 8 on there

 

[RaiderJ]

Just curious, for the sata power connectors is there any limitation on how many can be put on one cable aside from psu output?

I had always thought there was some electrical reason that there is not more than usually 4 connectors ( the max ive seen) on one cable. I see you have 6 and I have seen some builds that have had 8 on there

That should depend on a couple of things. My PSU is single rail, so the full power is available over any given connection. Multiple rail PSU's you'd have to make sure you're not exceeding the specific rail that drives are connected to.

I'd have to do some calculations, which would be to take the max load on the cable at drive spin up, and verify that the voltage drop across the cable isn't outside of specification. Each drive takes 2A at spin-up, so I'm pulling 12A total for a short while, then much less during regular operations.

My PSU is rated at 34A on the 12V rail, not sure on the 5V. I think hard drives primarily use the 12V rail on spin-up? Not quite sure. Either way, I have plenty of current available, and I'll assume that voltage drop due to wire resistance and wire length isn't a major factor.

For how many drives I could run off a single cable, I'd want to see how many amps everything else needs to run. I'll assume it needs 14A max for all other devices (which is very generous), leaving ~20A leftover on the 12V rail. That would make me think that I would want to run no more than 10 drives total, and probably not more than 8 of them on a single cable because of voltage drop.

In practice, I bet that you'd be limited on the number of drives connected to a single cable by routing concerns more than anything. Even if you have 20 drives all next to each other, chances are you'd want multiple cables just to make for a cleaner install.

 

[bigi]

Good stuff. I use 8 HDDs and highly recommend this

or this

 

[bigi]

That should depend on a couple of things. My PSU is single rail, so the full power is available over any given connection. Multiple rail PSU's you'd have to make sure you're not exceeding the specific rail that drives are connected to.

I'd have to do some calculations, which would be to take the max load on the cable at drive spin up, and verify that the voltage drop across the cable isn't outside of specification. Each drive takes 2A at spin-up, so I'm pulling 12A total for a short while, then much less during regular operations.

My PSU is rated at 34A on the 12V rail, not sure on the 5V. I think hard drives primarily use the 12V rail on spin-up? Not quite sure. Either way, I have plenty of current available, and I'll assume that voltage drop due to wire resistance and wire length isn't a major factor.

For how many drives I could run off a single cable, I'd want to see how many amps everything else needs to run. I'll assume it needs 14A max for all other devices (which is very generous), leaving ~20A leftover on the 12V rail. That would make me think that I would want to run no more than 10 drives total, and probably not more than 8 of them on a single cable because of voltage drop.

In practice, I bet that you'd be limited on the number of drives connected to a single cable by routing concerns more than anything. Even if you have 20 drives all next to each other, chances are you'd want multiple cables just to make for a cleaner install.

This. Also, keep in mind that HDDs require most power during start up process. If RAID card allows it, one can set up staggered HDD spin up with given interval to overcome this issue.

My controller allows it and I've set it up for 1 second delay with 8HDDs total. I don't reboot often, but nevertheless it has been working fine for years.

 

[RaiderJ]

This. Also, keep in mind that HDDs require most power during start up process. If RAID card allows it, one can set up staggered HDD spin up with given interval to overcome this issue.

My controller allows it and I've set it up for 1 second delay with 8HDDs total. I don't reboot often, but nevertheless it has been working fine for years.

Staggered boot is definitely a good option to take advantage of when it's available. I don't recall seeing it on any consumer class motherboard SATA controller though.

For most desktop users, I'd imagine that the power draw, even at spin-up, is very small when compared to things like video cards and CPU's.

For a rackmounted hard drive array that's dedicated to storage, power supplies are probably much more conservative. I'd think a PSU would be designed to run extremely efficiently, which would mean a narrower output band to aim for. Staggered boot would help maintain that, and keep the peak current from getting too high. Plus, I would think it would help to keep voltage drop to a minimum which would benefit reliability concerns with the drives.

 

[Dahak]

Ahh Ok, thanks for the info, which is what I thought as well but like having confirmation

 

[mds2004]

Extremely nice build. So nice that I modeled mine after yours. Did you have any issues installing Openindiana? After selecting the install on the grub it seems to be freezing on the OS banner.

 

[RaiderJ]

Extremely nice build. So nice that I modeled mine after yours. Did you have any issues installing Openindiana? After selecting the install on the grub it seems to be freezing on the OS banner.

That's a great compliment! I hope the build works out well for you too.

For the install, do you have "Legacy USB" disabled in the BIOS? I found that in order to install OI, I had to boot the optical off a SATA port on the motherboard while USB was disabled. Makes re-installs a real pain, since I have to unplug a data drive.

I used Clonezilla to make a copy of my 10GB OS partition once done, that way I can "re-install" by copying that partition back to the hard drive.

 

[mds2004]

That's a great compliment! I hope the build works out well for you too.

For the install, do you have "Legacy USB" disabled in the BIOS? I found that in order to install OI, I had to boot the optical off a SATA port on the motherboard while USB was disabled. Makes re-installs a real pain, since I have to unplug a data drive.

I used Clonezilla to make a copy of my 10GB OS partition once done, that way I can "re-install" by copying that partition back to the hard drive.

Maybe that is my issue. I am trying to install the OS onto a USB drive and am installing through an external DVD drive via USB.

 

[RaiderJ]

Maybe that is my issue. I am trying to install the OS onto a USB drive and am installing through an external DVD drive via USB.

OI, Nexenta, and OpenSolaris would all fail to install off a thumb drive, or off optical on a SATA port not on the motherboard. Seems to be an issue with more than just this motherboard. A real pain, hopefully it will be fixed down the road.

 

[mds2004]

OI, Nexenta, and OpenSolaris would all fail to install off a thumb drive, or off optical on a SATA port not on the motherboard. Seems to be an issue with more than just this motherboard. A real pain, hopefully it will be fixed down the road.

I wanted to give a quick update for anyone who decides to follow this build and has the same issues I ran into.

First I was able to install off of a USB optical. The reason it would not install at the beginning was because OI does not support Intel HD3000 graphics (ie. my Sandy Bridge). What I did to bypass this was take out the Sata card and ran a e-Sata to Sata cable to the OS drive, leaving the 6 data drives on the motherboard.

After this I found an old graphics card at work (Nvidia 7300GS) and everything installed fine.

 

[RaiderJ]

I wanted to give a quick update for anyone who decides to follow this build and has the same issues I ran into.

First I was able to install off of a USB optical. The reason it would not install at the beginning was because OI does not support Intel HD3000 graphics (ie. my Sandy Bridge). What I did to bypass this was take out the Sata card and ran a e-Sata to Sata cable to the OS drive, leaving the 6 data drives on the motherboard.

After this I found an old graphics card at work (Nvidia 7300GS) and everything installed fine.

I'm a bit confused what you did, you had a USB optical drive to install OI, but then had an eSATA card in the PCIe slot? Your OS drive was on the eSATA card?