Build Log: 12TB ZFS File Server (6x2TB)

[RaiderJ]

EDIT: See the final build at post [post=32022556]#35[/post]!

I've been waiting to update my current file server, an Atom based system with 4x750GB drives in a RAID 5. Gives me 1.99TB of usable space running on WinXP Pro. Since I'm low on space, time to upgrade!

My usage pattern for the file server is movie storage/playback and backups of my other machines. Nothing intensive, as my HTPC handles playback and streams data from the file server over Ethernet.

Old File Server

• Intel Atom (1.6GHz, x86)
• 2GB RAM
• 4x750GB (Data), 80GB (OS)
• Intel PCI RAID 5
• Slimline Optical

Initially, I was looking to just update the drives to 2TB. However, I had been reading about ZFS and felt that was an excellent base for a file server. The old file server hardware just wouldn't cut it for ZFS - only the hardware RAID controller allowed the Atom to function as it was.


Component Selection - See post [post=31907561]#22[/post] for build/cost list
So, off to figure out what I wanted in a new file server. Originally I had wanted to keep the same case - it's a really compact and functional system. Four hot-swap bays and room for an additional 2.5" drive beneath the optical drive.

Motherboard
For a new motherboard, I looked around for a mini-ITX motherboard that had ideally six SATA ports and was compatible with Solaris (or a ZFS-supporting fork). That narrowed things down to an Intel board that would support 8GB+ of RAM and a decent processor to handle the ZFS workload. With that in mind, I found one on Newegg that fit the bill:

ZOTAC H55ITX-C-E Link

• LGA 1156
• 2xDDR3 (8GB Max)
• 6 SATA 3.0Gb/s
• USB 3.0, 1xeSATA, 10xUSB2.0
• Wifi, Gigabit Ethernet
• PCIe x16

The motherboard had everything I needed! Supporting 16GB of memory would have been preferable, but not critical at this point in time. I imagine by the time I need to have more RAM I'll be due for another hardware upgrade. I don't plan to use any of the fancier ZFS features (encryption, deduplication, or tiered SSD caching), so I don't see any performance issues.

CPU
For the CPU, I just picked up a cheap dual-core i3-540. TDP is 73W, so it shouldn't run too hot. I'm not quite sure how to gauge CPU performance requirements in ZFS, but it's something I'll investigate once the build is complete.

RAM, OS Drive, Optical, HSF
Nothing exciting here, just 2x4GB sticks of DDR3, a $15 40GB 2.5" SATA drive from MicroCenter, a DVD burner I had laying around, and a slimline CPU cooler. The OS drive and optical drive weren't part of my initial parts list, as I was going to re-use the existing ones in my old file server. Those I picked up once I decided to go with a new case.

Server Case
I had previously been unable to find a case I liked better than my old file server, but I saw this case and decided it would be perfect.

Lian-Li PC-Q08B Link

• 6x3.5"
• 1x5.25"
• 1x2.5"
• ATX PSU (150mm max length)
• USB 3.0

So, now I had a way to run 6 data drives, and still have an OS drive and an optical drive (which I rarely use). But, my motherboard only supports 6 SATA ports. To add a couple SATA ports, I picked up a cheap controller that is compatible with ZFS . I won't get any great speeds out of the controller, but just for an optical and OS drive it would be fine.

Power Supply
This was a tricky one - the case only supports shorter power supplies, so most modular ones are out. As cramped as the case was going to get, I felt sure that modular was the best way to go. I looked around for awhile, hoping to find a short modular one, and sure enough, SilverStone makes one!

SilverStone Strider Plus ST50F-P Link

I also picked up a set of shorter cables to go with the PSU, no sense in having extra cable in a case that doesn't need it. I might end up custom making some SATA power cables once I get everything put together. EDIT: [post=31912258]Done![/post]

Data Drives
I initially picked up 4x2TB drives, the Samsung F4 EcoGreen drives. They would have been a perfect replacement for my older 750GB drives, but with ZFS they present a problem: they are 4K "Advanced Format" sector drives with 512k sector emulation. I'm still researching how best to handle this, but from what I understand so far it's more of a performance problem than reliability. I'm confident there will be a decent enough workaround in due time. So for now, I just picked up a couple more F4 drives to round out the build.

Server Build
Right now I'm just waiting on the case and controller to arrive (due today via UPS). I've run full sector scans on all the hard drives to make sure they're 100%, a memtest on the RAM, and installed OpenIndiana (build 148) on the 40GB drive.

Only quirk so far is that to boot the LiveCD, I had to disable legacy USB in the BIOS. Not quite sure why that was needed, and only after some googling did I find that to be the solution. Before that, the install would hang before even fully booting up. I think that I should be able to plug in the OS drive to the controller on the motherboard and be able to continue where I left off. If not, it's pretty quick to re-install.

I'm not yet sure how I want to arrange my drives, as the whole 512k/4K presents some issues with how data is striped across the array. According to this post, a 6 drive RAIDZ2 is preferable to a 6 drive RAIDZ1. I would rather lose a single drive in the array to parity data, but a RAIDZ2 would be better for uptime and give me a breather if a drive crashes.

Next Steps
Once I get in the case and controller, I'll post pictures of the build and a log of how I go about configuring the system!

 

[Yellowbeard]

Cool build,sub'd.

I just redid my FreeNAS rig. I am a complete NOOB to ZFS but my opinion is that you made a good call on the 8GB of RAM. I have 16GB in mine. While doing a simple file transfer from the old server to the new server while also loading files from my main PC to the new FreeNAS/ZFS rig, I have seen the memory usage go over 12GB according to the system monitor in FreeNAS.

As cheap as they are, I have considered grabbing a couple more of those Hitachi 2TB drives and moving to a RAIDZ2 setup for peace of mind.

 

[RaiderJ]

First Day Build Notes
Finally received the case and SATA controller and started the build. Everything worked fine until I plugged in the controller card - the system wouldn't boot. Figured out that there was a new BIOS for my board that would fix this.

A146PA07.ROM
.8Mbit Flash size
.Added support on "Multi Display Adaptor " - Integrated Graphic + Add-On Display Card
.Added support on PCI-E (x1) Cards - TV Card, RAID Card
(Aug/2010)

So, with that in place, I could now boot the system with both the 40GB OS drive and optical drive connected. However, the drives were only seen if I set the jumper on the controller to enable the BIOS. I know that ZFS prefers to see bare drives connected, no RAID. Since the BIOS on the controller was flashed with the RAID BIOS, I reflashed over the latest SIL3132 non-RAID BIOS. Easy enough from a bootable flash drive into FreeDOS.

Note: To flash the controller with the non-RAID BIOS, I had to use the SIL3132 update utility. The utility makes you select which chip is on the controller, which in my case is the 1Mb Pm39LV010.

Now the drives were seen by the motherboard BIOS as IDE drives, bootable, and no RAID BIOS to potentially cause problems. With the RAID BIOS I had found I was unable to fully boot off the OpenIndiana LiveCD. This was also the case with the non-RAID BIOS - I'm not exactly sure why. My workaround was to connect the optical drive to the motherboard SATA port, boot the LiveCD, and (re)install OpenIndiana to the OS drive connected to the controller. No issues there.

Once OpenIndiana was installed, I found that I still couldn't boot into the OS with "Legacy USB" enabled/auto in the motherboard BIOS. So, I'm still stuck having that disabled. Once into the OS, OpenIndiana has no problem seeing USB drives - but it's still not ideal since if I want to boot from a USB stick I have to change the BIOS setting first. Not sure how to fix this at the moment.

Will update with pictures of the case with all the gear installed soon! It's actually a pretty clean install - the modular PSU is a huge help.

EDIT: Picture of the build! Don't have the data drives connected just yet - will work that next.

 

[Silenus]

First of all congrats! Nice little ZFS build.

Good call on putting the OS and DVD on the controller card. They don't need to be fast. Having the data drives all on the motherboard will be better and will enable you to work around the 4K thing pretty easily. I have the same Samsung F4's (HD204UI) and run 9 of them in a RAIDZ2 + spare in our NAS here at work. Here is a relatively simple way to get them aligned:

1) Download ZFSguru stable LIVE CD ver 0.1.7 (http://zfsguru.com/download)
2) Boot to ZFS guru and then format each of your 6 drives drives from the web gui utility. NOTE: Format the drives with GEOM...NOT GPT (Solaris won't recognize GPT partitions).
3) Create your RAIDz2 pool in the web gui and select the 4K override option!
4) Export the pool (do this locally from command line. "zpool export poolname")
5) Now boot back up to your OI system and import the pool. It should find it and it will be correctly aligned for the drives.


Alternately...don't worry about the 4K issue. It IS only a performance difference, not a reliability one. Chances are quite good that you will still easily be able to saturate gigabit even with the slightly reduced performance of the drives being non-aligned.

 

[RaiderJ]

First of all congrats! Nice little ZFS build.

Good call on putting the OS and DVD on the controller card. They don't need to be fast. Having the data drives all on the motherboard will be better and will enable you to work around the 4K thing pretty easily. I have the same Samsung F4's (HD204UI) and run 9 of them in a RAIDZ2 + spare in our NAS here at work. Here is a relatively simple way to get them aligned:

1) Download ZFSguru stable LIVE CD ver 0.1.7 (http://zfsguru.com/download)
2) Boot to ZFS guru and then format each of your 6 drives drives from the web gui utility. NOTE: Format the drives with GEOM...NOT GPT (Solaris won't recognize GPT partitions).
3) Create your RAIDz2 pool in the web gui and select the 4K override option!
4) Export the pool (do this locally from command line. "zpool export poolname")
5) Now boot back up to your OI system and import the pool. It should find it and it will be correctly aligned for the drives.


Alternately...don't worry about the 4K issue. It IS only a performance difference, not a reliability one. Chances are quite good that you will still easily be able to saturate gigabit even with the slightly reduced performance of the drives being non-aligned.

Thanks for the advice! I'll give that utility a spin later today. I figured once I get the data drive put together I can leave them mostly alone - the OS issues with OpenIndiana won't affect the pool once built.

Definitely not concerned about performance - my existing server runs at 33MB/s max over the PCI controller. Good enough for movies!

Quick question: What OS are you running and why?

 

[Silenus]

I am running Nexenta community edition on my NAS. It was the first one I spent a lot of time with and got used to, though I have also experimented with OI and SE 11 (both with Napp-It). Although it is not the simplest design I like the built in web gui that comes with Nexenta. It is easy to see the status of all disks, pool, setup shares or iscsi targets. It is also nice to be able to see at glance the current CPU, network, and disk activity from the dashboard. One of the downsides however is Nexenta slightly awkward non-standard command structure when doing something from command line. It doesn't always recognize the standard ZFS commands and uses slightly different names for things. That is one of the advantages of the the pure Solaris based ZFS systems. They do have their advantages. One nice new addition is a real zfs replication feature Gea is adding to napp-it (it is going to be a paid plug, but way cheaper than nexenta licensing). Lastly, OI and SE are built on newer ZFS versions and developing faster than Nexenta so if there are advanced features you want to use (like encryption), you'll need one of them .

 

[ViviTheMage]

Neat little build. what controller card did you get?

 

[RaiderJ]

Neat little build. what controller card did you get?

Just a cheapo PCIe x1 from Newegg that is supported under Solaris. Wouldn't stick data drives on it, but for OS and optical it's fine (since they won't be bottlenecked by the x1 connection).

 

[RaiderJ]

Day Two - Data Drives & OpenIndiana & napp-it

First off, a picture of the system with the power and data cables connected.

I haven't finalized the cable layout - going to get some 90 degree data cables and route them back through the drive cage. For the power cable, I'm looking for a place where I can buy the cable and SATA adapters myself. The current power cables are way too bulky, although everything does fit in the case just fine.

OS
My current install of OpenIndiana is up and running, and for simplicity, I decided to just test out napp-it. It's just a single command from the terminal to get it installed, then a bit of configuring. I basically just followed the napp-it guide and truly had my pool and RAIDz2 up and running in a few minutes, including SMB shares.

It took me a bit to figure out how to enable TLS for Gmail alerts/status reports, but the readme on the napp-it GUI under TLS has all the terminal commands to use.

So, truly a quick and easy setup! Also ran some benchmarks. I've bolded the tests that napp-it had bolded in the results - from what I've read it's the sequential disk performance that's the most important.

Test - Speed - %CPU

• Seq-Wr-Chr - 78 MB/s - 98
• Seq-Write - 198 MB/s - 27
• Seq-Rewr - 113 MB/s - 17
• Seq-Rd-Chr - 89 MB/s - 99
• Seq-Read - 309 MB/s - 13
• Rnd Seeks - 481.3/s - 1

Currently I have only 2 of my 2TB drives in the new server, the remaining 4 are 750GB drives from the old server. What I want to do is test the online RAID expansion by swapping in the new 2TB drives and see if at the end I get my 8TB of storage.

 

[Silenus]

Very nice! Your sequentials may even go up a bit when the other 2TB drives are in. But it doesn't matter since you are way past gigabit line speed anyway. Wheres the 10Gb Ethernet for desktops already?! Oh and cheap switches...

 

[RaiderJ]

Very nice! Your sequentials may even go up a bit when the other 2TB drives are in. But it doesn't matter since you are way past gigabit line speed anyway. Wheres the 10Gb Ethernet for desktops already?! Oh and cheap switches...

I can't imagine speeds won't increase as I swap in the 2TB disks and then redo the pool using 4K alignment. Even so, it's pretty good speeds with 3 different types of drives (Samsung 2TB, WD 750GB, Seagate 750GB), and two sector types.

 

[Gigantopithecus]

Looks great! Looks like it's performing well, too. Quick question: how's that slim Thermaltake cooler working for you? Looks like a good, inexpensive step up from the stock cooler I'm using in my Q08 build right now.

 

[fffblackmage]

Just a cheapo PCIe x1 from Newegg that is supported under Solaris. Wouldn't stick data drives on it, but for OS and optical it's fine (since they won't be bottlenecked by the x1 connection).

http://detonator.dynamitedata.com/c...com/Product/Product.aspx?Item=N82E16816132008

I'm not sure if the PCIe x1 would even bottleneck anything. PCI 1.0/1.1 provides 250MBps.

Do you know if the devices you've attached to the SATA/RAID controller are running in AHCI mode or IDE mode? I have a couple of those controller cards myself, but the HDDs I've attached to them are listed as IDE devices (I'm using FreeBSD 8.2). So far, I haven't been able to figure out a way to have them run in AHCI mode.

BTW, you picked out a really nice little case (with room for 6 HDDs!). In comparison, my own NAS is inside a monolithic P183.

 

[RaiderJ]

Looks great! Looks like it's performing well, too. Quick question: how's that slim Thermaltake cooler working for you? Looks like a good, inexpensive step up from the stock cooler I'm using in my Q08 build right now.

Seems to work great. Haven't really had the system up enough to fully test temperatures during load, but it's a quiet cooler when I've had the machine on so far. There's plenty of room between the heatsink and the PSU, so you could fit a larger cooler I'm sure. Originally I was going to put the system in my old case, so I didn't really need the slim cooler.

 

[RaiderJ]

I'm not sure if the PCIe x1 would even bottleneck anything. PCI 1.0/1.1 provides 250MBps.

Do you know if the devices you've attached to the SATA/RAID controller are running in AHCI mode or IDE mode? I have a couple of those controller cards myself, but the HDDs I've attached to them are listed as IDE devices (I'm using FreeBSD 8.2). So far, I haven't been able to figure out a way to have them run in AHCI mode.

BTW, you picked out a really nice little case (with room for 6 HDDs!). In comparison, my own NAS is inside a monolithic P183.

The controller chip itself actually has a limitation, per this website: http://blog.zorinaq.com/?e=10

Warning: the overall bottleneck of the PCIe link is 150-175MB/s, or 75-88MB/s/port, but the chip has a 110-120MB/s bottleneck per port. So a single SATA device on a single port cannot fully use the 150-175MB/s by itself, it will be bottlenecked at 110-120MB/s.

But, the limitation isn't a problem for an optical and OS drive. Just wouldn't be good for data drives. Both drives run in IDE mode, I'm not sure if there is any way to get them to run in AHCI mode. I don't see it as a particular issue, you're not going to be needing NCQ or hot swap on the drives attached to that controller.

 

[fffblackmage]

Both drives run in IDE mode, I'm not sure if there is any way to get them to run in AHCI mode. I don't see it as a particular issue, you're not going to be needing NCQ or hot swap on the drives attached to that controller.

I just thought it was weird that the controller advertises support for how-swapping and NCQ, which isn't supported by IDE. Or is it?

 

[RaiderJ]

I just thought it was weird that the controller advertises support for how-swapping and NCQ, which isn't supported by IDE. Or is it?

I believe that is the case, but only when using the RAID BIOS (controller supports NCQ & hot-swapping). I flashed over the non-RAID BIOS, hoping that it would fix my non-booting optical drive (which it didn't). At least this way the motherboard BIOS can see the drives, and it's my understanding ZFS works best with non-RAID BIOS.

EDIT: ZFSGuru boots fine from the optical drive on the controller. Other discs might boot too.

 

[Emulex]

no kidding i have 4 lsi cards with 8 fan-out cables for like $50 each - zfs love buy me 4 cards on 4 slots would be alot of bandwidth

 

[RaiderJ]

no kidding i have 4 lsi cards with 8 fan-out cables for like $50 each - zfs love buy me 4 cards on 4 slots would be alot of bandwidth

Wow, what setup do you have? How many drives are you running?

 

[RaiderJ]

First of all congrats! Nice little ZFS build.

Good call on putting the OS and DVD on the controller card. They don't need to be fast. Having the data drives all on the motherboard will be better and will enable you to work around the 4K thing pretty easily. I have the same Samsung F4's (HD204UI) and run 9 of them in a RAIDZ2 + spare in our NAS here at work. Here is a relatively simple way to get them aligned:

1) Download ZFSguru stable LIVE CD ver 0.1.7 (http://zfsguru.com/download)
2) Boot to ZFS guru and then format each of your 6 drives drives from the web gui utility. NOTE: Format the drives with GEOM...NOT GPT (Solaris won't recognize GPT partitions).
3) Create your RAIDz2 pool in the web gui and select the 4K override option!
4) Export the pool (do this locally from command line. "zpool export poolname")
5) Now boot back up to your OI system and import the pool. It should find it and it will be correctly aligned for the drives.


Alternately...don't worry about the 4K issue. It IS only a performance difference, not a reliability one. Chances are quite good that you will still easily be able to saturate gigabit even with the slightly reduced performance of the drives being non-aligned.

I'm going through this process now - have a question you might be able to answer:

Does the version that ZFSGuru uses to create the RAIDz array allow for auto-expansion? I want to test that feature, so right now I have 5x2TB + 1x750GB (going to swap in the last 2TB). Also, should I do a ZFS upgrade from napp-it?


Also ran some benchmarks from this tool on the array - one using the drives in a RAIDz2, and another in a RAIDz1:

4K Sector Size Override
RAIDz2: 366.3 MB/s Read; 249.9 MB/s Write
RAIDz1: 412.5 MB/s Read; 289.2 MB/s Write

512Kb Sector Size
RAIDz2: 370.3 MB/s Read; 244.3 MB/s Write
RAIDz1: 393.3 MB/s Read; 280.4 MB/s Write

Odd, I'd have figured at least my write speed would have gone down on the z1, due to the 128Kb stripe size being split across 5 disks. Maybe that performance hit isn't as much as the performance increase by striping across 5 disks versus 4?


EDIT: Benchmarks after importing the new RAIDz1 pool with napp-it. Quite a bit faster, due to the RAIDz1 and in part since I'm running 5x2TB disks versus the 2 before. Curious to see what replacing the final 750GB disk does to performance.

Test - Speed - %CPU (increase versus earlier benchmark)

• Seq-Wr-Chr - 85 MB/s - 98 (9%)
• Seq-Write - 309 MB/s - 36 (56%)
  
• Seq-Rewr - 180 MB/s - 26 (59%)
• Seq-Rd-Chr - 93 MB/s - 99 (4%)
• Seq-Read - 430 MB/s - 19 (39%)
  
• Rnd Seeks - 629.2/s - 1 (31%)

 

[sygyzy]

Wow awesome bill. Would you mind posting a build and price list? I am curious how this compares to buying say a QNAP or Synology 4-6 bay in terms of cost, convenience, time, etc.

 

[RaiderJ]

Wow awesome bill. Would you mind posting a build and price list? I am curious how this compares to buying say a QNAP or Synology 4-6 bay in terms of cost, convenience, time, etc.

Sure thing. Also added in some similar products that I thought might be comparable, although it's really tough to find commercial products that I'd actually consider comparable.

Server Build - Note that price reflects what I paid at purchase, not current prices

• Silverstone ST50F-P 500W - $79.99
• SilverStone PP05 Short Cable Set - $19.99
• Optical Drive - $0.00
• Rosewill RC-211 Controller Card - $29.99
• Lian-Li PC-Q08B - $109.99
• PNY Optima 8GB DDR3 RAM - $67.99
• Thermaltake Slim X3 80mm CPU Cooler - $25.99
• Zotac H55ITX-C-E LGA 1156 - $129.99
• Intel Core i3-540 Clarkdale 3.06GHz - $119.99
• 6x Samsung F4 HD204UI 2TB SATA 3.0Gb/s - $479.94
• 40GB Hitachi 2.5" SATA - $15.00

TOTAL BUILD COST - $1,078.86

EDIT: Picked up 6 left-angle SATA cables for ~$15 shipped. Started with a 12" cable for the first drive, then up 2" each drive to 22" for the bottom drive.

Comparable Products

• QNAP TS-659 PRO+ (6 bay) - $1,224.99
• Thecus N7700+ (7 bay) - $999.99
• LACIE 301511U (5 bay) - $779.99
• Drobo S (5 bay) - $709.99
• Drobo FS (8 bay) - $1,799.00
• Thunderbolt based RAID system (6 bay) - $1,999 w/ 2TB drives

 

[RaiderJ]

Updates! Re-did the power cable by removing six plugs off two cables, and then cutting the cable down to just the length I needed. The plugs themselves were the punchdown type, so I just stuck them in the hard drives, ran the cable across each one, and crimped them with a small screwdriver. Eventually I'll redo the wire with just blue and black so it matches the color scheme.

Another shot of the cable itself:

Still on the lookout for some left angled SATA data cables, but that can wait.

 

[pitz]

That SATA power cable thing you did -- very cool.

Too bad you weren't able to find a Sandy Bridge board though, simply for the energy savings.

You probably could have booted the system from USB which would have saved you the un-reliability of an additional 2.5" SATA hard drive (nevermind having to provision a port or power for it!).



Of course, most of those prices you quote for 'comparable products' do not come with drives. So add another $500, or subtract $500 from your total.

What wireless card came on that board? You might be able to set that up as a WAP and further enhance your environment. Using, for instance, hostapd.

 

[RaiderJ]

That SATA power cable thing you did -- very cool.

Thanks!

Too bad you weren't able to find a Sandy Bridge board though, simply for the energy savings.

I actually bought the board back in January, but was on business travel the last 5 months and couldn't finish the build then. Not sure if there's a Sandy Bridge mini-ITX board out there with 6 SATA ports, but if so, it could be better than the one I used.

You probably could have booted the system from USB which would have saved you the un-reliability of an additional 2.5" SATA hard drive (nevermind having to provision a port or power for it!).

Unfortunately, I can't get OpenIndiana, Nexenta, or Solaris Express to boot with Legacy USB enabled in the BIOS. Sticks me with having to use a SATA disk for the OS. Hopefully this will be fixed in the future.

Of course, most of those prices you quote for 'comparable products' do not come with drives. So add another $500, or subtract $500 from your total.

Very true - if you don't mind the headache of configuring ZFS, or don't want to run ZFS, then this system is a far better deal IMO.

What wireless card came on that board? You might be able to set that up as a WAP and further enhance your environment. Using, for instance, hostapd.

Just the motherboard's wireless, not sure what exact chipset. It's a mini-PCIe module, so it could always be replaced. Whatever it is, it worked under OpenIndiana for wireless access.