Build Log: 12TB ZFS File Server (6x2TB)

Discussion in 'Memory and Storage' started by RaiderJ, Jun 21, 2011.

  1. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    EDIT: See the final build at post [post=32022556]#35[/post]!

    I've been waiting to update my current file server, an Atom based system with 4x750GB drives in a RAID 5. Gives me 1.99TB of usable space running on WinXP Pro. Since I'm low on space, time to upgrade!

    My usage pattern for the file server is movie storage/playback and backups of my other machines. Nothing intensive, as my HTPC handles playback and streams data from the file server over Ethernet.

    Initially, I was looking to just update the drives to 2TB. However, I had been reading about ZFS and felt that was an excellent base for a file server. The old file server hardware just wouldn't cut it for ZFS - only the hardware RAID controller allowed the Atom to function as it was.


    Component Selection - See post [post=31907561]#22[/post] for build/cost list
    So, off to figure out what I wanted in a new file server. Originally I had wanted to keep the same case - it's a really compact and functional system. Four hot-swap bays and room for an additional 2.5" drive beneath the optical drive.

    Motherboard
    For a new motherboard, I looked around for a mini-ITX motherboard that had ideally six SATA ports and was compatible with Solaris (or a ZFS-supporting fork). That narrowed things down to an Intel board that would support 8GB+ of RAM and a decent processor to handle the ZFS workload. With that in mind, I found one on Newegg that fit the bill:

    The motherboard had everything I needed! Supporting 16GB of memory would have been preferable, but not critical at this point in time. I imagine by the time I need to have more RAM I'll be due for another hardware upgrade. I don't plan to use any of the fancier ZFS features (encryption, deduplication, or tiered SSD caching), so I don't see any performance issues.

    CPU
    For the CPU, I just picked up a cheap dual-core i3-540. TDP is 73W, so it shouldn't run too hot. I'm not quite sure how to gauge CPU performance requirements in ZFS, but it's something I'll investigate once the build is complete.

    RAM, OS Drive, Optical, HSF
    Nothing exciting here, just 2x4GB sticks of DDR3, a $15 40GB 2.5" SATA drive from MicroCenter, a DVD burner I had laying around, and a slimline CPU cooler. The OS drive and optical drive weren't part of my initial parts list, as I was going to re-use the existing ones in my old file server. Those I picked up once I decided to go with a new case.

    Server Case
    I had previously been unable to find a case I liked better than my old file server, but I saw this case and decided it would be perfect.
    So, now I had a way to run 6 data drives, and still have an OS drive and an optical drive (which I rarely use). But, my motherboard only supports 6 SATA ports. To add a couple SATA ports, I picked up a cheap controller that is compatible with ZFS . I won't get any great speeds out of the controller, but just for an optical and OS drive it would be fine.

    Power Supply
    This was a tricky one - the case only supports shorter power supplies, so most modular ones are out. As cramped as the case was going to get, I felt sure that modular was the best way to go. I looked around for awhile, hoping to find a short modular one, and sure enough, SilverStone makes one!
    I also picked up a set of shorter cables to go with the PSU, no sense in having extra cable in a case that doesn't need it. I might end up custom making some SATA power cables once I get everything put together. EDIT: [post=31912258]Done![/post]

    Data Drives
    I initially picked up 4x2TB drives, the Samsung F4 EcoGreen drives. They would have been a perfect replacement for my older 750GB drives, but with ZFS they present a problem: they are 4K "Advanced Format" sector drives with 512k sector emulation. I'm still researching how best to handle this, but from what I understand so far it's more of a performance problem than reliability. I'm confident there will be a decent enough workaround in due time. So for now, I just picked up a couple more F4 drives to round out the build.

    Server Build
    Right now I'm just waiting on the case and controller to arrive (due today via UPS). I've run full sector scans on all the hard drives to make sure they're 100%, a memtest on the RAM, and installed OpenIndiana (build 148) on the 40GB drive.

    Only quirk so far is that to boot the LiveCD, I had to disable legacy USB in the BIOS. Not quite sure why that was needed, and only after some googling did I find that to be the solution. Before that, the install would hang before even fully booting up. I think that I should be able to plug in the OS drive to the controller on the motherboard and be able to continue where I left off. If not, it's pretty quick to re-install.

    I'm not yet sure how I want to arrange my drives, as the whole 512k/4K presents some issues with how data is striped across the array. According to this post, a 6 drive RAIDZ2 is preferable to a 6 drive RAIDZ1. I would rather lose a single drive in the array to parity data, but a RAIDZ2 would be better for uptime and give me a breather if a drive crashes.

    Next Steps
    Once I get in the case and controller, I'll post pictures of the build and a log of how I go about configuring the system!
     
    #1 RaiderJ, Jun 21, 2011
    Last edited: Jul 21, 2011
  2. Yellowbeard

    Yellowbeard Golden Member

    Joined:
    Sep 9, 2003
    Messages:
    1,542
    Likes Received:
    0
    Cool build,sub'd.

    I just redid my FreeNAS rig. I am a complete NOOB to ZFS but my opinion is that you made a good call on the 8GB of RAM. I have 16GB in mine. While doing a simple file transfer from the old server to the new server while also loading files from my main PC to the new FreeNAS/ZFS rig, I have seen the memory usage go over 12GB according to the system monitor in FreeNAS.

    As cheap as they are, I have considered grabbing a couple more of those Hitachi 2TB drives and moving to a RAIDZ2 setup for peace of mind.
     
  3. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    First Day Build Notes
    Finally received the case and SATA controller and started the build. Everything worked fine until I plugged in the controller card - the system wouldn't boot. Figured out that there was a new BIOS for my board that would fix this.

    So, with that in place, I could now boot the system with both the 40GB OS drive and optical drive connected. However, the drives were only seen if I set the jumper on the controller to enable the BIOS. I know that ZFS prefers to see bare drives connected, no RAID. Since the BIOS on the controller was flashed with the RAID BIOS, I reflashed over the latest SIL3132 non-RAID BIOS. Easy enough from a bootable flash drive into FreeDOS.

    Now the drives were seen by the motherboard BIOS as IDE drives, bootable, and no RAID BIOS to potentially cause problems. With the RAID BIOS I had found I was unable to fully boot off the OpenIndiana LiveCD. This was also the case with the non-RAID BIOS - I'm not exactly sure why. My workaround was to connect the optical drive to the motherboard SATA port, boot the LiveCD, and (re)install OpenIndiana to the OS drive connected to the controller. No issues there.

    Once OpenIndiana was installed, I found that I still couldn't boot into the OS with "Legacy USB" enabled/auto in the motherboard BIOS. So, I'm still stuck having that disabled. Once into the OS, OpenIndiana has no problem seeing USB drives - but it's still not ideal since if I want to boot from a USB stick I have to change the BIOS setting first. Not sure how to fix this at the moment.

    Will update with pictures of the case with all the gear installed soon! It's actually a pretty clean install - the modular PSU is a huge help.

    EDIT: Picture of the build! Don't have the data drives connected just yet - will work that next.
    [​IMG]
     
    #3 RaiderJ, Jun 22, 2011
    Last edited: Jun 22, 2011
  4. Silenus

    Silenus Senior member

    Joined:
    Mar 11, 2008
    Messages:
    359
    Likes Received:
    0
    First of all congrats! Nice little ZFS build.

    Good call on putting the OS and DVD on the controller card. They don't need to be fast. Having the data drives all on the motherboard will be better and will enable you to work around the 4K thing pretty easily. I have the same Samsung F4's (HD204UI) and run 9 of them in a RAIDZ2 + spare in our NAS here at work. Here is a relatively simple way to get them aligned:

    1) Download ZFSguru stable LIVE CD ver 0.1.7 (http://zfsguru.com/download)
    2) Boot to ZFS guru and then format each of your 6 drives drives from the web gui utility. NOTE: Format the drives with GEOM...NOT GPT (Solaris won't recognize GPT partitions).
    3) Create your RAIDz2 pool in the web gui and select the 4K override option!
    4) Export the pool (do this locally from command line. "zpool export poolname")
    5) Now boot back up to your OI system and import the pool. It should find it and it will be correctly aligned for the drives.


    Alternately...don't worry about the 4K issue. It IS only a performance difference, not a reliability one. Chances are quite good that you will still easily be able to saturate gigabit even with the slightly reduced performance of the drives being non-aligned.
     
  5. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Thanks for the advice! I'll give that utility a spin later today. I figured once I get the data drive put together I can leave them mostly alone - the OS issues with OpenIndiana won't affect the pool once built.

    Definitely not concerned about performance - my existing server runs at 33MB/s max over the PCI controller. Good enough for movies!

    Quick question: What OS are you running and why?
     
  6. Silenus

    Silenus Senior member

    Joined:
    Mar 11, 2008
    Messages:
    359
    Likes Received:
    0
    I am running Nexenta community edition on my NAS. It was the first one I spent a lot of time with and got used to, though I have also experimented with OI and SE 11 (both with Napp-It). Although it is not the simplest design I like the built in web gui that comes with Nexenta. It is easy to see the status of all disks, pool, setup shares or iscsi targets. It is also nice to be able to see at glance the current CPU, network, and disk activity from the dashboard. One of the downsides however is Nexenta slightly awkward non-standard command structure when doing something from command line. It doesn't always recognize the standard ZFS commands and uses slightly different names for things. That is one of the advantages of the the pure Solaris based ZFS systems. They do have their advantages. One nice new addition is a real zfs replication feature Gea is adding to napp-it (it is going to be a paid plug, but way cheaper than nexenta licensing). Lastly, OI and SE are built on newer ZFS versions and developing faster than Nexenta so if there are advanced features you want to use (like encryption), you'll need one of them .
     
    #6 Silenus, Jun 23, 2011
    Last edited: Jun 23, 2011
  7. ViviTheMage

    ViviTheMage Lifer

    Joined:
    Dec 12, 2002
    Messages:
    35,847
    Likes Received:
    1
    Neat little build. what controller card did you get?
     
  8. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Just a cheapo PCIe x1 from Newegg that is supported under Solaris. Wouldn't stick data drives on it, but for OS and optical it's fine (since they won't be bottlenecked by the x1 connection).
    [​IMG]
     
    #8 RaiderJ, Jun 23, 2011
    Last edited: Jun 25, 2011
  9. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Day Two - Data Drives & OpenIndiana & napp-it

    First off, a picture of the system with the power and data cables connected.

    [​IMG]

    I haven't finalized the cable layout - going to get some 90 degree data cables and route them back through the drive cage. For the power cable, I'm looking for a place where I can buy the cable and SATA adapters myself. The current power cables are way too bulky, although everything does fit in the case just fine.

    OS
    My current install of OpenIndiana is up and running, and for simplicity, I decided to just test out napp-it. It's just a single command from the terminal to get it installed, then a bit of configuring. I basically just followed the napp-it guide and truly had my pool and RAIDz2 up and running in a few minutes, including SMB shares.

    It took me a bit to figure out how to enable TLS for Gmail alerts/status reports, but the readme on the napp-it GUI under TLS has all the terminal commands to use.

    So, truly a quick and easy setup! Also ran some benchmarks. I've bolded the tests that napp-it had bolded in the results - from what I've read it's the sequential disk performance that's the most important.

    Test - Speed - %CPU

    • Seq-Wr-Chr - 78 MB/s - 98
    • Seq-Write - 198 MB/s - 27
    • Seq-Rewr - 113 MB/s - 17
    • Seq-Rd-Chr - 89 MB/s - 99
    • Seq-Read - 309 MB/s - 13
    • Rnd Seeks - 481.3/s - 1

    Currently I have only 2 of my 2TB drives in the new server, the remaining 4 are 750GB drives from the old server. What I want to do is test the online RAID expansion by swapping in the new 2TB drives and see if at the end I get my 8TB of storage.
     
  10. Silenus

    Silenus Senior member

    Joined:
    Mar 11, 2008
    Messages:
    359
    Likes Received:
    0
    Very nice! Your sequentials may even go up a bit when the other 2TB drives are in. But it doesn't matter since you are way past gigabit line speed anyway. :) Wheres the 10Gb Ethernet for desktops already?! Oh and cheap switches... ;)
     
  11. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    I can't imagine speeds won't increase as I swap in the 2TB disks and then redo the pool using 4K alignment. Even so, it's pretty good speeds with 3 different types of drives (Samsung 2TB, WD 750GB, Seagate 750GB), and two sector types.
     
  12. Gigantopithecus

    Gigantopithecus Diamond Member

    Joined:
    Dec 14, 2004
    Messages:
    7,665
    Likes Received:
    0
    Looks great! Looks like it's performing well, too. Quick question: how's that slim Thermaltake cooler working for you? Looks like a good, inexpensive step up from the stock cooler I'm using in my Q08 build right now.
     
  13. fffblackmage

    fffblackmage Platinum Member

    Joined:
    Dec 28, 2007
    Messages:
    2,548
    Likes Received:
    0
    I'm not sure if the PCIe x1 would even bottleneck anything. PCI 1.0/1.1 provides 250MBps.

    Do you know if the devices you've attached to the SATA/RAID controller are running in AHCI mode or IDE mode? I have a couple of those controller cards myself, but the HDDs I've attached to them are listed as IDE devices (I'm using FreeBSD 8.2). So far, I haven't been able to figure out a way to have them run in AHCI mode.

    BTW, you picked out a really nice little case (with room for 6 HDDs!). In comparison, my own NAS is inside a monolithic P183.
     
  14. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Seems to work great. Haven't really had the system up enough to fully test temperatures during load, but it's a quiet cooler when I've had the machine on so far. There's plenty of room between the heatsink and the PSU, so you could fit a larger cooler I'm sure. Originally I was going to put the system in my old case, so I didn't really need the slim cooler.
     
  15. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    The controller chip itself actually has a limitation, per this website: http://blog.zorinaq.com/?e=10

    But, the limitation isn't a problem for an optical and OS drive. Just wouldn't be good for data drives. Both drives run in IDE mode, I'm not sure if there is any way to get them to run in AHCI mode. I don't see it as a particular issue, you're not going to be needing NCQ or hot swap on the drives attached to that controller.
     
  16. fffblackmage

    fffblackmage Platinum Member

    Joined:
    Dec 28, 2007
    Messages:
    2,548
    Likes Received:
    0
    I just thought it was weird that the controller advertises support for how-swapping and NCQ, which isn't supported by IDE. Or is it?
     
  17. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    I believe that is the case, but only when using the RAID BIOS (controller supports NCQ & hot-swapping). I flashed over the non-RAID BIOS, hoping that it would fix my non-booting optical drive (which it didn't). At least this way the motherboard BIOS can see the drives, and it's my understanding ZFS works best with non-RAID BIOS.

    EDIT: ZFSGuru boots fine from the optical drive on the controller. Other discs might boot too.
     
    #17 RaiderJ, Jun 24, 2011
    Last edited: Jun 25, 2011
  18. Emulex

    Emulex Diamond Member

    Joined:
    Jan 28, 2001
    Messages:
    9,759
    Likes Received:
    0
    no kidding i have 4 lsi cards with 8 fan-out cables for like $50 each - zfs love :) buy me :) 4 cards on 4 slots would be alot of bandwidth
     
  19. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Wow, what setup do you have? How many drives are you running?
     
  20. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    I'm going through this process now - have a question you might be able to answer:

    Does the version that ZFSGuru uses to create the RAIDz array allow for auto-expansion? I want to test that feature, so right now I have 5x2TB + 1x750GB (going to swap in the last 2TB). Also, should I do a ZFS upgrade from napp-it?


    Also ran some benchmarks from this tool on the array - one using the drives in a RAIDz2, and another in a RAIDz1:

    4K Sector Size Override
    RAIDz2: 366.3 MB/s Read; 249.9 MB/s Write
    RAIDz1: 412.5 MB/s Read; 289.2 MB/s Write

    512Kb Sector Size
    RAIDz2: 370.3 MB/s Read; 244.3 MB/s Write
    RAIDz1: 393.3 MB/s Read; 280.4 MB/s Write

    Odd, I'd have figured at least my write speed would have gone down on the z1, due to the 128Kb stripe size being split across 5 disks. Maybe that performance hit isn't as much as the performance increase by striping across 5 disks versus 4?


    EDIT: Benchmarks after importing the new RAIDz1 pool with napp-it. Quite a bit faster, due to the RAIDz1 and in part since I'm running 5x2TB disks versus the 2 before. Curious to see what replacing the final 750GB disk does to performance.

    Test - Speed - %CPU (increase versus earlier benchmark)

    • Seq-Wr-Chr - 85 MB/s - 98 (9%)
    • Seq-Write - 309 MB/s - 36 (56%)
    • Seq-Rewr - 180 MB/s - 26 (59%)
    • Seq-Rd-Chr - 93 MB/s - 99 (4%)
    • Seq-Read - 430 MB/s - 19 (39%)
    • Rnd Seeks - 629.2/s - 1 (31%)
     
    #20 RaiderJ, Jun 24, 2011
    Last edited: Jul 21, 2011
  21. sygyzy

    sygyzy Lifer

    Joined:
    Oct 21, 2000
    Messages:
    14,001
    Likes Received:
    1
    Wow awesome bill. Would you mind posting a build and price list? I am curious how this compares to buying say a QNAP or Synology 4-6 bay in terms of cost, convenience, time, etc.
     
  22. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Sure thing. Also added in some similar products that I thought might be comparable, although it's really tough to find commercial products that I'd actually consider comparable.

     
    #22 RaiderJ, Jun 25, 2011
    Last edited: Jul 21, 2011
  23. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Updates! Re-did the power cable by removing six plugs off two cables, and then cutting the cable down to just the length I needed. The plugs themselves were the punchdown type, so I just stuck them in the hard drives, ran the cable across each one, and crimped them with a small screwdriver. Eventually I'll redo the wire with just blue and black so it matches the color scheme.

    [​IMG]

    Another shot of the cable itself:

    [​IMG]

    Still on the lookout for some left angled SATA data cables, but that can wait.
     
  24. pitz

    pitz Senior member

    Joined:
    Feb 11, 2010
    Messages:
    461
    Likes Received:
    0
    That SATA power cable thing you did -- very cool.

    Too bad you weren't able to find a Sandy Bridge board though, simply for the energy savings.

    You probably could have booted the system from USB which would have saved you the un-reliability of an additional 2.5" SATA hard drive (nevermind having to provision a port or power for it!).



    Of course, most of those prices you quote for 'comparable products' do not come with drives. So add another $500, or subtract $500 from your total.

    What wireless card came on that board? You might be able to set that up as a WAP and further enhance your environment. Using, for instance, hostapd.
     
  25. RaiderJ

    RaiderJ Diamond Member

    Joined:
    Apr 29, 2001
    Messages:
    7,580
    Likes Received:
    0
    Thanks!

    I actually bought the board back in January, but was on business travel the last 5 months and couldn't finish the build then. Not sure if there's a Sandy Bridge mini-ITX board out there with 6 SATA ports, but if so, it could be better than the one I used.

    Unfortunately, I can't get OpenIndiana, Nexenta, or Solaris Express to boot with Legacy USB enabled in the BIOS. Sticks me with having to use a SATA disk for the OS. Hopefully this will be fixed in the future.

    Very true - if you don't mind the headache of configuring ZFS, or don't want to run ZFS, then this system is a far better deal IMO.

    Just the motherboard's wireless, not sure what exact chipset. It's a mini-PCIe module, so it could always be replaced. Whatever it is, it worked under OpenIndiana for wireless access.
     
Loading...