30TB FreeNAS Hardware List & Questions

[imported_MrBungle]

Hi everyone - this is my first NAS build and I'm hoping to get some feedback on my hardware choices.

Here's some background info about my usage requirements and patterns:

- My main goal is to consolidate my file storage and provide some redundancy for peace-of-mind. I currently have 9 1.5TB disks in a JBOD configuration (scary, I know), 2 2TB disks in a USB dual external drive, and 2 1.5TB individual 1.5TB USB external drives. My plan is to reuse the 9 1.5TB disks for a separate project, and the rest I can either use for partial backup of my NAS, sell, or use for something else.
- I've bought 8 3TB drives for this build, and will be ordering 2 more (30TB total).
- I have ~2TB of widely assorted files that I *really* don't want to lose (music, photos, work-related files, personal files, etc.).
- I have about 16TB (and gradually growing) of video, almost entirely in files of ~3-50GB, which I can reconstruct if absolutely needed - but that would still be a huge time sink so I'm inclined to keep it as safe as I can. I'm not ready to spend the money necessary for a true backup solution, so I'm hoping that RAID-Z2 is sufficient there. I'm aware that RAID != backup, and I think I've gotten a good sense of the tradeoffs and possible pitfalls - but I'm willing to accept a bit of risk with these files to save a good amount of money.
- In most cases, losing an occasional file or two due to corruption, bit rot, etc., wouldn't be the end of the world, but I'd like to try to avoid it, or at least become promptly aware of it.
- The most frequent usage will be Time Machine backups from two Macbooks and a Mac Pro (approx. hourly during the daytime), followed by watching video on a networked HTPC.
- It will usually be accessed by one, but maximum two simultaneous users on a LAN.
- Internet access is a plus, but not required.
- Downtime isn't a big concern (even a couple days is fine).
- I'm hoping to keep power usage very low. The system will be idle much/most of the time, so I'm expecting to rely on Intel or AMD's latest power-saving technologies (e.g., Intel SpeedStep or AMD's EPU) for this.

My plan is to build a ZFS/RAID-Z6 based FreeNAS box, but I'm open to any other suggestions or ideas. I'm on the fence about whether to give special treatment to the ~2TB of more crucial files - i.e., create a separate RAID array for those or the like. I'd appreciate any opinions there.

Considering all of that, I've done a couple weekends'-worth of research and have come up with a lot of questions based on a FreeNAS build:

- Do I need to worry about support for my 3TB Hitachi drives? If I understand correctly, modern 64-bit OSs should support them out of the box as storage drives, and the compatibility issues mainly arise when they're used as boot drives; is that correct? The HBA that I've chosen (HighPoint Rocket 620) was actually bundled with the earliest 3TB drives on the market, so it should have solid support, I think.
- A commenter on another thread suggested that the 512KB sector size of the Hitachi 3TB drives could cause performance issues. Any idea what he is referring to there?
- Is ECC memory advisable (i.e., worth the $50-100 total premium)? I've heard mixed opinions, but the more legitimate sources seem to say it is. I assume registered/buffered is not that important?
- Would I be better off with an Intel-based server board with ECC memory (like http://www.newegg.com/Product/Produc...82E16813121526), or is an Asus AMD desktop motherboard sufficient? (The latter supports ECC memory.) The main advantage to the Intel that I can see is onboard dual LAN ports, but are there others I'm not aware of? I like the Asus route because they support nice consumer-grade features like dynamic power efficiency, onboard video (for cheap/easy initial setting up and BIOS tweaking), USB 3.0, etc. However, will I be able to use all these features running FreeNAS (primarily the EPU power-saving feature)?
- Is AMD a first-class citizen in the FreeBSD/FreeNAS world, or is Intel a safer bet in terms of stability and driver availability?
- What kind of CPU do I need to ensure that it won't be a performance bottleneck? (Please see my build list below for my best guess.)
- Is 8GB RAM sufficient? Does the need for RAM capacity increase with more storage space?
- Is the commonly-used Realtek 8111E LAN Chipset enough to see the ~100MB/s ideal local network speed, or will I need a good Intel NIC? Are there any other typical bottlenecks to worry about?
- Is Cat6 cable length much of an issue in terms of performance degradation? My longest stretch will be about 30-40 feet.
- I found a write-up (http://lime-technology.com/home/87-for-system-builders) that highly recommends a single-rail PSU, but that wasn't until I'd already bought a dual-rail PSU (Antec EarthWatts EA-380D). Is this likely to be a problem?
- Is there anything else I haven't considered, or may be missing?

Based on my research, current level of understanding, and inclinations, I've put together this build list:

ALREADY OWN:

Antec Three Hundred Illusion Black Steel ATX Mid Tower Computer Case
http://www.newegg.com/Product/Produc...82E16811129066

Antec EarthWatts Green EA-380D Green 380W Continuous power ATX12V v2.3 / EPS12V 80 PLUS BRONZE Certified Active PFC Power Supply
http://www.newegg.com/Product/Produc...82E16817371033

(8x) HITACHI Deskstar 0S03230 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive
http://www.newegg.com/Product/Produc...82E16822145493

READY TO BUY:

ASUS M4A89GTD PRO/USB3 AM3 AMD 890GX HDMI SATA 6Gb/s USB 3.0 ATX AMD Motherboard
http://www.newegg.com/Product/Produc...82E16813131631

AMD Athlon II X2 250 Regor 3.0GHz 2 x 1MB L2 Cache Socket AM3 65W Dual-Core Desktop Processor ADX250OCGMBOX
http://www.newegg.com/Product/Produc...82E16819103903

Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) Server Memory Model KVR1333D3E9SK2/8G
http://www.newegg.com/Product/Produc...82E16820139262

HighPoint Rocket 620-OEM PCI-Express 2.0 x1 Low Profile SATA III (6.0Gb/s) Controller Card - OEM
http://www.newegg.com/Product/Produc...82E16816115099

(2x) HITACHI Deskstar 0S03230 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive
SATA cables
PATA/CompactFlash adapter and CF card, or USB thumbdrive and internal USB adapter
fan adapter cable(s)
power adapter cable(s) for the extra SATA drives
PSU cable (Antec PSU didn't come with one)
Cat6 cables

How does that look? I'd appreciate any and all feedback - please let me know if I missed anything, or need to provide any more information. Thanks in advance!

 

[Puppies04]

No matter what you end up doing make sure that 2tb of files that you *really* dont want to lose are backed up to one or more of your external drives and not kept at the same property as your server. What you are planning on building has redundancy but a fire/burglary etc would still leave you with jack.

 

[mfenn]

What was so wrong with your other thread? It'd be better to just update that so that people will have the full context.

 

[Spikesoldier]

make sure your rocket 620 adapters are not the 620WDA. the WDA model's firmware will only allow WD branded 3TB drives to be used.

with that said, im having no compatibility issues with the rocket 620 WDA under freenas/freebsd with my samsung 1TB 103SJ's.

edit: saw the link you had for the 620. be aware you have selected the low profile version.

 

[mfenn]

To specifically answer your questions:

- Do I need to worry about support for my 3TB Hitachi drives? If I understand correctly, modern 64-bit OSs should support them out of the box as storage drives, and the compatibility issues mainly arise when they're used as boot drives; is that correct? The HBA that I've chosen (HighPoint Rocket 620) was actually bundled with the earliest 3TB drives on the market, so it should have solid support, I think.

As Spike noted, make sure there are the 620 and not the WD locked version. Since they have a FreeBSD driver, you should be OK.

- A commenter on another thread suggested that the 512KB sector size of the Hitachi 3TB drives could cause performance issues. Any idea what he is referring to there?

I'm not aware of any drives with 512KB sectors. The Hitachi drives have 4KB sectors. You will want to make sure that your file system blocks are aligned to the underlying HDD sectors. This is easy enough if you give a small amount of padding at the beginning of the drive.

- Is ECC memory advisable (i.e., worth the $50-100 total premium)? I've heard mixed opinions, but the more legitimate sources seem to say it is. I assume registered/buffered is not that important?

With the small amount of memory we are talking about here and the fact that uptime isn't necessary, ECC is probably not worth it. ZFS does plenty of redundancy checks on its own.

- Would I be better off with an Intel-based server board with ECC memory (like http://www.newegg.com/Product/Produc...82E16813121526), or is an Asus AMD desktop motherboard sufficient? (The latter supports ECC memory.) The main advantage to the Intel that I can see is onboard dual LAN ports, but are there others I'm not aware of? I like the Asus route because they support nice consumer-grade features like dynamic power efficiency, onboard video (for cheap/easy initial setting up and BIOS tweaking), USB 3.0, etc. However, will I be able to use all these features running FreeNAS (primarily the EPU power-saving feature)?

Probably not worth the cost for the server board IMHO. FreeNAS does use powerd, so it should give you the nice power saving features.

- Is AMD a first-class citizen in the FreeBSD/FreeNAS world, or is Intel a safer bet in terms of stability and driver availability?

AMD, yes. Some of the supporting chipsets, not so much. Though that can be an issue with consumer boards of any stripe. The Realtek 8111E is supported though, and that's the main thing that you care about.

- What kind of CPU do I need to ensure that it won't be a performance bottleneck? (Please see my build list below for my best guess.)

The X2 will probably suffer a bit under heavy random IO, but otherwise the network will be the bottleneck.

- Is 8GB RAM sufficient? Does the need for RAM capacity increase with more storage space?

ZFS loves memory, but 8GB is probably a good starting point.

- Is the commonly-used Realtek 8111E LAN Chipset enough to see the ~100MB/s ideal local network speed, or will I need a good Intel NIC? Are there any other typical bottlenecks to worry about?

I'm betting that you will get about 80-90MB/s out of the Realtek. The Intel server NICs will do better of course, but I dunno if it is worth the money.

- Is Cat6 cable length much of an issue in terms of performance degradation? My longest stretch will be about 30-40 feet.

Not a problem.

- I found a write-up (http://lime-technology.com/home/87-for-system-builders) that highly recommends a single-rail PSU, but that wasn't until I'd already bought a dual-rail PSU (Antec EarthWatts EA-380D). Is this likely to be a problem?

Basically, what he is saying is that many dual 12V rail PSUs will dedicate a bunch of power to the PCIe power connectors, which are fairly useless in this build. You will be pretty heavily crossloading the PSU, but I can't hazard a guess as to whether or not you will send it over the edge.

 

[imported_MrBungle]

No matter what you end up doing make sure that 2tb of files that you *really* dont want to lose are backed up to one or more of your external drives and not kept at the same property as your server. What you are planning on building has redundancy but a fire/burglary etc would still leave you with jack.

Thanks - that sounds like good advice. My initial thought was to use the dual 2TB external drive for this - i.e., make a copy of the 2TB of files from my NAS to each of the two drives so I have 3 copies in total at 2 different locations.

Is there a good way to automate that using FreeNAS, so that it will make smart, incremental backups at regular intervals as the files grow and change?

 

[imported_MrBungle]

What was so wrong with your other thread? It'd be better to just update that so that people will have the full context.

Sorry about that - I should have at least cross-linked the threads to avoid confusion. I figured that my build goals and hardware list had changed significantly enough that most of that thread was obsolete, and it made more sense to start a fresh one.

Also, I'm so wordy that I didn't expect anyone to wade through all of that at once. 😉

 

[imported_MrBungle]

make sure your rocket 620 adapters are not the 620WDA. the WDA model's firmware will only allow WD branded 3TB drives to be used.

with that said, im having no compatibility issues with the rocket 620 WDA under freenas/freebsd with my samsung 1TB 103SJ's.

edit: saw the link you had for the 620. be aware you have selected the low profile version.

Thanks for the heads-up! I think that version should be OK; I just noticed that it has a WD-branded OEM box, but it's not listed as the 620WDA model and some of the comments confirm using it with non-WD drives (including large-capacity 3TB drives). It also appears to come with a full-profile bracket (notice the 4th picture at http://www.newegg.com/Product/Produc...82E16816115099).

 

[imported_MrBungle]

As Spike noted, make sure there are the 620 and not the WD locked version. Since they have a FreeBSD driver, you should be OK.

Cool, thanks. It sounds like they should work out. Are there any other concerns with high-capacity drives to consider, though - e.g., do I need to research whether my motherboard's native SATA ports are compatible, or configure FreeNAS in a specific way?

I'm not aware of any drives with 512KB sectors. The Hitachi drives have 4KB sectors. You will want to make sure that your file system blocks are aligned to the underlying HDD sectors. This is easy enough if you give a small amount of padding at the beginning of the drive.

The Hitachi specifications list a 512KB sector size (http://www.hitachigst.com/internal-drives/desktop/deskstar/deskstar-5k3000). A few drives like the WD20EARS have "advanced format" 4KB sectors, but I'm fairly certain that these Hitachis don't.

If I'm correct, do I need to worry about aligning file system blocks, or be aware of any potential performance issues related to the sector size?

With the small amount of memory we are talking about here and the fact that uptime isn't necessary, ECC is probably not worth it. ZFS does plenty of redundancy checks on its own.

Though I'm far from an expert on the subject, this is the thread that convinced me of ECC's value:

http://forums.freebsd.org/showthread.php?t=14935

My understanding is that while ZFS is designed to detect/correct file system errors, these can be introduced transparently (i.e., unbeknownst to ZFS) if there is a memory error. Also note the comments on this article:

http://blogs.oracle.com/pgdh/entry/zfs_and_raid_i_is

Googling for "ZFS ECC" brought up a bunch of pertinent articles/posts that mostly seemed to agree that ECC is still important with a ZFS file system.

Again, though, I'm no expert - so ECC memory might still be overkill for my type of build. If I don't opt for it, am I at any considerable risk for data corruption?

Probably not worth the cost for the server board IMHO. FreeNAS does use powerd, so it should give you the nice power saving features.

Does powerd control the CPU power usage as well, or just the drives? I think that the Intel and AMD technologies work by actively underclocking the CPU as performance needs diminish (i.e., closer to idle). I just don't know whether those technologies will work while running any OS (like if they're implemented at the firmware level), or if they're only supported via drivers, which may not be supported in FreeNAS.

AMD, yes. Some of the supporting chipsets, not so much. Though that can be an issue with consumer boards of any stripe. The Realtek 8111E is supported though, and that's the main thing that you care about.

I haven't been able to find an up-to-date hardware compatibility list; where's the best place to look for that (if one exists)? My main concerns are that:

- I have 10 SATA ports to use (whether they're on the motherboard, or I have enough PCI-E 2.0 slots to add Rocket 620 HBAs as needed).
- There are as many power-saving features available as possible - ideally to take advantage of the CPU's dynamic underclocking, as well.
- I'm not spending a lot of extra money on features I can't/won't use.

I'm always a fan of going with tried & true approaches, but I don't really know where to look to find recommendations or success stories for certain motherboard models (Googling motherboard model numbers and "FreeNAS" hasn't been very fruitful).

The X2 will probably suffer a bit under heavy random IO, but otherwise the network will be the bottleneck.

It's not too expensive to upgrade, if needed - is more cores or a higher clock rate a better bet?

ZFS loves memory, but 8GB is probably a good starting point.

Cool. Most motherboards I've considered have 4 RAM slots, and I plan to use 4GB sticks, so upgrading to 16GB should be possible down the road if I encounter any bottlenecks.

I'm betting that you will get about 80-90MB/s out of the Realtek. The Intel server NICs will do better of course, but I dunno if it is worth the money.

Sounds good. I'm trying to look at motherboards with an least an extra PCI-E x1 slot in case I need to add an Intel NIC.

Basically, what he is saying is that many dual 12V rail PSUs will dedicate a bunch of power to the PCIe power connectors, which are fairly useless in this build. You will be pretty heavily crossloading the PSU, but I can't hazard a guess as to whether or not you will send it over the edge.

Ah, OK, I think I follow. So in other words, a large portion of the 380W from the PSU will be basically inaccessible?

Thanks again for your help!

 

[mfenn]

Cool, thanks. It sounds like they should work out. Are there any other concerns with high-capacity drives to consider, though - e.g., do I need to research whether my motherboard's native SATA ports are compatible, or configure FreeNAS in a specific way?

The SATA controller itself shouldn't be a factor in this day and age and the FreeNAS supports big volumes just fine (far in excess of 3TB).

The Hitachi specifications list a 512KB sector size (http://www.hitachigst.com/internal-drives/desktop/deskstar/deskstar-5k3000). A few drives like the WD20EARS have "advanced format" 4KB sectors, but I'm fairly certain that these Hitachis don't.

If I'm correct, do I need to worry about aligning file system blocks, or be aware of any potential performance issues related to the sector size?

That's 512B, not KB. No drive has 512KB sectors. So we're both wrong. 😉 The 512-byte sector is the old standard, so you won't have to worry about anything.

Though I'm far from an expert on the subject, this is the thread that convinced me of ECC's value:

http://forums.freebsd.org/showthread.php?t=14935

My understanding is that while ZFS is designed to detect/correct file system errors, these can be introduced transparently (i.e., unbeknownst to ZFS) if there is a memory error. Also note the comments on this article:

http://blogs.oracle.com/pgdh/entry/zfs_and_raid_i_is

Googling for "ZFS ECC" brought up a bunch of pertinent articles/posts that mostly seemed to agree that ECC is still important with a ZFS file system.

Again, though, I'm no expert - so ECC memory might still be overkill for my type of build. If I don't opt for it, am I at any considerable risk for data corruption?

If you're buying an enterprise-grade server, then ECC is part and parcel of the equation. But you're not. With a home file server, only time that data will be resident only in memory (that's the only time that ZFS itself doesn't protect you) is the short time between when it enters the NIC and gets written out to the disk.

The chances of one of those particular bits being flipped within the couple of seconds (max) that the data is resident only in memory is astronomical (like sun will go nova astronomical). You're far more likely to encounter a bit error as it the data goes across the network.

Does powerd control the CPU power usage as well, or just the drives? I think that the Intel and AMD technologies work by actively underclocking the CPU as performance needs diminish (i.e., closer to idle). I just don't know whether those technologies will work while running any OS (like if they're implemented at the firmware level), or if they're only supported via drivers, which may not be supported in FreeNAS.

Powerd will control the CPU and everything else as well. The power savings technologies in the CPU require OS support, but FreeBSD is an enterprise-class UNIX operating system, not somebody's science project. 😉

I haven't been able to find an up-to-date hardware compatibility list; where's the best place to look for that (if one exists)? My main concerns are that:

- I have 10 SATA ports to use (whether they're on the motherboard, or I have enough PCI-E 2.0 slots to add Rocket 620 HBAs as needed).
- There are as many power-saving features available as possible - ideally to take advantage of the CPU's dynamic underclocking, as well.
- I'm not spending a lot of extra money on features I can't/won't use.

I'm always a fan of going with tried & true approaches, but I don't really know where to look to find recommendations or success stories for certain motherboard models (Googling motherboard model numbers and "FreeNAS" hasn't been very fruitful).

The authoritative source would be the FreeBSD kernel mailing list, but that can be hard to wade though. FreeNAS plus a particular mobo is too specific of a search to find many hits. You need to search for FreeBSD (the underlying OS that FreeNAS is based on) and the particular chipset you're interested it. For example, the first hit on "freebsd rtl8111e" is this thread. Reading through the thread, you will see that it is supported.

It's not too expensive to upgrade, if needed - is more cores or a higher clock rate a better bet?

Higher clock rate is always easier to utilize than more cores. ZFS is multithreaded in the sense that it handles requests from many different sources very well, but servicing each request is single-threaded. (Much like a web server).

Cool. Most motherboards I've considered have 4 RAM slots, and I plan to use 4GB sticks, so upgrading to 16GB should be possible down the road if I encounter any bottlenecks.

Sounds good. I'm trying to look at motherboards with an least an extra PCI-E x1 slot in case I need to add an Intel NIC.

Both sound good.

Ah, OK, I think I follow. So in other words, a large portion of the 380W from the PSU will be basically inaccessible?

That's correct.

Thanks again for your help!

No prob. 🙂