Building ZFS NAS for Home Use: Mini-ITX, 6 SATA, low-power?

[RaiderJ]

Looking to upgrade my home storage server - posting a bit of a worklog here, although I'm still in the component selection stage. Would be curious to get thoughts and feedback, especially if you have any experience with ZFS!

*****************

My current storage server is getting a bit long in the tooth and needs an upgrade from the 4x750GB drives it currently runs in a RAID 5. While I could just swap in some 2GB drives, I'd like to also switch to a ZFS software storage solution. The data integrity checks, de-duplication, and storage configuration flexibility would make for a robust storage platform.

I currently have an Atom board an Intel PCI RAID controller running WinXP. 4 drives for storage on the controller, and a single laptop drive for the OS. For a ZFS build, I'd use 4 drives for storage, and 1 or 2 drives for the OS/log. I may look at expanding the number of drives for storage, but the case I'm using has 4 hot-swap bays, and I'd like to stick with it because of it's form factor (http://www.directdial.com/ES34069-BK-180.html). Will have to stay with a mini-ITX board for this same reason.

So, planning to run 4x2TB of drives in a RAID-Z under ZFS, I don't think an Atom board with 2GB RAM will cut it. I'm sure I'd run into performance bottlenecks with the CPU and RAM at some point, so it makes sense to upgrade there as well. Plus, I'm limited with 100Mbps ethernet and PCI slot - neither of which are very forward looking for a storage server.

Looking at mini-ITX motherboards with 6 SATA ports, I only came up with one option, the ZOTAC H55ITX (http://www.newegg.com/Product/Produc...82E16813500056). There are two versions, one with USB 3.0 and one without. There's also a PCI-e x16 slot, which could be useful for adding in a second ethernet connection or another way to connect more drives down the road (if not over USB 3.0). Didn't find a lot of information on this board and running OpenSolaris, but will keep tracking this. Don't want to run into any critical problems running OpenSolaris.

For the OS side, I'll probably just continue to use a single 2.5" HDD. Switching to an SSD, or using one in combination with a HDD, would be hard to justify from a cost perspective, as I don't think I'd see any practical improvement. However, having two SATA ports should allow me to easily change this in the future if it becomes an issue.

For memory, a couple 4GB sticks should be fine and provide plenty of caching for 8TB of (unformatted) storage. Not sure how high 8GB of RAM will scale, but hopefully to at least 16TB of storage, which would allow me to add in more/bigger drives as needed.

Any Core i3/i5/i7 processor should suffice for the system, so I'd probably just stick with a low-TDP i3. I'll have to investigate if running de-deduplication or other ZFS features across 8+TB would pose a problem. I'd like to try and keep heat to a minimum, so it wouldn't make any sense to get a faster processor than what's needed.


So, that's where I'm at now! Looking to put this together sometime in the next 3 months. I've been paying attention to how the Oracle/Sun merger might factor into the long-term for ZFS, but it seems pretty stable - enough to build a storage platform on for home use.

 

[sxr7171]

I think a server board is the way to go. Also get ECC RAM. If you are doing a new build, I would do it right. Also no matter what Intel NIC. Do not suffer with Realtek.

This is one suggested HW guide:

http://www.nexenta.org/projects/site/wiki/About_suggested_NAS_SAN_Hardware

This is another:

http://www.napp-it.org/hardware/index_en.html

Another:

http://www.nexenta.com/corp/supported-hardware/nexenta-certified-components

 

[RaiderJ]

Agree with you on the ECC memory, if I wasn't set on a mini-ITX platform I'd go that route. For my uses, non-ECC shouldn't be an issue. I'd rely on backups and integrity checks by ZFS to ensure safety of critical data. Most of my data is HD movies, so a random flipped bit wouldn't be an issue.

 

[tomoyo]

You're about the same exact build as I'm planning. My goal is mini-itx, probably 4 hd + small boot ssd, and low power. Since sandy bridge is coming out soon, I may wait until post-sandy bridge to decide on a board. The only major issue of the zotac board is that it's pretty wasteful on power consumption compared to similar boards.
I'll probably go with nexenta as my storage os for zfs since it seems pretty well supported and uses opensolaris/openindiana as a base, which has the best zfs support by far.

Also sxr, your suggestions would be nice in the an unlimited money world, but are there even any server mini-itx boards with ecc? Or anything actually cheap? I don't see ecc or server boards at any reasonable price generally. Supermicro has some atom mini-itx, but they have no ecc and no advantage at all over a good desktop board.

 

[sub.mesa]

Zotac has new AMD 880 Mini-ITX motherboard with 6x SATA on it, would be a nice NAS system and relatively low-power.

For true low power, looking at AMD Fusion is very interesting. The Ontario/Brazos platform cut idle power consumption to about half that of Atom platform. These chips have been 'previewed' already; google some for some info.

SuperMicro has nice boards too. The advantages are good BIOS support, IPMI, dual intel gigabit for link aggregation, low power and excellent support/stability/after-service. When building a ZFS server it might be worthwhile to pick server-class components. If you need the functionality, supermicro offers it at very reasonable cost; their highly integrated motherboards are affordable for home users looking for a powerful ZFS server.

For more modest server i would look at the Zotac AMD 880G solution; downside is that PCIe x1 means little expansion ability and 2 DIMM slots limits your RAM to 8GB DDR3 ECC. The plus side is cheap mobo, cheap cpu, and 'cheap' access to ECC support!

But i'm really looking forward to AMD Brazos platform; looks excellent for low-power ZFS servers.

 

[sxr7171]

You're about the same exact build as I'm planning. My goal is mini-itx, probably 4 hd + small boot ssd, and low power. Since sandy bridge is coming out soon, I may wait until post-sandy bridge to decide on a board. The only major issue of the zotac board is that it's pretty wasteful on power consumption compared to similar boards.
I'll probably go with nexenta as my storage os for zfs since it seems pretty well supported and uses opensolaris/openindiana as a base, which has the best zfs support by far.

Also sxr, your suggestions would be nice in the an unlimited money world, but are there even any server mini-itx boards with ecc? Or anything actually cheap? I don't see ecc or server boards at any reasonable price generally. Supermicro has some atom mini-itx, but they have no ecc and no advantage at all over a good desktop board.

They have Mini-ATX and at a good price. It's not really that expensive. One thing I can say for sure is don't do less than 8GB RAM. I would personally take the money from the boot SSD and put it into a server class MB. I don't think the boot drive matters at all. A fast processor is not required unless you use the compression and deduplicating features AFAIK. My i5 650 barely hits 20% cpu usage at most and only for a few seconds at that. It is also quite efficient at idle. Haven't run any kill-a-watt tests yet, but I believe that its idle power use can rival that of an Atom, and when it needs to work it just processes fast and goes back into idle quickly.

The only reason I am saying this is that I am currently using a build designed for a WHS machine and I feel like I should re-do it for a proper Nexentastor server. No matter what board you get, you need to budget in an Intel NIC. I think it's better to spend the extra 10% up front and not regret it later.

 

[RaiderJ]

Zotac has new AMD 880 Mini-ITX motherboard with 6x SATA on it, would be a nice NAS system and relatively low-power.

For true low power, looking at AMD Fusion is very interesting. The Ontario/Brazos platform cut idle power consumption to about half that of Atom platform. These chips have been 'previewed' already; google some for some info.

SuperMicro has nice boards too. The advantages are good BIOS support, IPMI, dual intel gigabit for link aggregation, low power and excellent support/stability/after-service. When building a ZFS server it might be worthwhile to pick server-class components. If you need the functionality, supermicro offers it at very reasonable cost; their highly integrated motherboards are affordable for home users looking for a powerful ZFS server.

For more modest server i would look at the Zotac AMD 880G solution; downside is that PCIe x1 means little expansion ability and 2 DIMM slots limits your RAM to 8GB DDR3 ECC. The plus side is cheap mobo, cheap cpu, and 'cheap' access to ECC support!

But i'm really looking forward to AMD Brazos platform; looks excellent for low-power ZFS servers.

A Fusion setup would probably be much better from a power perspective, but I wonder if there would be any compatibility issues with OpenSolaris/Indiana.

 

[RaiderJ]

They have Mini-ATX and at a good price. It's not really that expensive. One thing I can say for sure is don't do less than 8GB RAM. I would personally take the money from the boot SSD and put it into a server class MB. I don't think the boot drive matters at all. A fast processor is not required unless you use the compression and deduplicating features AFAIK. My i5 650 barely hits 20% cpu usage at most and only for a few seconds at that. It is also quite efficient at idle. Haven't run any kill-a-watt tests yet, but I believe that its idle power use can rival that of an Atom, and when it needs to work it just processes fast and goes back into idle quickly.

The only reason I am saying this is that I am currently using a build designed for a WHS machine and I feel like I should re-do it for a proper Nexentastor server. No matter what board you get, you need to budget in an Intel NIC. I think it's better to spend the extra 10% up front and not regret it later.

Are you seeing any 6x SATA, 8GB RAM, mini-ITX motherboards that support ECC? I haven't found any. There is a 6x SATA Atom board on Supermicro's site, which looks like it would be a great low-power option, if you could live with 4GB RAM. Not sure on price for it, might be more expensive than the ZOTAC boards.

 

[pjkenned]

Are you seeing any 6x SATA, 8GB RAM, mini-ITX motherboards that support ECC? I haven't found any. There is a 6x SATA Atom board on Supermicro's site, which looks like it would be a great low-power option, if you could live with 4GB RAM. Not sure on price for it, might be more expensive than the ZOTAC boards.

The big thing here is the 4GB limit. Otherwise, great boards. Also, in a non-ECC mITX world H57 Intel mITX board is a good option.

 

[sxr7171]

Are you seeing any 6x SATA, 8GB RAM, mini-ITX motherboards that support ECC? I haven't found any. There is a 6x SATA Atom board on Supermicro's site, which looks like it would be a great low-power option, if you could live with 4GB RAM. Not sure on price for it, might be more expensive than the ZOTAC boards.

Unfortunately the smallest with those boards is M-ATX. I currently use an H57 (ATX) for my Nexentastor, but I keep itching to "do it right". With 4GB I was getting low memory messages with a 9TB array. With 8GB no more messages.

 

[RaiderJ]

The big thing here is the 4GB limit. Otherwise, great boards. Also, in a non-ECC mITX world H57 Intel mITX board is a good option.

Except it just has 4x SATA ports... 5 or 6 I'd say is a minimum for a ZFS storage system. 4 for data, 1 or 2 for the OS.

Shouldn't ZFS theoretically catch most data errors, even if they come from non-ECC memory? I'm not convinced that ECC is really critical for a ZFS server. I did a bit of Googling, and it doesn't seem like the incidence of ECC correctable errors is really that high.

 

[sub.mesa]

Only 4 ports on this new Mini-ITX gigabyte mobo with AMD Zacate CPU:

URL: http://www.xbitlabs.com/news/mainbo...unctionality_for_Fusion_Zacate_Platform.html#

But, with the PCI-express x16 slot, you can put a full-blown HBA in there and add more ports easily. It does not have dual gigabit though. But it should make a cheap and very low idle power NAS platform excellent for low-end ZFS servers. If you want multi-gigabit and multiple HBAs then this platform is not for you.

 

[RebateMonger]

I did a bit of Googling, and it doesn't seem like the incidence of ECC correctable errors is really that high.

There are few "hard" numbers. Intel seems to say that memory errors are lower than they once thought them to be.

The main reason I like ECC isn't that there are never errors. It's that ECC notifies you when there ARE errors. When troubleshooting a problem in a server, for insstance, you can pretty much avoid a day-long memory test because if there was an error, ECC would tell you. That can save a LOT of troubleshooting downtime in a business situation.

 

[RaiderJ]

There are few "hard" numbers. Intel seems to say that memory errors are lower than they once thought them to be.

The main reason I like ECC isn't that there are never errors. It's that ECC notifies you when there ARE errors. When troubleshooting a problem in a server, for insstance, you can pretty much avoid a day-long memory test because if there was an error, ECC would tell you. That can save a LOT of troubleshooting downtime in a business situation.

Probably a moot point, since I haven't found a 6x SATA board that supports ECC - but wouldn't ZFS integrity checks correct memory errors anyway? You might not know if the errors were caused by the memory itself... but I'd think you could them run memtest to figure that out.

Not having ECC may not be ideal, but for a home use storage server it would most likely be fine. For a business or critical environment, you would probably not be concerned about mini-ITX as a form factor.

 

[RebateMonger]

The only file corruption I've seen that was definiitely caused by failing memory was on a Win98 box. Something scrambled much of the file system during a Win98 install. I didn't know what had happened until I sold the memory from that motherboard and the buyer did a Memtest and found a memory module was failing.

That was the first time I'd ever HEARD of Memtest, so I obviously wasn't testing memory before that. Since nine-chip Parity modules disappeared in the mid-90's, there's been no assurance at all that a bad memory module will be easily detected. The BIOS' startup memory tests are next-to-worthless.

When IBM's OS2 was first released, IBM gave me a copy. I had four 4MB ($160 apiece) memory modules that I thought were "perfect". I'd used them in Windows for a couple of years in a main computer. In those days, Windows machines would usually have to be rebooted daily anyway, with no way to know the real cause. Memory modules no longer had a Parity chip.

When I installed OS2, it refused to install, citing a memory failure in the third module. I took it to the vendor, who tested it, and reported that module as defective. I'd never have known except for OS2's testing.

 

[sxr7171]

Only 4 ports on this new Mini-ITX gigabyte mobo with AMD Zacate CPU:

URL: http://www.xbitlabs.com/news/mainbo...unctionality_for_Fusion_Zacate_Platform.html#

But, with the PCI-express x16 slot, you can put a full-blown HBA in there and add more ports easily. It does not have dual gigabit though. But it should make a cheap and very low idle power NAS platform excellent for low-end ZFS servers. If you want multi-gigabit and multiple HBAs then this platform is not for you.

So it should perform no differently from one built from an i5-655 outside of the 1 PCI-E slot limitation? If I drop a 16 port HBA in there it should cover me for years.

 

[sxr7171]

Except it just has 4x SATA ports... 5 or 6 I'd say is a minimum for a ZFS storage system. 4 for data, 1 or 2 for the OS.

Shouldn't ZFS theoretically catch most data errors, even if they come from non-ECC memory? I'm not convinced that ECC is really critical for a ZFS server. I did a bit of Googling, and it doesn't seem like the incidence of ECC correctable errors is really that high.

My reasoning stems from some performance options in Nexenta that claim to increase SMB performance in particular. I forget the name of the optimization but it states that it is advisable to have battery backed up RAM or at least a UPS before enabling. Honestly I don't know where exactly ECC comes into this, but I just developed the desire to do my next build with server grade parts, partly due to the fact that it really doesn't cost an arm and a leg, and partly to just do my first server grade build.

That having been said, my H57 consumer board running Nexenta server has been running like a charm. It doesn't skip a beat. I did put an Intel NIC on it though, and I did it even when it was a WHS build. I've had enough headaches with Realtek NICs on Windows alone. Again that could also be more of a perception issue.

 

[taltamir]

Using my 5x750GB RAID6 type array (raidz2) deduplication made it run out of ram and caused severe thrashing, my performance tanked to single digits, and upgrading the CPU from Athlon64 x2 2ghz to Athlon II x4 3ghz did nothing to improve it.
It simply requires lots and lots of ram, 4GB system ram is just not enough for it, but is enough for everything else you want.

Also I heard reports that dedup is not 100% stable yet (aka, some reported dataloss) so don't use it.

the other features of ZFS though work and are amazing. I had it catch and repair errors, it feels so... right to know that you have end to end checksumming and to actually see it in action catching and repairing errors.

For my OS drive I am using an ancient IDE 160GB drive... because it cost me nothing. I have been tempted to get fancy with it, but really, why waste the money?

 

[sub.mesa]

So it should perform no differently from one built from an i5-655 outside of the 1 PCI-E slot limitation? If I drop a 16 port HBA in there it should cover me for years.

You should compare AMD Ontario/Zacate with Intel Atom platform. It has got a dual core CPU which can do out-of-order execution (Atom is limited to static-order execution), but this still is very low-end CPU. In the best case, it is twice as fast as your usual dualcore Atom; so not that fast at all!

But it does allow DDR3 and ECC usage, using cheap memory sticks. It also gives you PCI-express and gigabit and 4 internal SATA. So it should be a very interesting platform for those interested in low-power ZFS NAS. Also the BIOS will allow overclocking (and undervolting i hope) so you can tweak a little between power consumption and performance.

Most HBAs that i see have only 8 ports though, like the Intel SASUC8i (3Gbps; LSI 1068e) or the new SuperMicro USAS2-L8i/l8e (6Gbps; LSI SAS2008). So that would get you 12 full-bandwidth SATA ports. Say 10 disks and one or two SSDs for additional performance (L2ARC/SLOG).

Note that i recommend buying an SSD with supercapacitor ONLY! That means waiting for either Intel G3 or Sandforce SF2000 offerings.

 

[taltamir]

I forgot to mention this, but IIRC ZFS require 64bit CPU.
So that atom better be the 64bit variety and not the 32bit kind.

 

[sub.mesa]

Using my 5x750GB RAID6 type array (raidz2) deduplication made it run out of ram and caused severe thrashing, my performance tanked to single digits, and upgrading the CPU from Athlon64 x2 2ghz to Athlon II x4 3ghz did nothing to improve it.
It simply requires lots and lots of ram, 4GB system ram is just not enough for it, but is enough for everything else you want.

4GiB is barely enough for ZFS alone, let alone de-dupe. For de-duplication you would need:

8GiB system memory exclusive to ZFS only
2GiB memory for every TiB storage (10TiB = 20GiB extra RAM)
SSD to store de-dupe tables.

So you would need alot more RAM to use de-duplication effectively. My advice: don't touch this feature its advantages are disappointing for home users anyway. Only if you have alot of identical data in special setups, would dedupe really be worth it.

Also I heard reports that dedup is not 100% stable yet (aka, some reported dataloss) so don't use it.

Correct, dedupe automatically means you're using unstable ZFS code. Solaris 10/9 release has ZFS v22 but has V21 (de-dupe) disabled on purpose because it is not deemed stable enough for production usage.

If you want more space, simply add one more disk; cheapest investment and also increases performance instead of lowering it. Much cheaper than de-dupe. ;-)

For my OS drive I am using an ancient IDE 160GB drive... because it cost me nothing. I have been tempted to get fancy with it, but really, why waste the money?

Well why waste power. Old PATA disks use alot of power; twice that of a 2TB 5400rpm drive. The spin-up current is often higher than 30W; so there can be a number of reasons not to use such old disks anymore. Consider them using power equal to $8 or €16 a year, depending on where you live. That is alot of money for a disk you don't really need. Switching it with a USB stick that uses virtually no power at all, would make sense.

I understand that alot of people have older disks they still don't want to throw away. But given their relatively high idle-power-per-gigabyte-storage ratio, meaning they use up alot of power relative to their storage capacity, perhaps they should be retired instead. Still using them in a configuration where they are spinned down most of the time, would also be an alternative.

But your system disk should feel best either on the main pool itself (Root-on-ZFS) or a separate USB stick.

 

[sub.mesa]

I forgot to mention this, but IIRC ZFS require 64bit CPU.
So that atom better be the 64bit variety and not the 32bit kind.

With 'that atom' you mean AMD Zacate? Even the Atom is 64-bit, just not the first generations and not the single core Z510 atom; but the D510 which is commonly used, does support amd64.

You're right that you don't want to run i386 32-bit distribution, particularly due to memory issues. This is unrelated from CPU power required. Simple ZFS setups can perform very well without a very fast CPU, if they have plenty of RAM. You could put two sticks of 4GiB in that Zacate motherboard; totalling 8GiB.

So 8GiB ECC DDR3 SDRAM; excellent for starting with ZFS. You won't have the ability to upgrade beyond that, though; so if you get 2x2GiB like alot of people will do, then you're limited by RAM and no real way to upgrade; which i do not recommend.

 

[taltamir]

With 'that atom' you mean AMD Zacate?

I thought the OP said he is using an atom CPU.

4GiB is barely enough for ZFS alone, let alone de-dupe. For de-duplication you would need:

8GiB system memory exclusive to ZFS only
2GiB memory for every TiB storage (10TiB = 20GiB extra RAM)
SSD to store de-dupe tables.

So you would need alot more RAM to use de-duplication effectively. My advice: don't touch this feature its advantages are disappointing for home users anyway. Only if you have alot of identical data in special setups, would dedupe really be worth it.

Thank you for that info. I was unable to actually find a ram requirement for dedup until you posted this. I will keep in mind those figures for the future.

Say, how does file compression affect performance in ZFS? I recently saw figures for windows file compression, where it increased random and sequential reads a little, decrease sequential writing a lot, and decreases random writes by an order of magnitude... How does ZFS compare?

 

[RaiderJ]

Hmm.. didn't realize de-dup wasn't ready for prime-time, or needed quite that much RAM. Kind of rules it out for a mini-ITX, 4 drive setup, even with 8GB of RAM. It wasn't a critical feature for me, since most of my stored data is movies, where de-dup wouldn't be helpful I'll wager.

Still want to stick to a 6x SATA mini-ITX board, that way I don't have to purchase and fiddle with an extra add-on card. More hardware to manage and keep track of. Now, if I wasn't limited by my case, then moving to a micro-ATX board and having a card for more drive connections would probably make sense.

 

[RaiderJ]

I thought the OP said he is using an atom CPU.

That's my current setup, and it's worked great for a file server setup. But, I want to switch to a ZFS system, and my current Atom board (first generation) just won't work. Want to keep my same case, and have a build that will support 4x data drives, and 1|2 for the OS.