Hardware and software questions about a home server/zfs-pool

[Essence_of_War]

I have some general hardware/software questions about building a home use back-up/archive and file server.

Basically, my GF and I have some data. She's a videographer, I do science, we have a modest media collection. I'd like to put together a server that can host some of the large data sets that I work with, serve as an archive for her work, host our media collection, all the while helping to protect from silent data corruption and being easy to back-up.

I think what I'm looking for is a linux or freenas server running a ZFS RAIDZ1 or RAIDZ2 pool. Does that sound like the right tool for the job?

A few options I'm trying to evaluate:
1) Ubuntu Server or FreeNas? They can both be attached to a ZFS pool, right? On the plus side, I know FreeNas can probably fit on a USB stick so that's one less drive/necessary sata-port.
2) RAIDZ1 or RAIDZ2? - I like that RAIDZ2 is still protected from silent corruption even after a drive failure, and I was looking at a case like the Fractal Design Node 304 that has 6 3.5''drive bays, which seems like a reasonable number to transition from Z1 to Z2.
3) We have about 1.5 TB of data now, assuming we digitally hoard more data in the future, how easy is it to expand a zfs pool with larger drives in the future?

What sort of core components does a ZFS server want?

I figure a low end pentium (G3220? $49 at MC!), the cheapest mitx mobo that has 6 sata ports (ASUS H87I-PLUS?), a quiet case with enough drive bays (like the node 304). What about RAM? What about PSU? This system shouldn't draw much power, even in prime95 type loads, would a pico psu be a good choice, or is it fine to go with a somewhat over spec'd ATX PSU? On the hard drive front, if this is going to be a software RAIDZ, does it matter if I use 'green' type drives or 'NAS' type drives?

Feel free to provide me with insight!

 

[dave_the_nerd]

Get a motherboard that supports ECC RAM, and buy ECC RAM. I don't bother, but my data isn't important. Yours is.

You can expand a RAID-Z, but not the way you would probably like to. (You can't add another 2TB drive to an existing 4x2TB drive RAID-Z, do a restripe, and have a functioning 5x2TB RAID-Z.)

You can expand the array by replacing the drives (one at a time) with larger capacity drives, but the number of drives you start with will be the number of drives in that ZPool from now until the end of time. Steps are googleable.

If you have a couple hundred bucks extra and are short on free time, you might just get a NetGear ReadyNAS. (They have ECC RAM, at least some models, and support restriping an existing volume onto additional disks.)

If you do still BYO, any case with a sufficient number of drive bays will have space for an ATX or SFF ATX-style PSU. A good quality 200-300w PSU will give you plenty of overhead for hard drives, won't need an external power brick, and will probably be cheaper than a PicoPSU of the same wattage. (Especially if you take into account the power brick.)

 

[smitbret]

I have some general hardware/software questions about building a home use back-up/archive and file server.

Basically, my GF and I have some data. She's a videographer, I do science, we have a modest media collection. I'd like to put together a server that can host some of the large data sets that I work with, serve as an archive for her work, host our media collection, all the while helping to protect from silent data corruption and being easy to back-up.

I think what I'm looking for is a linux or freenas server running a ZFS RAIDZ1 or RAIDZ2 pool. Does that sound like the right tool for the job?

A few options I'm trying to evaluate:
1) Ubuntu Server or FreeNas? They can both be attached to a ZFS pool, right? On the plus side, I know FreeNas can probably fit on a USB stick so that's one less drive/necessary sata-port.
2) RAIDZ1 or RAIDZ2? - I like that RAIDZ2 is still protected from silent corruption even after a drive failure, and I was looking at a case like the Fractal Design Node 304 that has 6 3.5''drive bays, which seems like a reasonable number to transition from Z1 to Z2.
3) We have about 1.5 TB of data now, assuming we digitally hoard more data in the future, how easy is it to expand a zfs pool with larger drives in the future?

What sort of core components does a ZFS server want?

I figure a low end pentium (G3220? $49 at MC!), the cheapest mitx mobo that has 6 sata ports (ASUS H87I-PLUS?), a quiet case with enough drive bays (like the node 304). What about RAM? What about PSU? This system shouldn't draw much power, even in prime95 type loads, would a pico psu be a good choice, or is it fine to go with a somewhat over spec'd ATX PSU? On the hard drive front, if this is going to be a software RAIDZ, does it matter if I use 'green' type drives or 'NAS' type drives?

Feel free to provide me with insight!

There's nothing wrong with ZFS, but WHS 2011 with this would fit your needs better:

www.flexraid.com

Your other option would be creating new arrays in ZFS when it's time for expansion and just add them to your pool. That would be a pot of parity drives, though.

 

[Essence_of_War]

Get a motherboard that supports ECC RAM, and buy ECC RAM. I don't bother, but my data isn't important. Yours is.

Haswell i3's support ECC RAM, right? So I'd probably want to go with an i3+ a C204 series MoBo if I go that route?

You can expand a RAID-Z, but not the way you would probably like to. (You can't add another 2TB drive to an existing 4x2TB drive RAID-Z, do a restripe, and have a functioning 5x2TB RAID-Z.)

You can expand the array by replacing the drives (one at a time) with larger capacity drives, but the number of drives you start with will be the number of drives in that ZPool from now until the end of time. Steps are googleable.

Understood. That seems reasonable enough with a little bit of planning.

If you have a couple hundred bucks extra and are short on free time, you might just get a NetGear ReadyNAS. (They have ECC RAM, at least some models, and support restriping an existing volume onto additional disks.)

If you do still BYO, any case with a sufficient number of drive bays will have space for an ATX or SFF ATX-style PSU. A good quality 200-300w PSU will give you plenty of overhead for hard drives, won't need an external power brick, and will probably be cheaper than a PicoPSU of the same wattage. (Especially if you take into account the power brick.)

I have enough free/hobby time that I think I'd be inclined towards building it myself, as long as that is somewhat price competitive with pre-built options. I'm not trying to give up a power of ten here, but I assign some value to the intangible of having put it together and set it up myself.

 

[code65536]

Intel's desktop CPUs all support ECC. IMHO, an i3 is overkill here. A G1620 (just $35 at Micro Center!) or G1820 has more than enough computational power, and they both support ECC.

ZFS is RAM-happy, with the general rule of thumb being 1GB of RAM per TB of storage.

I personally wouldn't go down the ECC route, but that all depends on how paranoid you are about the data and how important that data is to you, and how tolerant that data is of single-bit glitches. For really important stuff, eh, "better safe than sorry."

A good clean power source is, IMHO, more important than ECC. I've seen data corruption that occurred as a result of dirty power (which can cause a flux in the drive head and other nastiness). You're right that the power draw should be low, and you'll get more efficiency out of a PSU with a lower wattage. Just make sure that it's reliable and high quality. Seasonic makes a nice 300W 80+ Bronze, if you're going the ATX route. Haven't had any experience with power-brick-based PSUs, though. And needless to say, use a UPS.

And unless you want to make the server multi-purpose (e.g., also serve as a web server or other services), FreeNAS is probably the better way to go.

 

[Essence_of_War]

Intel's desktop CPUs all support ECC. IMHO, an i3 is overkill here. A G1620 (just $35 at Micro Center!) or G1820 has more than enough computational power, and they both support ECC.

The problem isn't the CPU, though, and as I mentioned in the OP, I know that an i-anything is a waste of money. The problem, I think, is finding the right board that also supports ECC, isn't it? I don't think mainstream MoBo's support ECC, and the C204 MoBo I mentioned only supports Xeons and i3's.

 

[mfenn]

FreeNAS makes sense and is easy to use if all you want is a NAS. If you're wanting to make the server do other tasks, then Ubuntu starts looking more attractive.

Regarding expanding a zpool, in addition to cycling drives out one by one, you can also add a new vdev. If you start out with 3 drives in a raidz1 in the Node 304 (for example), you can later expand the pool with another 3-drie raidz1 vdev.

I don't think I would bother with ECC for this machine. The only time your data can be permanently corrupted is between when it arrives at the NIC and when it gets flushed out to disk or during a resilver. The amount of time that any given bit of data is in flight during either of those tasks is small. Sure it can get corrupted while sitting in the ARC, but that would go away upon reboot. Bottom line: ECC is great if you can afford it, but don't feel like it is a must for a home NAS.

 

[code65536]

the C204 MoBo I mentioned only supports Xeons and i3's.

Nope. Take, for example, a C204 board by Super Micro. Their official specs explicitly list Pentium and Celeron alongside Xeon and Core i. I've never heard of a board that supports Core i but not a Pent/Celery of the same socket/generation.

One quirk of ECC is that used ECC is much cheaper than used non-ECC. Not much demand in the ECC aftermarket, it seems. That oughta be enough to offset the premium for the board.

 

[Essence_of_War]

You're absolutely right about that SuperMicro board, I checked the specs and it lists pentiums/celerons, although Newegg strangely lists it for Xeon + i3 only.

What's the deal here? Are Intel and the MoBo manufacturers trying not-so-subtley to steer people towards particular choices? As I was checking out some other boards, I ran into AsRock doing this too, on their web specs for this board:

http://www.asrock.com/server/overview.asp?Model=E3C224D2I

they claim haswell Xeon's + i3's, as does Newegg, but then on their official CPU support list they include all of the Xeons, pentiums, celerons, and i3s but notably NOT i5/i7s?

 

[kevinsbane]

I recently built a freenas box, and ended up using the Asrock Mini-ITX with a Pentium G3220 + 8GB ECC ram. I don't see the point of going any higher for a file server.

 

[Torn Mind]

The info I come across seems to indicate that ECC RAM is essentially a necessity. The cheapest ECC option is an AM3+ mobo that supports it and and compatible processor.

 

[Essence_of_War]

The cheapest ECC option is an AM3+ mobo that supports it and and compatible processor.

I don't think AM3+ comes in mitx form factor, though.

Do the FM2/FM2+ sockets support ECC?

 

[Torn Mind]

I don't think AM3+ comes in mitx form factor, though.

Do the FM2/FM2+ sockets support ECC?

FM2/FM2+ don't seem to support it.

This is the thread that seems to be the most convincing argument for ECC RAM.
http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/

Seems like that the Mini-ITX server mobos sold at Newegg aren't cheap, and the cheapest one supporting at least 16 GB of RAM is $180.
http://www.newegg.com/Product/Produc...CE&PageSize=20


Or you can just get the board discussed earlier in the thread.

 

[smitbret]

I don't think AM3+ comes in mitx form factor, though.

Do the FM2/FM2+ sockets support ECC?

No and No

Just looked into this last month when I was considering moving some hardware around between my desktop and my server.

 

[mvbighead]

According to this article:
http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/

If you use ZFS, you SHOULD use ECC. I have ZFS currently, but no ECC. Guess i might change to something else. Not sure my board/CPU support it, and I don't feel like replacing 12GB of memory on my ESXi host.

 

[Knavish]

they claim haswell Xeon's + i3's, as does Newegg, but then on their official CPU support list they include all of the Xeons, pentiums, celerons, and i3s but notably NOT i5/i7s?

You can check Intel's site to confirm ECC support. It does look like all Haswell i3's support ECC. The i5's and i7's do not support ECC because they would eat into the workstation / small server market that pays a premium for Xeon E3's.

 

[Torn Mind]

You can check Intel's site to confirm ECC support. It does look like all Haswell i3's support ECC. The i5's and i7's do not support ECC because they would eat into the workstation / small server market that pays a premium for Xeon E3's.

E3s are actually discounted i7s, sometimes without the IGP. The E3-1245 v3nis about ten dollars cheaper, but its Turbo Boost max is 3.8 GHz instead of 3.9 GHz

 

[mfenn]

The info I come across seems to indicate that ECC RAM is essentially a necessity. The cheapest ECC option is an AM3+ mobo that supports it and and compatible processor.

According to this article:
http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/

If you use ZFS, you SHOULD use ECC. I have ZFS currently, but no ECC. Guess i might change to something else. Not sure my board/CPU support it, and I don't feel like replacing 12GB of memory on my ESXi host.

The scenario presented in that article doesn't make a lot of sense. It's extremely unlikely for a single bit location in DRAM to get stuck to a given position for a long period of time and it's even more unlikely for a given bit of data to get shifted relative to that location in RAM.

Obviously memory corruption is bad and can effect the files that get stored on disk along with anything downstream (backups, etc). However, the argument that the author of the above linked post is making, which is that ECC is more required for ZFS than for other filesystems doesn't hold water.

So like I said before, ECC is definitely nice to have, but don't let zealots convince you that ZFS somehow magically requires ECC RAM.

 

[Cerb]

Also, consider that Google's study pretty well tore apart the idea of most errors occurring with RAM at rest, or during refresh cycles. With any DIMM causing errors, there was a strong correlation to utilization, indicating that they either were mainly bus errors (most likely case), or were disturbances from reading and writing occurring nearby on-chip. So, an application or DB server would warrant ECC much more than a file server. In a business, you'd pay the paltry amount for ECC anyway, since it's so little extra, but for home, you might want to weigh the RAM and platform costs.

 

[bruceb]

I would set it up using 3 hard drives.
One for your Media Library
One for the Original Videos before any editing
One for the Videos after they are edited

This would make backups somewhat easier. I would not keep the original videos and the edited ones on the same drive. If it fails, you lose both copies.

 

[smitbret]

Also, consider that Google's study pretty well tore apart the idea of most errors occurring with RAM at rest, or during refresh cycles. With any DIMM causing errors, there was a strong correlation to utilization, indicating that they either were mainly bus errors (most likely case), or were disturbances from reading and writing occurring nearby on-chip. So, an application or DB server would warrant ECC much more than a file server. In a business, you'd pay the paltry amount for ECC anyway, since it's so little extra, but for home, you might want to weigh the RAM and platform costs.

I think the general attitude about ZFS+ECC is the idea that if you are going to bother to use a file system like ZFS that has such robust corruption prevention and detection, then why wouldn't you just step up and go whole hog with ECC memory, too. Otherwise, it would be like putting $300 high performace tires on a Toyota Camry when some $80 Goodyears would have done just as well.

From Wikipedia:

Work published between 2007 and 2009 showed widely varying error rates with over 7 orders of magnitude difference, ranging from 10−10–10−17 error/bit·h, roughly one bit error, per hour, per gigabyte of memory to one bit error, per millennium, per gigabyte of memory.[2][4][5] A very large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance’09 conference.[4] The actual error rate found was several orders of magnitude higher than previous small-scale or laboratory studies, with 25,000 to 70,000 errors per billion device hours per megabit (about 2.5–7 × 10−11 error/bit·h)(i.e. about 5 single bit errors in 8 Gigabytes of RAM per hour using the top-end error rate), and more than 8% of DIMM memory modules affected by errors per year.

When something has to be meaured in billion device hours then I feel pretty comfortable that it won't be a real world issue for my meager needs. If a memory error crashes my server once or twice a year, I can stomach the reboot process unless I have tendinitis in my index finger that makes hitting the power button extremely painful.

I just went ECC because I obsess. If I hadn't, I would have been sitting around at work one day thinking to myself, "Ya know, maybe I should have spent the extra $30 and gotten an ECC setup." Then I would spend the next 2 weeks devoting most of my free thought cycles to how or if I should do it now until I finally cave in and replace things to get the ECC.

In reality, it will probably never make a single bit of difference. In a low IO setup like most home server situations, the necessity of ECC is highly questionable in the first place. I just did it because I would get obssessed with the fact that I didn't and would have spent the $$$ later on plus the cost of a new CPU and board that supported it. It was really a budgetary move.

If you think you'll be fine knowing that you don't have ECC then you'll probably be just fine.

 

[smitbret]

I would set it up using 3 hard drives.
One for your Media Library
One for the Original Videos before any editing
One for the Videos after they are edited

This would make backups somewhat easier. I would not keep the original videos and the edited ones on the same drive. If it fails, you lose both copies.

Agreed, my server has 3 drives:

OS - 120GB SSD
Cache/Scratch - 1TB WD Green
Data Storage - 8TB RAID 5 array

I backup the OS because, hey, it's 30GB and backs up automatically in about 30 minutes while I sleep so I might as well just backup the whole thing.

I backup the important irreplaceable data from the array every day because it's irreplaceable.

The Cache and Scratch drive never gets backed up.

 

[_Rick_]

A reason to consider an i3 or low end Xeon over something cheaper is encryption acceleration. I would expect that if you run proper Unix ZFS, you'd encrypt it for simplicity when it comes to selling/RMAing hard disks, and not having to worry about data leaks. Also saves a lot of headaches in case of theft.
I would also hope, that AES-NI is supported by ZFS at this point in time - something to verify.

 

[Essence_of_War]

I would also hope, that AES-NI is supported by ZFS at this point in time - something to verify.

It looks like it is, so the i3 might actually be worth considering, I was thinking about encrypting the data, as you mentioned, to guard against theft.

 

[mvbighead]

The scenario presented in that article doesn't make a lot of sense. It's extremely unlikely for a single bit location in DRAM to get stuck to a given position for a long period of time and it's even more unlikely for a given bit of data to get shifted relative to that location in RAM.

Obviously memory corruption is bad and can effect the files that get stored on disk along with anything downstream (backups, etc). However, the argument that the author of the above linked post is making, which is that ECC is more required for ZFS than for other filesystems doesn't hold water.

So like I said before, ECC is definitely nice to have, but don't let zealots convince you that ZFS somehow magically requires ECC RAM.

True enough. It just frankly concerns me the way he puts it, as I used to do a lot of retail service and have replaced at least 50 bad sticks of RAM. In no case of a bad stick of RAM did I ever see complete data loss. I'd agree with you more than he, but he gave me the impression that ZFS just does something different, and it potentially could nuke all the data.