Hardware and software questions about a home server/zfs-pool

[mfenn]

True enough. It just frankly concerns me the way he puts it, as I used to do a lot of retail service and have replaced at least 50 bad sticks of RAM. In no case of a bad stick of RAM did I ever see complete data loss. I'd agree with you more than he, but he gave me the impression that ZFS just does something different, and it potentially could nuke all the data.

The nightmare pathological case that the author describes is possible, but it is incredibly unlikely. Like might happen to one or two ZFS systems for the useful life of the technology. What's more likely is that the bad memory will crash the box (as is your experience) or cause a bitflip in a file.

 

[mvbighead]

The nightmare pathological case that the author describes is possible, but it is incredibly unlikely. Like might happen to one or two ZFS systems for the useful life of the technology. What's more likely is that the bad memory will crash the box (as is your experience) or cause a bitflip in a file.

The only thing I can add is that in reading his shpeel, the ZFS system is simply different than traditional systems. He had at least a few cases of complete data loss that he referred to, which leads me to believe his thoughts are not completely bizarre.

I just know that it seems like a terrible idea if it can destroy the entire volume.

And to me, I am now looking for a NAS package that is low on CPU. Most of my computers are Windows 7 (one Win8), and every time I upload a file, it goes quick for the first half or so, then the CPU spikes on the NAS and the transfer slows to 20mb. I'd like something that simply reads and writes data without stressing the CPU much. Windows would obviously do it simply, but I'd rather not buy another license. Maybe Windows Home Server as a VM. I dunno, just not thrilled with the way FreeNAS is working at the moment. I simply want an always on VM that can quickly write the data to the disk at disk and wire speed.

 

[mfenn]

Are you running ZFS right now or something else?

 

[mvbighead]

Are you running ZFS right now or something else?

Running ZFS on a single disk at the moment. Need to move some data from another to it, and then mirror the drives. 2 x 2Tb is what I have, which is currently adequate for my needs.

 

[mfenn]

Running ZFS on a single disk at the moment. Need to move some data from another to it, and then mirror the drives. 2 x 2Tb is what I have, which is currently adequate for my needs.

You should log into the server while you're writing and run iostat -xd -w 1 . The output will look similar to this:

freenas:~# iostat -xd -w 5
                        extended device statistics  
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  
ad0        8.8   3.6   500.8    21.2    0 195.5  11 
ad1        8.8   3.5   501.5    21.1    0 196.9  11 
ad2        8.8   3.4   510.0    21.5    0 187.9  10 
ad3        8.8   3.6   510.1    21.1    0 183.7  10 
ad10       8.8   3.5   543.9    21.5    0  82.9   5 
da0        0.2   0.0     3.4     0.0    0   1.0   0 
pass0      0.0   0.0     0.0     0.0    0   0.0   0 
^C
freenas:~#

If the service time (svc_t) and utilization ratios (%b) are high, then the disk is going as fast as it can and there's not much you can do. If the disk appears to be mostly idle when performance goes down, then there is probably some ZFS tuning that you can do to make things better.

Keep in mind that ZFS isn't really designed to be used in single-disk environments though.

 

[Essence_of_War]

Suppose I have an enclosure + MoBo that will fit 6 drives.

Which of these two would be better practice for a balance of redundancy and capacity?

1) To have a zpool made up of 1 VDEV w/ all 6 drives in a raidz2?
OR
2) To have a zpool made up of 3 VDEV's w/ 2 mirror'd drives in each vdev?

The latter would be nice since it allows you to expand the zpool by adding the vdev's one at a time, not as much money up-front. The former seems like it would be better for overall redundancy since it can recover from any two drive failures, rather than up 1-to-3 drive failures depending on which physical drives failed.

 

[Cerb]

Either are good, and either are current best practices for business, much less home NASes. I would make the choice entirely based on when and how I expected to expand each setup.

The catch is that, while RAIDZ2 gives you 2 parity drives, it doesn't give you 2 drives that can fail, without the risk of data loss getting fairly high. The 2 parity drives are so that 1 drive can fail, without losing protection from errors. If both drives fail, it's time to blow it away and get your backups. With all parity drives worth failed, you have what amounts to RAID 0, and you may get close to the drives' rated URE rate during rebuild, as all the drives are seeking the whole time, as each seek from each drive affects each other drive, increasing wear and lowering performance.

RAIDZ3 really gives you 2 drives to fail. RAIDZ1/RAID 5, making big arrays with big SATA disks, is generally expected to lose data if a drive fails, so should only be used if that's a minor issue (in home use, that's often true), or if using SSDs (where the URE rate is tied to NAND wear, so will be less of an issue in practice). ZFS is going to be better in terms of knowing what data is bad where and when, but it can't fix bad data without enough good data to read, which is what the remaining functional "parity drive" ensures--that any single data error can be recovered from.

In a mirrored RAID, only the drives in that mirror containing the data are read, and only the replaced drive is written to, as far as recovery goes. So, that's all drives seeking at once, guaranteed, against 2, and those 2 can be doing simple large sequential transfers between each other. The relatively quick rebuild, low amount of data needing to be read, and lowered drive stress, generally makes it much more resistant to errors during rebuild, compared to parity RAID (IE, the actual URE rate should be much less than spec, and the drives are reading and writing far less data in the process of the rebuild). If it fails to rebuild, assuming it had been scrubbing regularly, you got unlucky (probably with 2 drives of the same batch, with some kind of minor issues from the factory), and probably would have had just as much of a chance of that failure with the RAID 6/RAIDZ2.

Both setups will bring the practical data loss risk down to that of either bad hardware or power issues (IE, luck), at which point other means of keeping your data safe become far more cost-effective, like offline backups, keeping the data on client PCs, too, cloud backups, etc.. You can crunch stats all day, but if you get 2 drives that are going to give out within days of each other, or a bum PSU, or a storm causes data corruption cross multiple drives, you're faced with a random URE during a mirror rebuild, etc., you're better off having copies in totally separate computers, than worrying about additional redundancy within a single unit.

 

[mfenn]

The catch is that, while RAIDZ2 gives you 2 parity drives, it doesn't give you 2 drives that can fail, without the risk of data loss getting fairly high. The 2 parity drives are so that 1 drive can fail, without losing protection from errors. If both drives fail, it's time to blow it away and get your backups. With all parity drives worth failed, you have what amounts to RAID 0, and you may get close to the drives' rated URE rate during rebuild,as all the drives are seeking the whole time, as each seek from each drive affects each other drive, increasing wear and lowering performance.

This may be true with a traditional RAID 5/6, but it is not true for ZFS RAID Z1/Z2. You still have object checksums after a double disk failure. You cannot repair a read error, but ZFS will know about any errors that may occur and you can do a targeted recovery of a few files (at most) rather than a full recovery.

To Essence's question, I would probably go with the 3 mirrored pairs if I wasn't hurting for space. You'll get less CPU-intensive rebuilds, and easier upgrades. Note, that you can't remove a vdev, even with mirrors. It does make upgrading easier because you only have to cycle out two drives rather than every drive in the array.

 

[Essence_of_War]

Thanks Cerb and mfenn, good stuff to chew on.

I like the upsides to the mirror'd vdev's even if it costs a little bit of capacity over the one large raidz2 vdev.

 

[mvbighead]

You should log into the server while you're writing and run iostat -xd -w 1 . The output will look similar to this:

freenas:~# iostat -xd -w 5
                        extended device statistics  
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  
ad0        8.8   3.6   500.8    21.2    0 195.5  11 
ad1        8.8   3.5   501.5    21.1    0 196.9  11 
ad2        8.8   3.4   510.0    21.5    0 187.9  10 
ad3        8.8   3.6   510.1    21.1    0 183.7  10 
ad10       8.8   3.5   543.9    21.5    0  82.9   5 
da0        0.2   0.0     3.4     0.0    0   1.0   0 
pass0      0.0   0.0     0.0     0.0    0   0.0   0 
^C
freenas:~#

If the service time (svc_t) and utilization ratios (%b) are high, then the disk is going as fast as it can and there's not much you can do. If the disk appears to be mostly idle when performance goes down, then there is probably some ZFS tuning that you can do to make things better.

Keep in mind that ZFS isn't really designed to be used in single-disk environments though.

Eh, what I was referring to was CPU. The CPU load spikes BIG time. I've not messed with it lately, but for me, a file server shouldn't use much CPU. Only thing I can figure is it is converting something during the transfer. Doesn't make a lot of sense. All I know is, something seems off. My Server 2k3 box that I originally had never had this sort of issue.

 

[Viper GTS]

If you're comparing CPU use on a ZFS based system compared to 2003 and faulting ZFS for its CPU load perhaps you don't understand what ZFS does?

Viper GTS

 

[mvbighead]

If you're comparing CPU use on a ZFS based system compared to 2003 and faulting ZFS for its CPU load perhaps you don't understand what ZFS does?

Viper GTS

Quite true. What does ZFS do?

And more to the point, what out there doesn't do whatever ZFS does? I am simply looking for a simple intuitive interface that acts as a file share allowing me to write to my disk based on local accounts. I got all that setup (currently in ZFS), but the fact that a file server gouges the CPU seems odd when I simply want to write data to a disk.

I'm not looking for some massively powerful storage pool. I am simply looking for a simple NAS appliance with good enough security to only allow connections that match those specified in the appliance.

 

[dave_the_nerd]

Quite true. What does ZFS do?

And more to the point, what out there doesn't do whatever ZFS does? I am simply looking for a simple intuitive interface that acts as a file share allowing me to write to my disk based on local accounts. I got all that setup (currently in ZFS), but the fact that a file server gouges the CPU seems odd when I simply want to write data to a disk.

I'm not looking for some massively powerful storage pool. I am simply looking for a simple NAS appliance with good enough security to only allow connections that match those specified in the appliance.

Then keep it simple - use FreeNAS or NAS4Free with UFS instead of ZFS. It's the "traditional" file system, doesn't really have any limits a home user is likely to run into.

 

[Viper GTS]

Quite true. What does ZFS do?

And more to the point, what out there doesn't do whatever ZFS does? I am simply looking for a simple intuitive interface that acts as a file share allowing me to write to my disk based on local accounts. I got all that setup (currently in ZFS), but the fact that a file server gouges the CPU seems odd when I simply want to write data to a disk.

I'm not looking for some massively powerful storage pool. I am simply looking for a simple NAS appliance with good enough security to only allow connections that match those specified in the appliance.

ZFS moves functions that are typically done in hardware XOR engine (parity calculation), dedicated storage processors like a NetApp or similar (deduplication, compression), or simply not done at all (bit level error detection/correction) and puts them in software. Because you're running it in a VM vs a dedicated appliance you're seeing all of this CPU load in a place you don't want it, but those functions require that load to be handled somewhere.

If you don't need the functionality or data protection then it is just wasted overhead for you.

For what it's worth here's a typical hour of my Nexenta install (also in a VM):

Viper GTS

 

[mfenn]

Then keep it simple - use FreeNAS or NAS4Free with UFS instead of ZFS. It's the "traditional" file system, doesn't really have any limits a home user is likely to run into.

This is a good idea. You don't have to use ZFS with FreeNAS. It still supports the tradition UFS and software RAID setup. FreeNAS has a really nice interface for managing things, and you get to keep that even when you use UFS.

 

[mvbighead]

ZFS moves functions that are typically done in hardware XOR engine (parity calculation), dedicated storage processors like a NetApp or similar (deduplication, compression), or simply not done at all (bit level error detection/correction) and puts them in software. Because you're running it in a VM vs a dedicated appliance you're seeing all of this CPU load in a place you don't want it, but those functions require that load to be handled somewhere.

If you don't need the functionality or data protection then it is just wasted overhead for you.

For what it's worth here's a typical hour of my Nexenta install (also in a VM):

Viper GTS

Yep, definitely far more than I need. At this point, I am trying to get everything moved to a NAS VM so I can discard the Windows VM. Problem is, I haven't been too happy with the performance considering what I want out of it. UFS it is.

I did read some of the articles on ZFS, but it all seemed like it was more memory happy than CPU happy. With a simple 2TB HD (and a second for redundancy later), I didn't figure there would be any overhead considering it is only writing data to one disk.