ZFS - the next generation of data storage

[sub.mesa]

It shocks me to see only one thread of the last 10 pages here on AnandTech are related to ZFS. Are you guys so old-fashioned, or hasn't the word reached Anand that ZFS provides reliable storage and modern features? Are there people using RAID here who do not know what ZFS is? Impossible! :sneaky:

Should you be willing to learn, in this thread I'd gladly explain what I can about this great technology. Just a quick ZFS primer for the impatient:

ZFS is a modern 'third generation' filesystem that could be described as the successor of the ubiquitous RAID which we're all too familiar with. However, while RAID was initially used to create a reliable storage device out of multiple cheap drives, it has hardly delivered us just that. In many cases, the RAID layer itself can fail and there can be a number of issues that threaten your data. Long story short: ZFS delivers where RAID fails.

ZFS uses checksums to protect the integrity of your data and protect the filesystem itself from corruption. Silent corruption is a thing of the past, and better yet ZFS is virtually immune to bad sectors in a redundant configuration. Unlike RAID which can have an entire array failed by just two 512-byte sectors being unreadable, ZFS offers much more protection and presents a new level of reliability to your data.

Especially with modern harddrives increasing their data density, the need for the features ZFS offers becomes critical. Those unaware of this issue - commonly referred to as the uBER issue - might want to read up on:

Why RAID 5 stops working in 2009
( I love controversial articles, especially when they're true )

ZFS is not available on Windows, and barely runs on Linux. But anyone who can operate a web-interface, can utilise ZFS. If you can use Synology, then you're smart enough to use ZFS as well using a similar web-interface provided by several ZFS platforms you can choose from. So the question is, who is interested in ZFS but perhaps requires some motivation and guidance to go ahead?

Kind regards,
sub.mesa 😉

 

[Emulex]

You know modern raid controllers have PI (sas) to do equivalent checksum on read?

it is still imperative to have end to end ECC IOECC/IOEDC to prevent other points of failure from compromising this protection!

 

[corkyg]

... ZFS is not available on Windows, and barely runs on Linux. ...

Well, as good as ZFS might be, that alone is reason to read, absorb, and move on. But, you are quite right to hoist it up on the flagpole and see if anyone salutes. 🙂

BTW, how does it run on Macs?

 

[sub.mesa]

Why would home users need End to End protection? Yes to protect their backups from contamination. But, what if you use ZFS send/receive for that?

The fun thing is, RAID was supposed to provide reliable storage using cheap hardware. Hence the I of Inexpensive in the acronym RAID, though often substituted for Independent because it doesn't apply any more. RAID has deviated a lot from its original intent and now is chuckfull of enterprise stop-gap measures instead of real solutions.

It isn't a solution to use 'dumb checksums' - you can still have corruption! The best method is to let the filesystem be aware of the contents and checksums, which is exactly what ZFS does. The problem is that it is incompatible with RAID and what ZFS does can never be imitated by RAID - dynamic stripesize for one.

So what you end up is two routes. The 'dumb' route requires you to spend a shitload of money on very expensive drives with uBER 10^-16 (a hundred times less bad sectors than casual drives). But also required are expensive RAID controllers, expensive battery backup, special drives with TLER functionality and as you indicated drives with additional spare space for poor-man's checksumming.

ZFS offers you a new route. Without expensive enterprise hardware but with the cheapest hardware you can find. Drives which generate bad sectors a factor hundred faster? No problem for ZFS. You also do not need TLER, no RAID controller, no battery backup. The regular chipset SATA works the best, the cheapest mass storage drive works the best. That is the cheap route.

Now the whole difference is that the expensive route requires virtually 'perfect' hardware to provide a decent level of protection, while the ZFS route grants comparable (actually MUCH better) protection without the need of all the expensive hardware.

In other words, you can solve the problem the 'dumb' way by throwing a shitload of money at it, or the 'intelligent' way by simply solving the reliability issue in software design. Of course, the companies promote their cashcows, but the 'smart' route is much more appealing, especially on the long term.

 

[sub.mesa]

Well, as good as ZFS might be, that alone is reason to read, absorb, and move on.

I don't understand this. Because ZFS runs on UNIX people discard this. But people use Linux on their phones, and Mac OSX is built on FreeBSD. I don't hear them complaining.

Even Synology - as well as most off-the-shelf NAS appliances - runs Linux. Of course the geeky world remains hidden from the user through the use of a user friendly shell; often a web-interface.

So then, how is ZFS at a disadvantage? Many Windows users use Synology, others use FreeNAS/NAS4Free/ZFSguru. All those provide easy frontends to the user, but Synology doesn't provide ZFS. Still, ZFS is 'out of the question' because it runs on UNIX. Arghh! :'(

BTW, how does it run on Macs?

It ran natively on Mac OSX with ZFS version 6 if I recall correctly. But Apple decided to remove it once Sun was acquired (usurped) by Oracle. Too bad, now Apple doesn't have any reliable filesystem and no prospect of getting one in a reasonable timeframe. HFS is legacy in capitals. But so are NTFS and Ext4.

But the idea is not that you run ZFS on your desktop, but that you use a dedicated NAS for storage and access your files over the network. Windows users will have a casual drive letter like Z:\ (applicable to ZFS :biggrin🙂.

 

[zir_blazer]

I think that what ZFS is useful the most for is to protect from silent data corruption. Silent data corruption is statistical very rare, so ZFS is as useful as ECC for RAM Memory. Unless you run something mission critical, you need deep pockets to justify prevention for such ocurrence.

 

[_Rick_]

ZFS to me is mostly a thing of convenience - collapsing two layers into one, because the storage and FS layer can be slightly smoother when combined.

I won't use it in the near future (probably the next decade), because I will stick to Linux for my file servers, and there's no encryption support for ZFS in Linux, and traditional encryption schemes operate between the device management and FS layer - which has disappeared with ZFS.

If somebody sits down and ports encryption to the Linux ZFS implementation (and it passes two or three years of productive use), then I am willing to reconsider. Until then RAID + XFS/JFS/EFS will be my weapon of choice.
Well, I could imagine virtualizing my servers on next gen hardware. If IO-virtualization between BSD and Linux works properly, then I might set up a BSD VM and share the ZFS...depending on how well NFS/CIFS works on BSD, maybe even via a quasi-local iSCSI export to a Linux VM that handles serving the data to the network.
Earliest chance of this happening is when I upgrade my server (likely soon - within 24 months), but more reasonably, I expect it to be interesting when I lose my Windows Server license, and set up DNS and AD on BSD and Samba. Probably in around 36-48 months, judging by my recent upgrade cycle.

 

[_Rick_]

Unlike RAID which can have an entire array failed by just two 512-byte sectors being unreadable, ZFS offers much more protection and presents a new level of reliability to your data.

That's also plain wrong.
You lose one 512 byte sector (if and only if both read errors are aligned!). You fix the read error, by overwriting the missing bit with zeros, and continue the rebuild. Sure, you lose some data, but that's the nature of read errors.

You can also use routine error-scrubbing/surface scans (reading the entire array regularly) to make this a complete non-issue.
Sure, it may be more comfortable and automatic in ZFS, but then, I'm not sure ZFS is simple enough that you can actually fix it by hand, when things go wrong. RAID on the other hand is easily navigated and troubleshooting it is reasonable easy, because the schemes are so trivial (and documented in the header, unless that's shot too...)

 

[Carson Dyle]

Not interested, but thanks for posting.

 

[Doomer]

ZFS is just too damned complicated for the average Joe to get his head around. I'm really amazed at how it often boils down to one persons opinion against another. every time I come across a thread about ZFS it invariably includes at least 2 posters arguing some obscure point. No thanks, I'll stick with what I know and understand until such time as ZFS becomes lest complicated and easier to set up and use.

"You had a HD fail in a RAID-Z2 array and now you don't know what to do? Really?"

 

[smitbret]

I think it's because you can get ZFS for free, so you don't get to buy an expensive RAID card and then brag about it. So, even though ZFS would be a much better and cost efficient solution for ninety percent of people here, no one is going to be impressed by your ability to download free software.

Don't underestimate the importance of being able to brag about the size of your penis.

 

[dighn]

"the next generation" would have been appropriate 5 years ago. anyone who cares and can use it is already using it.

it runs fine on linux actually because of zfs-on-linux, but the next generation will likely be btrfs simply due to it being officially supported on linux.

 

[Cerb]

It's not going to catch on outside of *n*x geeks, until it is very easy to use, which means a webmin GUI, curses interface, Qt GUI, etc., or just set it up and forget about it.

The big advantages of ZFS are:
1. Greater data protection on COTS drives
2. Integrating many services that are annoying to handle separately into an FS (RAID management, device management, snapshot/archive management, etc.)
3. Adding CRCs to RAID
4. No FSCK
5. Software support for caching devices.

#1 is going to be common in any upcoming FSes.
#2 is probably its greatest strength, in practice. The upfront learning curve saves a lot of niggling problems managing different software later on.
#3 allows RAID 1 and parity RAIDs to never need to guess or vote, if TSHTF with some drives, or, more likely, the transport mechanism. ZFS has helped diagnose faulty cables, PSUs, and networking hardware, thanks to this feature, so really, any new FS aught to be designed with CRCs of everything going to and coming from the drive added in.
#4 is needed for large FSes, but not an issue for most home users, or any client users. I can count the number of times I've needed a full-drive FSCK on EXT4, FI, on 0 fingers. It would be nice to have, though, if it wouldn't hurt normal performance much.
#5 is still interesting. Since the kernel has about as much information as the FS does to work with, I'm not sure if it will or will not be as good, or better, in practice. Since it could be optimized based on user behavior and what causes slowdowns seen by users, regardless of FS, it could be better just due to more eyes and more testers. But, the FS devs, having deeper knowledge of their FS' weaknesses and strengths, might be able to tailor caching to their weaknesses, and provide a better experience. Time will tell. Either way, software-based caching like Intel's SRT is going to be available to Linux users that want it, and will surely become good enough to use for clients and servers that need capacity that an SSD can't provide.

Negatives of ZFS, though, are:
1. Slow, if you use compression and/or dedup.
2. A RAM hog, especially using dedupe.
3. Needs an SSD cache, if using said features, to stay faster than a snail, if using parity RAID (but, forgo them, and it will be about as fast as hardware RAID, often faster).
4. Lack of [stable] TRIM support.
5. Substantially slower as a single-drive client FS, than EXT4.

I like ZFS, and use it on Linux now, but it was the pioneer. What we really need are future FSes to integrate the good features of it into their designs. If you removed the copies feature, dedupe, and parity RAID, FI, there's no reason you couldn't make a fast memory-efficient client FS with all the benefits of ZFS, and maybe more of their own. Give it RAID 0 and 1 support, and you could make it server-friendly, too.

 

[Maximilian]

ZFS is not available on Windows, and barely runs on Linux.

Well its humped for me then im afraid.

 

[blastingcap]

Many people are scared of non-Windows/non-Linux for whatever reason. I'm no pro in FreeBSD, but the way NAS4FREE is done (with GUI from your browser window once you install it), it's easy to set up a NAS box using ZFS. I could set one up from scratch in less time than it takes to install Windows. Try it before you knock it.

 

[aigomorla]

"You had a HD fail in a RAID-Z2 array and now you don't know what to do? Really?"

response..

" you setup a Raid-Z array without knowing anything about it including on how to rebuild your Raid-Z array?" 😱
Lets be realistic here... Raid-Z is almost identical to a normal Raid-5.
Only file system format is different... u would need to do the same thing on a ZFS that you would need to do on a standard RAID.
The disk assignment rules also applies to ZFS that it would apply to raid.

Well its humped for me then im afraid.

its used on FreeNAS as a headless box with drives.
It takes JBOD and turns it into a fault tollerant Raid-Z array.
The entire OS boots off a USB Thumb drive, so you can have all your SATA ports populated with data drives.
ZFS is also not hardware dependant... its SOFTWARE dependant... meaning if ur motherboard and sata ports ever die... you transfer the flash + drives onto a new platform and rebuild.
No need to: "omg, i lost my raid controller, and my raid drives.. oh noes... i need to buy another 500 dollar raid contoller to get my drives up."

Why would u need linux on a ZFS file system which is mainly used to store data.

I can go on and on about the different file systems...
Personally as long as it IS NOT windows server 2012 failed RTSF! you are golden!
Dont get me started on RTSF....

Anyhow... all my real NAS's are on ZFS.
My Window Network drives which i need speed on are all on a dedicated controller tho in windows.

 

[Zxian]

You know what I want? Drobo's redundancy (even better, add double redundancy) with the flexibility of any and all drive configurations. I don't want to have to buy batches of disks to upgrade my storage in future. I'd rather add them to the existing pool of drives and have the redundancy features take care of themselves.

ZFS doesn't support any sort of drive migration AFAIK. This is a dealbreaker for me. That and the ease-of-use reasons mentioned before. I'd really rather not throw yet another OS that I have to manually manage and fight with to work nicely with all of my devices.