Question ZFS and DeDupe, QTS Hero OS, and backups using Macrium Reflect (Free)?

[VirtualLarry]

Right now, I'm using some QNAP NAS units to store backups on, and using Macrium Reflect Free to run the backups on the client PCs. I have been using the default settings, which use "Medium Compression" for the backups. They also backup rather slowly, but I think that may be due to the write speed to the NAS units, being that I'm using volume encryption, then folder encryption, and then SMB encryption for transmission over the LAN. Three layers of encryption (even with hardware support), may be a little too much for those little NAS processors.

But QNAP is working and soon (maybe Dec.?) to release their "QTS Hero" OS, which is based on ZFS filesystem, technology.

Among the other gee-whiz features, is DeDupe.

So what I was thinking was, instead of using compression on the backups coming from the client PCs, instead, have them back up using "no compression", and then let ZFS DeDupe, Compression, and Encryption, work in the background on the NAS. If I have multiple backups, of the same OS-level system binary files, or even, across multiple client PCs, all running roughly the same OS (Win10, various upgrade versions, though generally the newest), then I should be able to save a lot of space on the back-end, at the cost of possibly the backup software sending more data over the LAN each time I back up. Indeed, it might be better, to even schedule daily or weekly "full" backups, to the ZFS NAS, rather than use the backup software's own sort of block-change algorithm, using "differential" backups, which track changed files since the last "full" backup.

 

[gea]

ZFS realtime dedup (outside Oracle Solaris with a native ZFS and dedup2) is currently a ZFS feature that offers mostly more disadvantages than advantages mainly because of its huge RAM needs (count up to 5GB RAM per used TB dedup data) additionally to the RAM that your OS needs and the RAM that you want to use for read/ write caching on ZFS. This is why one prefers mostly lz4 compress over dedup.

This may change now with the upcoming ZFS Allocation classes. This are special vdevs ex a high performance NVMe to hold the dedup table or to improve performance for metadata, small io or single filesystems, see http://napp-it.org/doc/downloads/special-vdev.pdf

Currently Allocation Classes (like the other new ZFS features ZFS encryption or fast sequential resilvering) is in Illumos (Free Solaris Fork) and ZoL and the next Free-BSD. In the past, the Qnap ZFS solution was based on Free-BSD. I am not aware about the platform for Hero OS. Maybe they switch to ZoL.

 

[VirtualLarry]

In the past, the Qnap ZFS solution was based on Free-BSD. I am not aware about the platform for Hero OS. Maybe they switch to ZoL.

That's interesting. Let's hope it's ZoL. Their regular QTS OS is based on Linux.

I was reading your thread the other night too, about ZFS and ZoL developments. Interesting stuff!