I should like an archival file server -- data integrity, not speed, is the paramount goal. ZFS seems like the optimal foundation for such a server
Indeed! In fact, I would say that ZFS is the only reliable and mature way to store files accessible to us mortals at this time. All other solutions have serious shortcomings or simply belong to a different era.
Foremost are the hardware requirements. Most enthusiasts will urge a capable cpu and 8+ GB of memory to run ZFS, but are these strictly necessary?
You should not go lower than 8GiB on RAM whenever possible. But ZFS is not CPU-intensive unless you will be making heavy usage of features like compression, encryption or deduplication.
What ZFS wants is to have lots of RAM memory, more than anything else. This enables ZFS to achieve higher performance; if you have 40 drives but only 4GiB RAM, then you will only achieve about 10% of potential performance. Any lower and ZFS could become unstable as well unless properly tuned.
As for performance, your most immediate problem is high latencies on transaction group cycles. Whenever a transaction group is written, your clients may have higher latencies when reading data for example when streaming a video. This could lead to pauses or stuttering every x seconds. If you encounter this, either your disks are to slow to keep up with the transaction group cycles, you have too few RAM or the system tuning is way off.
I am concerned with data integrity, and ZFS's checksums could use a respectable cpu
You can use the slowest CPU you like. AMD Brazos is the slowest you can get on desktop, but it works pretty much great. I do recommend you get something a bit more powerful. Intel chips generally are preferable to AMD due to Intel offering better CPU performance. The better GPU performance of AMD is not very beneficial to a ZFS server.
As you can probably guess by now, my preference would be to recycle obsolescent tech in this server.
Generally this is not recommended. ZFS likes new hardware in particular because DDR2 memory is expensive while DDR3 is cheap. Newer hardware is also a lot more power efficient. Buying a new system may be free if you consider 24/7 power savings over a three year period.
Additionally, is it still inadvisable to run ZFS on 4K-sector hard drives?
It has never been inadvisable to use 4K sector harddrives with ZFS. The 'mother' platform of ZFS, Solaris, just has a weak infrastructure to deal with 4K sector harddrives. That is a weakness of the operating system - not ZFS.
ZFS itself is properly designed to cope with all sector sizes, from 1 byte to 128KiB. Sector sizes 512B, 4K and 8K are the most common.
I recommend you run a FreeBSD platform to utilise ZFS. It has the best ZFS implementation in my opinion.
The system will craw to a halt on 2GB of RAM. You need MINIMUM of 1GB of RAM per Terrabyte of storage
This '1GB per 1TB' rule of thumb is simply false. You need 1GB, you want 8GB+. You don't need a lot more memory when your disks get bigger. Instead, you need more memory to utilise the performance potential of faster drives or bigger arrays.
A better rule of thumb would be '1GB RAM per 100MB/s of performance' but such statements are too generic for me to promote.
Unless you buy a small SSD for use as the ZFS ZIL log, you will need to turn the ZIL log off (as your performance will be horrible)
NEVER TURN THE ZIL OFF!
The ZIL should also not impact performance that much, depending on usage. There are exceptions. Some clients use NFS with sync writes which means performance will suck without a dedicated log device, properly called a SLOG (separate log).
The ZIL is very much misunderstood. Many people think ZIL is a write cache. It is not. It is an intent log or 'journal'. NTFS uses a comparable feature. ZFS just allows storing the journal/ZIL on a dedicated device instead. In most cases, don't worry about it. But please, do not turn it off?! If you don't care about application consistency, you can turn sync writes off: zfs set sync=disabled tank. But never disable the ZIL!
You can run ZFS on 4k drives, however, you need to do some tuning when you create the zpool, and ALL drives in the pool need to use 4k sectors (you can NOT mix them with older 512byte, or with drives that use 4k but emulate 512byte to the OS).
You can mix 4K and 512B drives just fine. You are right that tuning is recommended when you create the ZFS pool. But not strictly necessary and in some cases it is better not to optimize for 'ashift 12'.
Optimal for 4K harddrives:
Mirror: all are optimal
Stripe: all are optimal
RAID-Z: 3, 5 or 9 disks (or 17- but that is not safe with RAID-Z)
RAID-Z2: 4, 6 or 10 disks (or 18 - but that is not very safe with RAID-Z2)
RAID-Z3: 5, 7 or 11 disks (or 19)
If you stick to these optimal configurations, you will have more storage space available and general better performance. The storage space issue is not very well known, by the way.
However, the general consensus is that you want a 64-bit server specifically to overcome the 32-bit 4GB RAM limitation and that anything less for a multiterabyte ZFS SAN server will hold you book.
The 64-bit versus 32-bit is not just about RAM addressing. ZFS was basically written for 64-bit operating systems. On 32-bit OS it can even become unstable. Today, ZFS on 32-bit should be usable, probably on embedded platforms. Still, I strongly recommend not to use older 32-bit hardware like Pentium 4 and Athlon XP.
The memory requirement is based on how ZFS itself operates. The writes are cached first into RAM because the system will wait for a certain amount of data is received before it can write to disk as it writes in 128k blocks, which also require a checksum to be created first before it can write it to disk.
Writes are buffered in RAM because ZFS is a transactional filesystem. The 128K recordsize doesn't really have much to do with this. In fact, other RAID5 engines have a much harder time; they have to buffer all writes and wait until they form optimally sized blocks before writing them out. RAID5 knows read-modify-write, whereas RAID-Z does all I/O in one pass.
Just don't run parity usually a dedicated raid card does that, if you don't have one then your cpu becomes the raid card and has to do all the calcs. It will slow ya down, so stick with non parity raid 1 or 10 in an old system.
You have been taught that RAID5 is slow because of the XOR parity calculations, right? Actually, that is one big myth! You can verify it yourself. When booting one of many Linux distributions, the md software RAID driver actually performs mini benchmarks to choose what parity algorithm works fastest on your computer. Generally XOR is bottlenecked by memory bandwidth; over 10GB/s on modern systems. The truth is that XOR is one of the easiest tasks your CPU can do.
The REAL reason why RAID5 can be hard on the CPU is that all the splitting and combining of I/O requests in order to optimise for the RAID5 full block stripe, will consume lots of memory bandwidth, which shows up as CPU usage.
AMD Brazos doing RAID-Z3 (triple level parity) would work just fine. Yes, it'll be a bit slower, but not that much.
Please also let me remind others that
ZFS should never be used with hardware RAID controller. You need non-RAID controllers to properly utilise ZFS. Using a controller like Areca or HighPoint to simply give raw disks to ZFS is
not going to work properly in particular to bad sectors and other timeouts.
I just upgraded my old nas server this past week to an amd c-60 board and I was on the fence on going between 8gb of ram and 16gb, and I am glad I went with 16. I haven't gotten all of my data transferred yet, but it is already using over 8gb at times.
All filesystems (actually: VMM infrastructure) will use as much RAM as possible for filecaching. NTFS and Ext3 just as easily can use over 100 gigabytes of RAM if you have that much. All you need to do is read some data from your harddrive. All free RAM will be utilised as filecache.
You are right though; 16GiB RAM is very much preferable. To ZFS, you want as much RAM as you can afford. Pairing a modern quad core Intel chip with 32GiB RAM is the practical maximum, however in my assessment the proper balance for that much CPU power would be at least 128GiB of RAM.
ZFS preferring to have lots of RAM is often cited as a weakness. I see it as one of its stronger points. ZFS introduces a much more intelligent way of caching as opposed to the dumb last-access cache of virtually all other filesystems. ZFS chooses what data to cache much more intelligently, causing it to adapt to the most recent workload but also lesser recent workloads. This advantage is strengthened when using L2ARC SSD caching. The only requirement to make this work effectively is that ZFS runs 24/7.
). Even with cheap electricity, the power consumption argument is compelling, these days.
Facebook
Twitter