ZFS Tuning

[XavierMace]

Do we have any experts on here on ZFS tuning? I got my new storage build setup and it's running great but benchmarks are showing what seem like odd results to me. I can post screenshots when I get home, but when I run HDTune from VM's running off that storage, it sits around 100MB/s for the first 10-15% of the bench, then goes up to 800MB/s-1,100MB/s. Averages out to about 800MB/s which isn't bad (faster than my striped SSD's), I'm more just confused by the initial slowness. Or is that normal for ZFS?

Setup:
Solaris 11 + Napp-it
Norco 4220
2x Xeon 5560's
48Gb Kingston DDR3 ECC
HP P410/512 in HBA mode
8x Seagate 2TB 6Gb/s SATA drives (ZFS, RaidZ2)
QLogic Quad Port 4GB Fibre HBA

The ESX hosts all have the same 4Gb HBA's and it's running through a Brocade Silkworm 4100.

Before you ask why I need that kind of setup... Because. That's why.

 

[Martin2341]

Sorry but I'm afraid I'm NOT a ZFS tuning expert however I wanted to give you kudos for running the fiber channel setup with the Brocade 4100. Sorry again!

 

[Atreidin]

I am going to be putting together a server for ZFS soon and I was looking into ZFS support for 4k sectors. I am guessing your 2TB hard drives have 4k sectors, aka "advanced format", but the drive lies to the OS and says it has 512 byte sectors instead.

Did you do anything special to account for that? If the sectors aren't aligned properly that will give a performance hit.

I would post a link but this forum is being retarded and mangling the URL.
http://wiki.************/display/illumos/ZFS+and+Advanced+Format+disks

 

[Tristor]

If you haven't yet, you should read this ENTIRE document: http://webcache.googleusercontent.c...nals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

Google Cache because the site is down right now for some reason, and as far as I know that's the only extant copy (I've saved a copy locally, I may try to resurrect it somewhere).

After you read that, you should read this:

http://webcache.googleusercontent.c...s.com/wiki/index.php/ZFS_Best_Practices_Guide


As a general rule, it is not really possible to give ZFS tuning advice since it changes considerably based on your particular use case. The default settings tend to be pretty decent, however I generally only make one change which is to set ARC min/max size. In many cases even this is not necessary.

I don't really know what to tell you on your performance dip, I tend to see the opposite play out in my configuration, which is to be expected due to caching. That is, large sequential reads tend to be faster in the beginning and then drop off in performance at some point, which I assume to be the point it's moved from hitting the ARC/L2ARC and is hitting spinning disk.

You should also consider using SSD caching for both ZIL and L2ARC if you are not already, if you do, ensure you set up your log devices as a mirror.

 

[XavierMace]

I am going to be putting together a server for ZFS soon and I was looking into ZFS support for 4k sectors. I am guessing your 2TB hard drives have 4k sectors, aka "advanced format", but the drive lies to the OS and says it has 512 byte sectors instead.

Did you do anything special to account for that? If the sectors aren't aligned properly that will give a performance hit.

I would post a link but this forum is being retarded and mangling the URL.

My understanding is with the newer versions of ZFS (which Solaris 11 is the newest) it's a non-issue as long as you aren't adding drives to an existing pool. This was a brand new pool. That said, I'll check that to be safe.

If you haven't yet, you should read this ENTIRE document: http://webcache.googleusercontent.c...nals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

Google Cache because the site is down right now for some reason, and as far as I know that's the only extant copy (I've saved a copy locally, I may try to resurrect it somewhere).

After you read that, you should read this:

http://webcache.googleusercontent.c...s.com/wiki/index.php/ZFS_Best_Practices_Guide


As a general rule, it is not really possible to give ZFS tuning advice since it changes considerably based on your particular use case. The default settings tend to be pretty decent, however I generally only make one change which is to set ARC min/max size. In many cases even this is not necessary.

I don't really know what to tell you on your performance dip, I tend to see the opposite play out in my configuration, which is to be expected due to caching. That is, large sequential reads tend to be faster in the beginning and then drop off in performance at some point, which I assume to be the point it's moved from hitting the ARC/L2ARC and is hitting spinning disk.

You should also consider using SSD caching for both ZIL and L2ARC if you are not already, if you do, ensure you set up your log devices as a mirror.

I'm having trouble getting those links to load here at work, I'll check them from home. That said I'm familiar with most of the best practices as I read up quite a bit when I did my first ZFS build. However, I wasn't too concerned with performance on that build as it had other bottlenecks.

SSD caching would be great, but I'm out of bays until I add my second controller. The mobo only has SATA II onboard so I don't think the cache would be much help there. Plus, doesn't having the extra memory help alleviate that?