Info Fast ZFS Storageserver with Oracle Solaris, OmniOS and napp-it

[gea]

napp-it Z-Raid vCluster v.2

I am on the way to finish my Z-Raid Cluster in a Box solution.
In the past a Cluster in a Box consists of two servers with a common pool of mpio SAS disks. One of the servers builds a pool from the disks and offers services like NFS or SMB. On a failure or for maintenance you can switch over to the second server within a few seconds. Management is done for example with RSF-1 from high-availability. SuperMicro offers special cases that can hold two mainboards

Such solutions are expensive and management is quite complex but offers high performance and high availability. To reduce costs, my solution is based on ESXi (any license) to virtualize the two nodes and a control instance. It uses the shared controller/ shared raw disk options of ESXi so my solution does not need multipath SAS but can work with any disks.

Setup see http://www.napp-it.org/doc/downloads/z-raid.pdf

If you want to try the preview state that allows a manual failover within 20s, you can update napp-it to the free 18.02 preview (or 18.12dev).

 

[gea]

vCluster Beta2 is available (napp-it 18.12g dev)

Current state:
Manual Failover between nodes working for NFS and SMB
For SMB it supports failover for local user and AD users connected to the share during failover.

Todo
auto-failover (on tests)

Expect vCluster to next Pro (Jan 2019)

 

[gea]

I have added a sample configuration for a 2000 Euro/USD ZFS 6TB SAS HA Cluster (whole hard and software) with a step by step setup manuals to http://www.napp-it.org/doc/downloads/z-raid.pdf

 

[gea]

next napp-it 18.12p free is available
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux Changelog

 

[johnylesnar123]

AiO with Solaris 11.4 on ESXi 6.7
No support from current Vmware tools for Solaris 11.4 final from today
vmware-tools / open-vm-tools on 11.4b |Oracle Community

My findings/ "just a hack"

VMware vmtools for Solaris from ESXi 6.7
Executing on a textonly setup of S11.4 final on ESXi 6.7

Installer vmware-install-pl on 11.4 installs but fails with a message
Package "SUNWuiu8" not found when executing vmtool/bin/vmware-config-tools.pl

This can be skipped by editing vmtool/bin/vmware-config-tools.pl in line 13026
when you comment out the check for SUNWuiu8

When you then run vmtool\bin\vmware-config-tools.pl it hangs due a missing /usr/bin/isalist
I copied isalist over from a Solaris 11.3, made it executable and then vmware-config-tools.pl works

After a reboot I got the message vmools installed with a console message
Warning: Signature verification of module /kernel/drv/amd64/vmmemctl failed

same with verification of the vmxnet3s driver
vmxnet3s reports deprecated "misc/mac"

Not sure if this is critical

vmxnet3s and guest restart from ESXi works

Gea

Valuable information

 

[gea]

Use VMware tools from ESXi 6.7u1
6.7 tools are older than Solaris 11.4

 

[gea]

New

OmniOS 151030 LTS/long term support is out.

omniosorg/omnios-build
Downloads

​

 

[gea]

OpenIndiana Hipster 2019.05 is out
Release notes: 2019.05 Release notes - OpenIndiana - OpenIndiana Wiki

Editions:
GUI/live for server and desktop use with a Mate 1.22 desktop
Text (prefer this this for server use)
Minimal (minimal Ilumos)

about OpenIndiana.
OpenIndiana is always based on the most current Illumos (the common development platform for all Illumos based distributions) with quite a huge repository including many server and desktop apps. This is different to the new OmniOS 151030 LTS (long term stable) from may 2019 that is a freeze of the current Illumos state for the next 2 years with only security and bug fixes. Beside extras like LX zones and Bhyve OmniOS 151030 is quite similar to OpenIndiana 2019.05 text.

 

[gea]

Native Open-ZFS encryption is in Illumos since today
https://illumos.topicbox.com/groups/developer/Tf7694793544b0431

OmniOS:
https://gitter.im/omniosorg/Lobby

OpenIndiana
if illumos is at 0.5.11-2018.0.0.18656 or newer

 

[gea]

OmniOS bloody 151031 now supports native ZFS encryption

napp-it 19.dev from today (jul 04) supports encryption
in menu ZFS filesystem (create, lock, unlock)

 

[gea]

What happens to your data on RAM problems without ECC?
Why it is superiour to have checksums then (ZFS)

nice example, see
https://forums.servethehome.com/ind...on-scrub-–-how-do-i-see-affected-files.24990/

 

[gea]

SMB3 (kernelbased Solarish SMB server) is announced for OmniOS 151032 in November 2019,
see omniosorg/Lobby

If you want to try it now, use OpenIndiana (always newest Illumos) or OmniOS bloody 151031

 

[gea]

ESXi 6.7 U3 is available
https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-67u3-release-notes.html

Fixed from 6.7U2
Import of ova templates not possible from the web-ui in 6.7u2

Deployment of my napp-it all-in-one webmanaged ZFS server template works now again
https://napp-it.org/doc/downloads/napp-in-one.pdf

 

[gea]

ZFS Allocation Classes
This is a new feature in Open-ZFS

It allows to add special vdevs or dedup vdevs to a pool. Dedup vdevs are used to hold the dedup table to end its ram problem. Special vdevs hold metadate or small io. This allows to create mixed pools of disks and SSD/NVMe where performance critical data or filesystems can land on the faster vdev(s).

I have made some performance benchmarks,
http://napp-it.org/doc/downloads/special-vdev.pdf

I am really impressed about the result as this allows to use a slow disk pool where you can decide per ZFS filesystem based on the "recsize" vs "special_small_blocks" settings that data of this filesystem land on the special vdev ex an Intel Optane.

http://www.napp-it.org

 

[gea]

Update

Dedup and special vdevs are removable vdevs. This works only when all vdevs in the pool have the same ashift setting, ex ashift =12, best for 4k disks.

At least in current Illumos there is a problem that a pool crashes (corrupted) when you try to remove a special vdev from a pool with different ashift settings, ex a pool with ashift=12 vdevs and a special vdev with ashift=9. In current napp-it-dev I therefor set ashift=12 instead "auto" as default to create or extend a pool.

If you want to remove a special or dedup vdev, first check the ashift setting of all vdevs (menu Pool, click on the datapool). I have send a mail to illumos-dev and hope that this bug is solved prior next OmniOS stable.

If you create or extend a pool, I suggest to care about same ashift. When you try to remove a regular vdev (ex basic, mirror) from a pool and vdev is different then it stops with a message that this cannot be done due different ashift settings (but no crash like with special vdevs).

 

[gea]

Just to complete for those using ZoL
https://github.com/zfsonlinux/zfs/issues/9363

 

[gea]

Some more insights about the new ZFS Allocation Classes feature

http://napp-it.org/doc/downloads/special-vdev.pdf

1. About Allocation Classes
2. Performance of a slow diskbased pool
3. With special vdev (metadata only)
4. With special vdev (for a single filesystem)
5. With special vdev (for a single filesystem) and Slog (Optane)
6. Performance of a fast diskbased pool
7. Fast diskbased pool vwith special vdev
8. NVMe Pool vs special vdev (same NVMe)
9. Compare Results
10. Conclusion
11. When is a special vdev helpful
12. When not
13. General suggestions

 

[gea]

OmniOS 151032 stable is out
This is the most feature rich update for Open-ZFS and OmniOS ever.
download: https://omniosce.org/download.html

Release notes

omniosorg/omnios-build

Build system for OmniOS. Contribute to omniosorg/omnios-build development by creating an account on GitHub.

github.com

Update

http://www.napp-it.org/doc/downloads/setup_napp-it_os.pdf

New Open-ZFS features:
- native ZFS encryption
- raw zfs send of locked and encrypted filesystems
- sequential/sorted resilver (can massively reduce resilver/scrub time
- manual and auto trim for the SSDs/NVMes in a pool
- Allocation classes for metadata, dedup and small io (mixed pool from disk/SSD/NVMe)
see https://www.napp-it.org/doc/downloads/special-vdev.pdf
a warning at this point: a zpool remove of a special vdev with a different ashift than the pool crashes Illumos/ZoL
- force ashift on zpool create/add

OmniOS related
-updated NVMe driver (with basic support for NVMe/U.2 hotplug)
-updates for newer hardware
-installer supports UEFI boot

- SMB 3.02 (kernelbased SMB) with many new features
see https://github.com/illumos/illumos-gate/pulls?q=is:pr+SMB3
- improvement for LX/ Linux zones, newer Linux distributions
- improvements for Bhyve
- improvements to the Enlightened Hyper-V drivers for running under Hyper-V or Microsoft Azure.

Napp-it 19.dev/19.h2.x supports the new features

 

[thecoolnessrune]

Thanks as always for your updates gea! I'm still running Napp-It Pro Home+Backup from back in ESXi 5.5. I've got to get on 6.7 and get the newest All-in-One set up.

 

[gea]

napp-it v20.x

I have uploaded a pre version of next napp-it v19.12 noncommercial homeuse
and napp-it 20.01 pro to support the newest features of Oracle Solaris and expecially OmniOS/OpenIndiana.

- ZFS encryption with web/filebased keys, a http/https keyserver with an HA option,
keysplit for two locations, automount after reboot and user lock/unlock via SMB
http://napp-it.org/doc/downloads/zfs_encryption.pdf

- special vdevs

https://www.napp-it.org/doc/downloads/special-vdev.pdf

- trim

- force ashift when adding a vdev

- protection against accidentially adding a basic vdev to a pool

more (all features from 19.dev)

napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana and Solaris Changelog

napp-it.org - ZFS NAS SAN Server with user editable web-gui // All In One server = virtualized ready to run ZFS-Server

napp-it.org

 

[gea]

OmniOS has added minIO (client, server and rclone) to their extra repository. This is an Amazon S3 compatible cloud backup service.

Installation and management is very fast and easy, see

Amazon S3 compatible ZFS cloud with minIO

Due a customer/my request, OmniOS added minIO client and server and rclone to the extra repo Package Catalog MinIO | Enterprise Grade, High Performance Object Storage MinIO | MinIO Client Quickstart Guide MinIO | The MinIO Quickstart Guide MinIO Enjoying Role in Emerging Cloud Architecture...

forums.servethehome.com

 

[gea]

Info
I have added minIO support to napp-it

MinIO is an S3 compatible opensource cloud service, ultrafast and
scalable nearly without limits and really easy to setup and maintain
without the complexity of other cloud services. Basically the whole
miniIO S3 server is a single file. Not the complexity of the solutions
based on an Apache AMP stack.

With minIO you can share a ZFS filesystem locally via NFS/SMB
and additionally for the Internet via S3 on a per ZFS filesystem base.

Setup see https://forums.servethehome.com/ind...atible-client-server-minio.27524/#post-257208

 

[gea]

OmniOS 151032w is there
omniosorg/omnios-build

to fix a bug with last OpenSSL 1.1.1e that hinders TLS email to work
Topicbox

 

[gea]

I have just added support for Apple Time Machine via SMB into napp-it up from 19.12 (OmniOS, OI, kernelbased/ Solaris SMB server, see menu Services > Bonjour)

 

[gea]

There is a new stable of OmniOS (free Solaris fork) v. 151034

New features since 151032 stable

https://github.com/omniosorg/omnios-build/blob/r151034/doc/ReleaseNotes.md