SAS v. SATA

[destrekor]

So I'm putting together my plan/wish-list of components for a ESXi/NAS box.

Yes, I plan on putting FreeNAS -- and likely Sofos UTM or XG -- on ESXi.

But now I'm at the point of deciding which disks are best for this. I'll likely use RAIDZ2 (similar to RAID-6). Should I just go with cheaper 3 or 4TB SATA disks, or is it worth while going for SAS?

For instance, take this drive: http://www.amazon.com/Seagate-Enterprise-Capacity-Constellation-ST4000NM0023/dp/B00A45JFJS.

There are both a SATA and SAS variant. Is there going to be any benefit going one route as opposed to the other?

The plan is to have these connected to a hot-swap backplane, and connect that to either an add-on HBA or to the SAS connectors on a motherboard that includes an LSI HBA.

 

[frowertr]

So the price is nearly identical. So I say that as long as you have a SAS backplane and either a SAS connector on the MB or a SAS HBA is there isn't really any reason to use SATA.

The drives themselves are identical but with SAS you get deeper que depths, full duplex, multipath, etc... You won't get much if any increase in speed between that drive with SAS or SATA connectors but the SAS protocol is much more "robust" than SATA.

Edit: Make sure those are native 512n sector format drives. ESXi doesn't support 512e nor 4K.

 

[dave_the_nerd]

So the price is nearly identical. So I say that as long as you have a SAS backplane and either a SAS connector on the MB or a SAS HBA is there isn't really any reason to use SATA.

The drives themselves are identical but with SAS you get deeper que depths, full duplex, multipath, etc... You won't get much if any increase in speed between that drive with SAS or SATA connectors but the SAS protocol is much more "robust" than SATA.

Edit: Make sure those are native 512n sector format drives. ESXi doesn't support 512e nor 4K.

If you're running FreeNAS inside a VM on ESX, you just pass the HBA through to the VM - ESX never sees the drives.

 

[destrekor]

If you're running FreeNAS inside a VM on ESX, you just pass the HBA through to the VM - ESX never sees the drives.

So I guess that will be good to keep in mind for OS drives, but yeah, for FreeNAS, the storage and HBA will be passed through entirely untouched by the hypervisor. FreeNAS needs raw access, no middlemen.

 

[dave_the_nerd]

So I guess that will be good to keep in mind for OS drives, but yeah, for FreeNAS, the storage and HBA will be passed through entirely untouched by the hypervisor. FreeNAS needs raw access, no middlemen.

ESX fits on a thumb drive.

Actually, FreeNAS does too... although that's messy. Probably should get a proper small RAID-1 for ESX boot and a small datastore, stick the FreeNAS vmdk and any other mission-critical VMs (dhcp server, pfsense server, etc.) on that, and then let FreeNAS serve up iSCSI LUNs back to your ESX box for all the other fun stuff.

 

[destrekor]

So the price is nearly identical. So I say that as long as you have a SAS backplane and either a SAS connector on the MB or a SAS HBA is there isn't really any reason to use SATA.

The drives themselves are identical but with SAS you get deeper que depths, full duplex, multipath, etc... You won't get much if any increase in speed between that drive with SAS or SATA connectors but the SAS protocol is much more "robust" than SATA.

Edit: Make sure those are native 512n sector format drives. ESXi doesn't support 512e nor 4K.

Thanks.

At that point, is most of the benefit simply in the better grade HDD? I was previously set on the WD Red models, I hear wonderful things about them. So I guess it basically boils down to this:
is it worth paying a premium, in comparison to WD Red SATA, to get WD Re or Seagate Constellation ES.3 with SAS connection?

I know the word "value" is ambiguous, as what is a value to one person is an extravagant cost to another person that doesn't bring enough value. I like the idea of data integrity and minimizing the chances of drive failure, but not necessarily if that means paying 2x as much for a drive.

And with the drive densities that they are now, where does SAS and ECC memory and ZFS come into play in relation to non-recoverable read errors? I guess that is where it all comes down to: besides a drive failing, I am trying to minimize the chances of array failure if there is ever a URE on one of the disks. I think with RAIDZ2 I would be afforded two additional errors when rebuilding the array before I lost the array entirely. Do I have that right?

 

[frowertr]

If you're running FreeNAS inside a VM on ESX, you just pass the HBA through to the VM - ESX never sees the drives.

I am unfamiliar with FreeNas. Are you still not creating a datastore inside of ESXi and then storing and running that FreeNas VM from that datastore?

 

[frowertr]

Thanks.

At that point, is most of the benefit simply in the better grade HDD? I was previously set on the WD Red models, I hear wonderful things about them. So I guess it basically boils down to this:
is it worth paying a premium, in comparison to WD Red SATA, to get WD Re or Seagate Constellation ES.3 with SAS connection?

I know the word "value" is ambiguous, as what is a value to one person is an extravagant cost to another person that doesn't bring enough value. I like the idea of data integrity and minimizing the chances of drive failure, but not necessarily if that means paying 2x as much for a drive.

And with the drive densities that they are now, where does SAS and ECC memory and ZFS come into play in relation to non-recoverable read errors? I guess that is where it all comes down to: besides a drive failing, I am trying to minimize the chances of array failure if there is ever a URE on one of the disks. I think with RAIDZ2 I would be afforded two additional errors when rebuilding the array before I lost the array entirely. Do I have that right?

It depends on what you are doing with this NAS. If it's a simple media server NAS that hosts movies, music, etc... you won't notice a different between this drive or that drive so long as it's designed for 24/7 uptime. I prefer the HGST Ultrastars drives myself but there is nothing wrong with WD Red or WD RE or Seagate Constellations. I'd still stick with 7200 drives though for sure. If you are buying a drive with a SAS interface you can already be assured it's designed for Enterprise use. I'm not aware of any drive with a SAS interface that isn't Nearline/Enterprise. SATA not so much like the WD Greens. Just make sure it's Nearline which every SAS 3.5" drive is.

As far as URE, any parity array is at risk. Sure, RAID 6 mitigates it by writing two parity stripes. But you incure double the write penalty by doing that when compared with RAID 5.

My stance on RAID when using spinning rust is pretty simple. Unless there is a real need for RAID 6 (like squeezing out every last bit of drive capacity in an array) use RAID 10 instead. RAID 10 and RAID 1 are immune to URE as there is no resilvering going on when a drive fails, just remirroring.

I'm not familiar with RAIDz so I'm not sure how it works. It sounds like RAID 6 in your description which gives you two drive failures max.

 

[destrekor]

ESX fits on a thumb drive.

Actually, FreeNAS does too... although that's messy. Probably should get a proper small RAID-1 for ESX boot and a small datastore, stick the FreeNAS vmdk and any other mission-critical VMs (dhcp server, pfsense server, etc.) on that, and then let FreeNAS serve up iSCSI LUNs back to your ESX box for all the other fun stuff.

So, that would be all boot volumes on the RAID1 boot array, and then all data volumes served up from the storage array?

I've never worked with iSCSI... would I designate a certain amount of space from the array and set an iSCSI target? Would it be carved out from the array once it is built but before it is formatted as a volume? So that the ZFS volume is one volume on the array, alongside any other volume(s) set aside for iSCSI?

 

[dave_the_nerd]

I am unfamiliar with FreeNas. Are you still not creating a datastore inside of ESXi and then storing and running that FreeNas VM from that datastore?

You are - at least for the FreeNAS Boot volume. And you _can_ feed it a bunch of VMDKs to manage. But FreeNAS wants very, very badly to manage its own bare-metal hard drives. So it's much, much better if you have ESX hand FreeNAS its own hardware to play with.

(This is generally true for any NAS OS that's doing something that looks more or less like software RAID.)

That's not to say that this is considered a good idea.

https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/

It's just possible, and it makes it doable to set up all-in-one-box-at-home system.

If I were doing a new home server tomorrow, I'd probably install Linux bare metal, use ZFS-On-Linux to manage my storage, and use KVM (not ESX) as my hypervisor for any VM needs. But many people have work- or professional-development-related reasons to use certain platforms, so I'm not really in a position to judge.

 

[destrekor]

I am unfamiliar with FreeNas. Are you still not creating a datastore inside of ESXi and then storing and running that FreeNas VM from that datastore?

For the FreeNAS OS? Yes, it would be inside of an ESXi datastore.

But the actual storage array managed with FreeNAS will be untouched by the ESXi host. FreeNAS requires absolute control of the entire storage chain, so generally the idea is to attach the disks to an RAID/HBA controller, and in ESXi enable passthrough, which prevents ESXi from interacting with whatever hardware is passed through. So whichever VM you designate as the one which receives something by means of passthrough, is the only one that can directly, bare-metal, interact with the controller.

 

[MongGrel]

Have had the capability to use SAS for a long time on the main rig here on the mobo for a couple but something I just have never done.

You're doing a separate NAS, but I'm just curious if it makes a big difference really, SAS is probably not something I'd do myself as have a hardware RAID controller I stuck in it all ready.

I'm just not a huge Seagate fan to begin with myself personally, from the past.

Still have 4 old WD RE3's in RAID10 on my array from long ago on one LUN, but yeah you're doing things I do not have.

 

[dave_the_nerd]

So, that would be all boot volumes on the RAID1 boot array, and then all data volumes served up from the storage array?

Yeah, pretty much. Not necessary ALL the boot volumes - depends what you do. Just know that if your FreeNAS VM takes a dive, any vmware data stores that live on FreeNAS-managed storage will disappear. (That's why I used the term 'mission-critical' earlier.)

I've never worked with iSCSI... would I designate a certain amount of space from the array and set an iSCSI target? Would it be carved out from the array once it is built but before it is formatted as a volume? So that the ZFS volume is one volume on the array, alongside any other volume(s) set aside for iSCSI?

Nah, iSCSI LUNs are thin-provisioned.

ZFS doesn't work as one volume in the array, it IS the array. From the FreeNAS side of things, each iSCSI LUN just looks like a (very big) file. The iSCSI LUNs are created after your storage is configured/managed by FreeNAS.

You could probably just use NFS datastores, actually - that'd be simpler. iSCSI is theoretically better in terms of latency, but with everything in the same box, that probably doesn't matter.

 

[MongGrel]

It depends on what you are doing with this NAS. If it's a simple media server NAS that hosts movies, music, etc... you won't notice a different between this drive or that drive so long as it's designed for 24/7 uptime. I prefer the HGST Ultrastars drives myself but there is nothing wrong with WD Red or WD RE or Seagate Constellations. I'd still stick with 7200 drives though for sure. If you are buying a drive with a SAS interface you can already be assured it's designed for Enterprise use. I'm not aware of any drive with a SAS interface that isn't Nearline/Enterprise. SATA not so much like the WD Greens. Just make sure it's Nearline which every SAS 3.5" drive is.

As far as URE, any parity array is at risk. Sure, RAID 6 mitigates it by writing two parity stripes. But you incure double the write penalty by doing that when compared with RAID 5.

My stance on RAID when using spinning rust is pretty simple. Unless there is a real need for RAID 6 (like squeezing out every last bit of drive capacity in an array) use RAID 10 instead. RAID 10 and RAID 1 are immune to URE as there is no resilvering going on when a drive fails, just remirroring.

I'm not familiar with RAIDz so I'm not sure how it works. It sounds like RAID 6 in your description which gives you two drive failures max.

+1

But basically, when you go into using a seperate NAS setup, is not really something I have ever done, so I'll just read for now.

The internal one on the main is probably similar to one.

 

[frowertr]

Ok, I see what your saying about the storage array being unseen and untouched by ESXi, but I would think you have snapshot problems on the actually FreeNas boot volume if your hosting it on a non native 512n sector format drive. At least according to this kb article: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2091600

It may or may not be a problem. I don't have any first hand knowledge with using AF drives but I just wanted you to be aware. It may work just fine.

 

[destrekor]

You are - at least for the FreeNAS Boot volume. And you _can_ feed it a bunch of VMDKs to manage. But FreeNAS wants very, very badly to manage its own bare-metal hard drives. So it's much, much better if you have ESX hand FreeNAS its own hardware to play with.

(This is generally true for any NAS OS that's doing something that looks more or less like software RAID.)

That's not to say that this is considered a good idea.

https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/

It's just possible, and it makes it doable to set up all-in-one-box-at-home system.

If I were doing a new home server tomorrow, I'd probably install Linux bare metal, use ZFS-On-Linux to manage my storage, and use KVM (not ESX) as my hypervisor for any VM needs. But many people have work- or professional-development-related reasons to use certain platforms, so I'm not really in a position to judge.

If this were a production business environment, I would certainly not pursue this route, it's admittedly asking for some trouble at some point down the line. But I generally like the challenge of tinkering, and, hopefully, as long as data is preserved, it won't be any big deal to have the system offline for however long it takes to repair the hardware, or reconfigure the software after some cataclysmic screw-up. But, I wouldn't do that. Nope.

I still do have to investigate the different hypervisors out there. I'd be worried that performance on KVM might be less than a true dedicated hypervisor, but I could be wrong. Going with a minimal headless Linux install may keep free the resources that would make a difference.

With ZFS On Linux, I imagine you don't get the plugin compatibility that FreeNAS has, right? I've even considered getting the community-ported Synology OS for the additional plugin capability. I'll be running Plex Media Server and likely a few other neat media tools. I just gravitated toward FreeNAS because I've heard so much about it and know that it is supported. That, and I've read about people getting the HDHomeRun DVR recording engine (alpha/beta) up and running in FreeNAS, which is the project that got me looking to fast track my NAS dreams. Instead of a few years down the road, likely after my next gaming PC rebuild, it's now on the list to get done this year, earlier rather than later, and other little projects are down the line.

 

[destrekor]

Ok, I see what your saying about the storage array being unseen and untouched by ESXi, but I would think you have snapshot problems on the actually FreeNas boot volume if your hosting it on a non native 512n sector format drive. At least according to this kb article: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2091600

It may or may not be a problem. I don't have any first hand knowledge with using AF drives but I just wanted you to be aware. It may work just fine.

Well as I said, if that is the case, I would definitely use 512n disks for the ESXi boot volume(s). Only for the storage array would any other sector size be used.

 

[frowertr]

Well as I said, if that is the case, I would definitely use 512n disks for the ESXi boot volume(s). Only for the storage array would any other sector size be used.

That is probably the best idea. Removes a potential issue.

You mentioned you are still shopping hypervisors. I'm a VMware shop now but if I had to do it over again I go with Xen hands down. It's totally free and comes loaded with features. Check it out with Xen Orchestra.

 

[Burner27]

ESXi isnt free is it?

 

[dave_the_nerd]

With ZFS On Linux, I imagine you don't get the plugin compatibility that FreeNAS has, right? I've even considered getting the community-ported Synology OS for the additional plugin capability. I'll be running Plex Media Server and likely a few other neat media tools. I just gravitated toward FreeNAS because I've heard so much about it and know that it is supported. That, and I've read about people getting the HDHomeRun DVR recording engine (alpha/beta) up and running in FreeNAS, which is the project that got me looking to fast track my NAS dreams. Instead of a few years down the road, likely after my next gaming PC rebuild, it's now on the list to get done this year, earlier rather than later, and other little projects are down the line.

I tinkered with them a bit when I still used FreeNAS. That plugin stuff doesn't really have anything to do with the ZFS file system - it's just a pre-canned and pre-configured FreeBSD Jail that you download/install. Theoretically, they're idiot-proof, since you're just installing a complete working environment that somebody else set up for you. It's a very simple GUI-ey way to install an application, and using Jails should hopefully keep them all playing in their own sandboxes where they can't conflict.

But it doesn't always work that way. I had a lot of trouble getting the Crashplan "plugin" to work (and stay working) back when, and I wasn't the only one - some differences in Linux vs. FreeBSD java implementations I guess. Same story with MineCraft.

Also, you're ultimately limited in a FreeNAS Jail/Plugin environment, to whatever applications will run on FreeBSD. (Although there is a VirtualBox FreeNAS Plugin now, which would give you a working hypervisor too - but that's a full type 2 hypervisor, and performance is probably going to be worse than ESX.)

VMs give you more flexibility, with multiple possible guest operating systems and so on. But yeah, the downside is you have to stand up and configure a full blown guest OS for every VM. (Or use a template.) After that, though, installing something like Plex is pretty straightforward - "apt-get install plexmediaserver -y & service plexmediaserver start" and you're off to the races.

 

[dave_the_nerd]

ESXi isnt free is it?

ESX isn't free. ESXi actually is (you have to register with vmware, but then you get a serial number). For a single hypervisor. Most of the "fun" features are disabled.

Explains: http://www.vladan.fr/esxi-free-vs-paid/

 

[dave_the_nerd]

Oh, and FreeNAS can use bhyve now too. That's cool. Haven't tried it yet.

 

[destrekor]

ESX isn't free. ESXi actually is (you have to register with vmware, but then you get a serial number). For a single hypervisor. Most of the "fun" features are disabled.

Explains: http://www.vladan.fr/esxi-free-vs-paid/

What, exactly, will be missing when using the free version of ESXi in a home environment? It sounds like snapshots are out, but, should I really be worried about that? I'm not planning on running a test lab where I'm testing patches or new versions and worried about rolling back.

Beyond the obvious ease of use, is there a significant benefit to snapshots that other backup tools within each guest couldn't approach?

The way I figure, I'll have a router/firewall VM, FreeNAS, and depending on what Xeon D-1500 package I end up getting, perhaps one or two VMs just to screw around with different things that I may fancy at some point. Perhaps some kind of communications server, but who knows. To me, that doesn't sound like a setup that really demands snapshot capability.

But are there other features that may be somewhat important?

I'm trying to research what management software is available for the free edition. I'm slightly accustomed to the Web Client, and it sounds like that is often preferred? I mean, I'll always have a Windows box, but I kind of shuffle back and forth between Windows and OS X being the primary boot OS. I hate having to boot into a different OS to simply do one single thing and then boot back into the other one. The vSphere Client (desktop?) appears to only be available in Windows. Is the Web Client not available for the free edition?

And in that, I'd say I'm not tied down to the idea of using ESXi out of any particular need. I've heard many prefer it over Xen when it comes to true Type 1 hypervisors, and I've generally wanted to stick to a Type 1 for this kind of task. That, and I figure, learning ESXi management may be better for any long-term career plans than KVM. But I also don't want to restrict myself for that purpose only and have to bang my head against a wall when a better and easier solution exists elsewhere. Similar principals can be applied to all the hypervisors.

However, while I want to get more CLI time, I also would very, very much like the web client approach for management of my servers. FreeNAS's client looks terrific, and if I can manage everything through web clients, I certainly will. Sometimes, I just like to do things the simple way to get it done with so I can move on. Some times I want to get my hands dirty and figure out the more fulfilling way to get something done.

So if using the ZFS on Linux approach means I lose the nice NAS web client, I'd be quite disappointed.

But I am definitely curious about the bhyve on FreeNAS/FreeBSD. I had never heard of bhyve until you mentioned it.

This is interesting: https://b3n.org/vmware-vs-bhyve-performance-comparison/

I tinkered with them a bit when I still used FreeNAS. That plugin stuff doesn't really have anything to do with the ZFS file system - it's just a pre-canned and pre-configured FreeBSD Jail that you download/install. Theoretically, they're idiot-proof, since you're just installing a complete working environment that somebody else set up for you. It's a very simple GUI-ey way to install an application, and using Jails should hopefully keep them all playing in their own sandboxes where they can't conflict.

But it doesn't always work that way. I had a lot of trouble getting the Crashplan "plugin" to work (and stay working) back when, and I wasn't the only one - some differences in Linux vs. FreeBSD java implementations I guess. Same story with MineCraft.

Also, you're ultimately limited in a FreeNAS Jail/Plugin environment, to whatever applications will run on FreeBSD. (Although there is a VirtualBox FreeNAS Plugin now, which would give you a working hypervisor too - but that's a full type 2 hypervisor, and performance is probably going to be worse than ESX.)

VMs give you more flexibility, with multiple possible guest operating systems and so on. But yeah, the downside is you have to stand up and configure a full blown guest OS for every VM. (Or use a template.) After that, though, installing something like Plex is pretty straightforward - "apt-get install plexmediaserver -y & service plexmediaserver start" and you're off to the races.

Yeah, I get that it is a canned Jails system for the plugins, but from what I've read, outside of the plugins, FreeNAS isn't a totally open FreeBSD environment, in that getting other Jails to work can be near impossible. I've only read lightly into it, but there's only so much I'm interested in running on FreeNAS directly... but that's mostly because I don't know what I don't know. I've read up on a lot of the different plugins available for the different NAS OS's, especially the likes of QNAP and Synology, and I haven't been interested in that much at all. But I bet I'll find something I desperately need to get working down the road, and it'd suck if it was an absolute pain in the ass when it ought not to be, had I just chose the right system.

But again, as a non-production system just for home server purposes, I can't say I'm terribly afraid of coming upon such an obstacle, even if it requires the need to completely scrap all of it and start with different OS's. I'd rather NOT do that, obviously, but hopefully by that point in time, I'll be able to get enough stand-alone large drives in other systems so I could backup any data on the array I really need/want to keep before I wipe and start anew.

 

[frowertr]

The web client is vCenter which is not free. It requires the minimum Essentials license. You can still manage with the regular Windows thick client and you do lose a bit of the newer VM features but it's not a deal breaker.

You can also manage with this: https://labs.vmware.com/flings/esxi-embedded-host-client

It's just in fling status but I think that will eventually replace the Windows thick client for single host management. There is no downside to picking Hyper-V, Xen, and vSphere for learning to further your career path. All are major players in the Enterprise world. Hell id try all three if I were you. They are all free in some form or fashion.

 

[XavierMace]

Realistically, you're not missing anything important (for a home lab) with the free version of ESXi.

You can use the free version of Veaam for VM level backups since you won't have snaps available.

I still feel obligated to point out as I always do, that I'm not a fan of running your NAS OS as a VM and then using that VM to provide storage for other VM's.

The only people I know who prefer the web client are the people not running Windows. The Windows client is substantially faster than the web client. Update Manager (not an option with the free version) also requires the Windows client.