Best way to configure this setup - VM & fileserver (hardware in hand)

[JimPhreak]

Couldn't find a better section to post this in so here goes.

Ok so in my possession I have the following:

Box #1 (current fileserver):
Intel Q6600 CPU
DFI Blood Iron P35 LGA775 board
8GB DDR2-667 RAM
160GB SATA OS Drive
RAID10 Array (4x2TB drives)
- currently configured on a Dell Perc 6i and storing all my files
4x2TB brand new drives (same drives as in my current array) not installed yet


Box #2 (planned VM server):
Intel i7 3770K CPU (got this to OC a bit as I transcode multiple files at a time with Plex but could trade for a 3770 if VT-D is that important)
Asus P8Z77-V ATX board
32GB DDR3-1600 RAM
500GB Sata Drive (plan was to partition for OS and less used VM's)
2x128GB Crucial M4 SSD's (for frequently used VM's)
Intel EXPI9301CT PCIe x1 Gigabit Adapter


I bought Box #2 with the intention of moving some services (Plex Media Server, FTP, DNS) to VM's and to also use it as a testing environment for both VMware ESXi 5.1 and Hyper-V as I'm looking to learn both for work related purposes. I then planned to convert Box #1 to a standalone fileserver/NAS. I was planning on just putting Win7 on this box as my base OS and running all my VM's through VMware workstation so Plex can run on Win7 and have all 8 cores (with HT obviously) available to it. However I'm not sure if that's the best option here or if it'd be better to just put ESXi 5.1/Hyper-V as my base and run all VM's from that.

I'm looking for critiques, suggestions, ideas for how best to go about achieving my goals here. I'm not sure what to do with Box #1 whether I should re-install a new OS (currently running WS 2008 R2) and use something like FreeNAS or just stick with Windows with wintarget as I'm really only familiar with Windows OS's. I want this file server to be accessible to a few of the VM's on Box #2 via iSCSI (or NFS?) so I can use it for Plex and for imaging/snapshots/backups.

 

[mfenn]

I think that your "production" (Plex, backups, etc.) requirement is somewhat in conflict with your testing (ESXi, Hyper-V) requirement. Production says to go for ESXi/Hyper-V directly on Box #2 whereas testing says to put Windows + VMWare Workstation on Box #2. ESXi/Hyper-V gives you a more stable/optimized environment for VMs, but Win7 and Workstation is more flexible for running nested hypervisors. You can of course run any non-hypervisor testing you want on a pure ESXi/Hyper-V setup. Only you can decide which aspect is more important to you.

As for storage, you really only have a couple of (sane) options with a pure ESXi server:

1. Use the Windows iSCSI target to export iSCSI to ESXi and format that with VMFS. This will work totally fine, and should perform pretty well, but you lose external visibility into the VM folders (perhaps not a big deal). Windows VMs are served via SMB same as they are now.

2. Reinstall your file server with some UNIX system (FreeNAS) is fine, so that you can use NFS exports to the ESXi server. This will let you see what's going on with the VMs more closely and also let you make more sane backups since you only have to back up the files and not the whole volume. You can still do SMB sharing to Windows from FreeNAS.

 

[JimPhreak]

I think that your "production" (Plex, backups, etc.) requirement is somewhat in conflict with your testing (ESXi, Hyper-V) requirement. Production says to go for ESXi/Hyper-V directly on Box #2 whereas testing says to put Windows + VMWare Workstation on Box #2. ESXi/Hyper-V gives you a more stable/optimized environment for VMs, but Win7 and Workstation is more flexible for running nested hypervisors. You can of course run any non-hypervisor testing you want on a pure ESXi/Hyper-V setup. Only you can decide which aspect is more important to you.

As for storage, you really only have a couple of (sane) options with a pure ESXi server:

1. Use the Windows iSCSI target to export iSCSI to ESXi and format that with VMFS. This will work totally fine, and should perform pretty well, but you lose external visibility into the VM folders (perhaps not a big deal). Windows VMs are served via SMB same as they are now.

2. Reinstall your file server with some UNIX system (FreeNAS) is fine, so that you can use NFS exports to the ESXi server. This will let you see what's going on with the VMs more closely and also let you make more sane backups since you only have to back up the files and not the whole volume. You can still do SMB sharing to Windows from FreeNAS.

I have been going back and forth as to which route to go in terms of just putting ESXi on the server and going from there or putting Win7 on and using VMware workstation. I'm leaning towards putting ESXi on because I can always nest another ESXi server (and Hyper-V for that matter) within my host ESXi server for the testing environment. My main motivation for going with Win7 is that Plex can at times require a lot of CPU power and if I put that on a VM (instead of on Win7 itself) it won't leave many CPU cores left for the rest of my VM's.

As for my fileserver, remember I won't be loading/running my VM's off this server as those VM's will be stored locally on my VM server. I just want to use it to serve my media files to Plex as well as to do backups of my VM's/and my main PC.

 

[mfenn]

I can always nest another ESXi server (and Hyper-V for that matter) within my host ESXi server for the testing environment.

Sort of. You need to be able to use VM version 9 to do nested ESXi at fully performance. Unfortunately, that is not available through the old (now deprecated) Windows vSphere client, you need the vSphere Web Client, which only comes with vCenter. If you have a vCenter license (the basic kit is not too expensive <$1000) then you are golden. Otherwise you are stuck with low performance for nested hypervisors. VMware Workstation can do nested hypervisors at full speed (go go market segmentation).

My main motivation for going with Win7 is that Plex can at times require a lot of CPU power and if I put that on a VM (instead of on Win7 itself) it won't leave many CPU cores left for the rest of my VM's.

CPU performance is of zero concern with ESXi. If it's not fast enough within ESXi, it won't be fast enough natively either.

I think you are a little confused about CPU allocation among VMs. Just because you create a Plex VM with 8 CPUs doesn't mean that you can't also assign CPUs to other VMs. That's the whole point of virtualization!

Note that you would be doing the same thing if you ran Plex natively on the host, it would be implicitly sharing CPU cycles with the VMs.

As for my fileserver, remember I won't be loading/running my VM's off this server as those VM's will be stored locally on my VM server. I just want to use it to serve my media files to Plex as well as to do backups of my VM's/and my main PC.

You're not really getting into the possibilities opened up by virtualization until you use shared storage IMHO. I would put my VM storage on the file server and only use the local disk for ESXi 5.1 SSD caching. It will work either way though.

 

[JimPhreak]

Sort of. You need to be able to use VM version 9 to do nested ESXi at fully performance. Unfortunately, that is not available through the old (now deprecated) Windows vSphere client, you need the vSphere Web Client, which only comes with vCenter. If you have a vCenter license (the basic kit is not too expensive <$1000) then you are golden. Otherwise you are stuck with low performance for nested hypervisors. VMware Workstation can do nested hypervisors at full speed (go go market segmentation).

I have VMware workstation 9 already and I can get vCenter Server for pretty cheap through my work.


CPU performance is of zero concern with ESXi. If it's not fast enough within ESXi, it won't be fast enough natively either.

I think you are a little confused about CPU allocation among VMs. Just because you create a Plex VM with 8 CPUs doesn't mean that you can't also assign CPUs to other VMs. That's the whole point of virtualization!

Note that you would be doing the same thing if you ran Plex natively on the host, it would be implicitly sharing CPU cycles with the VMs.

Thank you for clearing that up, you're right I was confused. With that in mind I think running everything with in ESXi 5.1 is my best option for both learning and production.


You're not really getting into the possibilities opened up by virtualization until you use shared storage IMHO. I would put my VM storage on the file server and only use the local disk for ESXi 5.1 SSD caching. It will work either way though.

By shared storage you mean what exactly? Running my VM's off a server instead of locally? What kinds of options do I have doing it that way as opposed to just running my VM's of the local server? I kind of bought my 2 SSD's specifically for the speed at which I'd be able to load/run my VM's locally but if you're saying I'd have more options doing it another way I'm all ears?

Responses in bold.

 

[mfenn]

By shared storage you mean what exactly? Running my VM's off a server instead of locally? What kinds of options do I have doing it that way as opposed to just running my VM's of the local server? I kind of bought my 2 SSD's specifically for the speed at which I'd be able to load/run my VM's locally but if you're saying I'd have more options doing it another way I'm all ears?

Yeah, by shared storage I mean putting your VM files on Box #1 and running them on Box #2. You can use iSCSI or NFS to present the storage to Box #2.

Admittedly, this doesn't get you too much benefit for your home production use, but it is really useful for learning/testing. Two reasons for this:

1. You don't have to worry about anything that's local on the hypervisor. You can blow it away and reconfigure it as many times as you want and be back up and running in minutes.

2. If you get a 3rd box (event a weak one) and put ESXi on it, you can put Boxes #2 and #3 in a VMware cluster. This allows you to play with the more advanced features of VMware like, HA, Fault Tolerance, vMotion, and DRS. While these don't really have an immediate benefit for your home setup, they are a great thing to learn about for work and career development purposes.

 

[JimPhreak]

Yeah, by shared storage I mean putting your VM files on Box #1 and running them on Box #2. You can use iSCSI or NFS to present the storage to Box #2.

Admittedly, this doesn't get you too much benefit for your home production use, but it is really useful for learning/testing. Two reasons for this:

1. You don't have to worry about anything that's local on the hypervisor. You can blow it away and reconfigure it as many times as you want and be back up and running in minutes.

2. If you get a 3rd box (event a weak one) and put ESXi on it, you can put Boxes #2 and #3 in a VMware cluster. This allows you to play with the more advanced features of VMware like, HA, Fault Tolerance, vMotion, and DRS. While these don't really have an immediate benefit for your home setup, they are a great thing to learn about for work and career development purposes.

Well that's partly why I'm considering running everything through VMware Workstation 9 since I can nest 2 ESXi servers from within it.

 

[mfenn]

Well that's partly why I'm considering running everything through VMware Workstation 9 since I can nest 2 ESXi servers from within it.

That's true. You could also nest ESXi within ESXi since you can get vCenter (which you'd need anyway for any advanced features).

 

[JimPhreak]

That's true. You could also nest ESXi within ESXi since you can get vCenter (which you'd need anyway for any advanced features).

You need to nest ESXi within itself to use vCenter? I thought you just needed two ESXi servers in total.

 

[mfenn]

You need to nest ESXi within itself to use vCenter? I thought you just needed two ESXi servers in total.

No, you can use vCenter with even just one ESXi hypervisor. You just won't be able to take advantage of most of the features since they require two hypervisors to function.

What I'm saying is the other way around. You need:

1. vCenter to allow you to use the vSphere Web Client.
2. vCenter Web Client to allow you to create Version 9 VMs.
3. Version 9 VM's to allow you to nest ESXi.

Clear as mud, I know.

 

[JimPhreak]

No, you can use vCenter with even just one ESXi hypervisor. You just won't be able to take advantage of most of the features since they require two hypervisors to function.

What I'm saying is the other way around. You need:

1. vCenter to allow you to use the vSphere Web Client.
2. vCenter Web Client to allow you to create Version 9 VMs.
3. Version 9 VM's to allow you to nest ESXi.

Clear as mud, I know.

Haha gotcha. Well the good thing is that I have licensing for 2 ESXi 5.1 servers, vCenter, and VMware Workstation 9 so I should be able to pull it off .