ESXi + NexentaStor Community build log

[Viper GTS]

Some of you might find this interesting, I've got most of the parts now & will be doing the initial build this weekend. I think some critical parts may be straggling a little so I probably won't have it fully running but I'll post pictures as I go & benchmarks when I'm done.

I've had an ESXi machine at home for a couple years now. It's a Q6600 with 16 GB of RAM, a few dual port Intel Gigabit NICs for networking, & two PERC 5 adapters (one internal, one external). Externally I've got 8x 7200 RPM SATA drives, internally 5 more for media storage & random stuff. Since this was an older platform it won't do VT-D. This necessitated storing my 500 GB music collection in VMDK which I wasn't totally thrilled with.

A few weeks back when a hot deal popped up for Samsung Spinpoint F3 1 TB's I picked up 8 to upgrade my storage array. This quickly snowballed into an entirely new system.

Parts list:

Motherboard, CPU, & RAM
TYAN S7025WAGM2NR
2 x Xeon E5606
6 x 4 GB DDR3 UDIMM

Chassis & Power
Norco RPC-4224 4U chassis
Corsair AX1200 PSU
Noctua fans for everything (fan wall, exhaust, & CPUs)

Drives
16 x Spinpoint F3 for NAS/SAN duty
8 x 320 GB Seagate (re-used)

Storage controllers:
2 x IBM BR10i for passthrough to Nexenta VM

Networking
Intel EXPI9404PTL quad port PCI-e NIC
Dell 2748 switch

Plan is to use the existing 8x320 GB on a PERC 5i for direct attached VMFS storage. I'll use that to host a NexentaStor Community VM with 4 GB of RAM and a couple cores. I'll be running mirrored with hotspares, so either 5 zpools using 3 way mirror with one disk as a hotspare or 7 zpools using 2 way mirror with two hotspares.

For the immediate future I'll use it as an integrated box with Nexenta just for bulk storage. If it works well I may eventually turn it into a dedicated storage box & move ESXi out into its own box again (with newer hardware).

Viper GTS

 

[Emulex]

let me know how it works out. i've got 6 RE4 2TB in raid-10 with BBWC and it sucks butt compared to the sas drives big time. latency goes through the roof when doing a storage migration/backups (write cache enabled on drives). not really usable for running sql servers but okay for development/desktop virtualization where you can afford latency. I might try expanding the drive count to 8 to see if that helps but i think the whole 7200rpm (versus 15K SAS) there is more to it than just double the rpm.

I believe it might be possible to crash the esxi server if i did a few vmotion and backups and 4-5 vm's at once.

 

[Viper GTS]

This will not be used for VM storage, it's just bulk storage (music/movies type stuff).

At some point I may experiment with an iSCSI target and/or NFS shares to see what the performance is like but for now no VMs will be stored there. That's what the 8 drive RAID 10 set is for (and I've had no issues with that, as this is just for home tinkering).

Viper GTS