That's a cool case.
I'm probably due to upgrade my storage myself, and also add bigger drives to my cold backup rotation pool. I also want to get off NFS and go iSCSI but I need a cluster aware file system if I want to do VMs as I want to do a multi host setup. I want to look at Proxmox, it may even be able to handle that stuff on it's own. What I'd actually like to do is 1 lun per VM, and the lun is presented directly to the VM at the "hardware" level as a disk. That would probably cut back on lot of overhead by not even needing to map them to the host, they'd be accessed directly. Kinda like if the VM had an iSCSI card installed. Not sure if there are free/open solutions that can do that though.
My NFS is retarded slow, I never figured out why, but I'm thinking my best bet is to just get rid of it and do block storage over a separate network. Ex: doing a real SAN setup.
I actually have a VM that keeps acting like it has a hard drive failure, it's weird. IO timeouts, and process crashes etc. Like I even get ata2 status link errors etc. Normally that means a failing hard drive... but it's a VM. I think it's because my NFS is just so damn slow. If there's too much load it just crawls.