Question Virtualization Machine - Intel or AMD?

[JayMX]

Hello I am looking to build a new computer which I want to use as a VM host - I use Windows Hyper-V (hypervisor) sometimes VirtualBox too. Doing some research (not extensive) I read that latest big/little Intel CPUs have issues with Virtualization in that p/e cores are assigned in a random manner (?) or not optimally, hence hampering performance of VMs. Is this true and if yes to what extent? Also read some people disabling e-cores but I wouldn't like to do that.

On the other hand AMD offers same type of high performance cores so theoretically there shouldn't be an issue - but never worked on AMD machine before so would appreciate any kind of tip or things to watch out.

I am looking at the latest offers from both, either Zen4 or Raptor Lake platforms preferably with DDR5 for the ECC support.

 

[Tech Junky]

I have an ADL setup and in this instance for bare metal VM I would go AMD.

Going with the latest AMD offers you DDR5 and raw performance. E cores on Intel are more applicable to a full os taking advantage of the lower clocks to conserve resources for background tasks.

 

[moinmoin]

How many VMs would you want to run concurrently and with how many processors each? And how much dependent on performance are these? With ECC is the on memory ECC sufficient, or do you want ECC capable boards, and is this a hard requirement?

The more the better fit is one of the higher end Ryzen processors (7900X and 7950X). With Raptor Lake you'd need to worry about the exact processor assignment if high performance is a goal, E cores simply won't reach the peak single thread performance that P or Zen cores have.

 

[TheELF]

AMD looses performance on all cores if all cores are loaded...intel only on the e-cores.
Partitioning the resources to the VMs is a user obligation, in both cases intel or amd you will have to assign a specific number of threads to each VM and make sure they run on the p-cores for intel* and on the same ccd for amd.
Having the cores of one VM split between two ccds on amd would be at least as bad as having threads on an e-core.

* unless you have a VM that needs less performance in which case you would want to send it to the e-cores.

 

[igor_kavinski]

What kind of performance do you want from your VMs? I think compute workloads can get around 95% of the host CPU core but if there is something that requires a lot of context switching (most notorious for that being databases), the I/O performance tanks considerably inside the VM.

This is a Samsung 980 Pro 1 TB SSD on an Ivy Bridge-E Xeon server:



Now look what happens to the 4K IOPS inside Hyper-V with the VHD stored on the same SSD:



IOPS suffer 42% loss.

Maybe this loss is much less on modern platforms.

Another thing to consider is how parallel your computing workloads are. Generally, AMD is really good at multitasking. With Intel, the foreground process will have the lion's share of the performance but everything in the background will run with much reduced performance.

 

[JayMX]

I am doing labs mostly with Microsoft products (Windows Server, Exchange, Windows clients, firewall, etc.) in VMs and ideally I 'd love to have the most cores possible. Sometimes I 've seen that some VMs (like a windows server with a number of 'heavy' roles) need more than one physical core assigned. Also ram capacity is important (DDR5 on-die ECC is fine) but that is something I can add any time so I think most important is the CPU. I am looking for at least 8 physical cores but if budget allows it would like to go more.

@TheELF you 're right about assigning the resources optimally and I assume it is not possible with a 12900K to assign the p and the e cores on specific VMs. On the AMD scenario I guess you are referring to the latency in between the CCDs but is this really equally bad to having an e core assigned to a machine you need CPU power?

@Tech Junky would you be so kind to add some more info from your experience on AL and Virtualization?

@igor_kavinski great info something I need to dig into as well - I now use a Storage Spaces with mechanical HDDs (RAID0) but in this new setup I was thinking either a single big capacity M2 or 2x M2 in raid0.

Thank you all for your replies.

 

[Tech Junky]

Another thing that's useful if you want to setup an array for the VM's using a PCIE card vs relying on the M2 ports on the MOBO you get away with cheaper costs on AMD because you can bifurcate the X16 slot whereas you have to use a PLX chip on Intel. Huge price difference where AMD could be $40 for 4 drives and Intel shoots to $175 for 2 drives.

https://www.amazon.com/dp/B09F31ZXKQ -AMD
https://www.amazon.com/dp/B07CVGZZCS - intel

@JayMX - I don't do VM's on ADL. I've done them in the past on prior generations but, at this point don't need them or feel like dealing with them.

 

[igor_kavinski]

To be honest, you don't have a mission critical requirement for maximum and guaranteed performance. A Ryzen 7950X, even with the added latency of communicating across the two CCDs, is not going to feel slow to you. The performance disparity between the P-cores and E-cores may also not be that much noticeable for your use case. What you are mostly doing is OS level tasks that aren't very compute intensive to begin with.

Only Microsoft Exchange has the possibility of bogging down your machine. If you run into performance issues with it, you could always put it on a dedicated older PC or even a Ryzen 5500/5600/Core i3-12100/i5-12400 with 16GB RAM and connect it to your testlab domain over physical LAN.

 

[igor_kavinski]

feel like dealing with them.

I know what you mean. Especially Hyper-V VMs are not fun to play with in my experience and VirtualBox will often get buggy one way or the other.

An important thing for the OP to consider. Only one hypervisor may use the special virtualization features of the CPU. So if both Hyper-V and VirtualBox need to be used, one of them will need to work in pure emulation mode which is slower.

 

[moinmoin]

one of them will need to work in pure emulation mode which is slower.

much much slower even

 

[Shivansps]

What kind of performance do you want from your VMs? I think compute workloads can get around 95% of the host CPU core but if there is something that requires a lot of context switching (most notorious for that being databases), the I/O performance tanks considerably inside the VM.

This is a Samsung 980 Pro 1 TB SSD on an Ivy Bridge-E Xeon server:

View attachment 69370

Now look what happens to the 4K IOPS inside Hyper-V with the VHD stored on the same SSD:

View attachment 69371

IOPS suffer 42% loss.

Maybe this loss is much less on modern platforms.

Another thing to consider is how parallel your computing workloads are. Generally, AMD is really good at multitasking. With Intel, the foreground process will have the lion's share of the performance but everything in the background will run with much reduced performance.

For that reason i would recomend a type 1 hypervisor, Proxmox is good (and free), that it will be less affected by IOPS loss usng the proper iSCSI driver. And you can easily pass a entire SATA o PCI-E NVME drive directly to the VM.

The pcie passthrough should have zero performance impact and should perform the same in the guest as it would on the host, altrought you would be dedicating a entire drive to only one VM, it is recomended if you are working with DBs. Just use multiple smaller nvmes instead of one big one.

 

[igor_kavinski]

Are you a human?

www.newegg.com

Not sure if this is still available because I'm behind a firewall but I think this is the cheapest Optane drive available right now. If you can afford to RAID-0 two of these, the performance will be insane. Just know that that these can get pretty hot.

 

[LightningZ71]

For just MS labs and not a complete "every second counts" installation, if budget matters AT ALL, I strongly suggest that you can find immense value in the 5900X on a decent X570 board. The Taichi offers full bifurcation on the x16 slot (last I checked) and is known to support ECC DDR4 ram. I see that you are using MS storage spaces, have you setup tiered storage for your space? For your purposes, you can use a boot and VM OS host drive on the first M.2 drive, then install a decent M.2 drive on the second M.2 port (first chipset one) and set it up as the SSD tier for your drive array. Then you can use that whole array as your VM data drives by assigning hypervisor virtual drives to files on that array, or just partitioning the array.

That will give you plenty of performance for lab machines. I do something very similar myself, both having used Hyper-V and later VirtualBox. You could do something quite similar with Proxmox provided that you know your way around ZFS.

 

[desrever]

I am doing labs mostly with Microsoft products (Windows Server, Exchange, Windows clients, firewall, etc.) in VMs and ideally I 'd love to have the most cores possible. Sometimes I 've seen that some VMs (like a windows server with a number of 'heavy' roles) need more than one physical core assigned. Also ram capacity is important (DDR5 on-die ECC is fine) but that is something I can add any time so I think most important is the CPU. I am looking for at least 8 physical cores but if budget allows it would like to go more.

@TheELF you 're right about assigning the resources optimally and I assume it is not possible with a 12900K to assign the p and the e cores on specific VMs. On the AMD scenario I guess you are referring to the latency in between the CCDs but is this really equally bad to having an e core assigned to a machine you need CPU power?

@Tech Junky would you be so kind to add some more info from your experience on AL and Virtualization?

@igor_kavinski great info something I need to dig into as well - I now use a Storage Spaces with mechanical HDDs (RAID0) but in this new setup I was thinking either a single big capacity M2 or 2x M2 in raid0.

Thank you all for your replies.

It sounds like you don't really require that much hardware, most of the VMs (exchange, firewall) you listed can be 1 core/thread assigned if you are only doing labs. I would say you can do everything on a 4 core 8 thread machine. You are way overthinking things imo.

 

[JayMX]

Thank you all once more for the great ideas and information.

@desrever I am now on a 4/8 and often have to compromise with sluggish performance - And I don't like sluggishness at all.
4/8 definitely not enough and I am thinking of something capable and future-proof as I am not building /upgrading often.

 

[Tech Junky]

Well, most current offerings have 8/16 these days so that would be significant.

 

[TheELF]

Thank you all once more for the great ideas and information.

@desrever I am now on a 4/8 and often have to compromise with sluggish performance - And I don't like sluggishness at all.
4/8 definitely not enough and I am thinking of something capable and future-proof as I am not building /upgrading often.

Unless you made sure and checked this could just as well be an issue with ram though...
Also again if you don't limit your VMs and the VMs can use a lot of processing power then no matter how many cores you have you might encounter sluggishness.

 

[igor_kavinski]

I am now on a 4/8 and often have to compromise with sluggish performance - And I don't like sluggishness at all.

Are you running your VMs from an SSD? If not, that's the source of the sluggishness. Hard drives are slowwwwwwww.

 

[Shivansps]

On some pcs Virtualbox and Hyper-V are just slow, even if you have enoght ram and you are running everything off a ssd. Im not really sure why.

The day i discovered Proxmox i not longer used VirtualBox or Hyper-V ever again, i just run everything i need off my "vm pc" with the web gui. The main problem that you need to dedicate a pc as the "vm server".

 

[iamgenius]

AMD looses performance on all cores if all cores are loaded...intel only on the e-cores.
Partitioning the resources to the VMs is a user obligation, in both cases intel or amd you will have to assign a specific number of threads to each VM and make sure they run on the p-cores for intel* and on the same ccd for amd.
Having the cores of one VM split between two ccds on amd would be at least as bad as having threads on an e-core.

* unless you have a VM that needs less performance in which case you would want to send it to the e-cores.

That's interesting. I didn't know that. I will also be building a machine for virtualization. I will most likely get the AMD Ryzen™ 9 7950X since HEDT is no more . So Vmware or virtualbox will allow you to pick which ccd to select cores from???

 

[JayMX]

I use resource monitor to check on the VM so I see high to max (100%) CPU usage, in either short or longer bursts. The HDDs Spaces stripe I use has an effect on VMs' loading times but other than that not much - not intensive reads/writes occurring.

About Proxmox it's a bare-metal hypervisor and needs the hardware to be dedicated for it. I plan to have this computer for personal use as well so windows native hyper-v is very practical in that sense. I briefly checked if I could run proxmox inside Hyper-V (nested) so that I can test it but seems like only the opposite is possible (?).

Having checked on platform proposals - Intel seems to offer more value with 13th gen while AMD offers longevity and an upgrade path in the future but with a premium cost. Still unsure.

 

[positivedoppler]

I use resource monitor to check on the VM so I see high to max (100%) CPU usage, in either short or longer bursts. The HDDs Spaces stripe I use has an effect on VMs' loading times but other than that not much - not intensive reads/writes occurring.

About Proxmox it's a bare-metal hypervisor and needs the hardware to be dedicated for it. I plan to have this computer for personal use as well so windows native hyper-v is very practical in that sense. I briefly checked if I could run proxmox inside Hyper-V (nested) so that I can test it but seems like only the opposite is possible (?).

Having checked on platform proposals - Intel seems to offer more value with 13th gen while AMD offers longevity and an upgrade path in the future but with a premium cost. Still unsure.

I question the value part. Intel consumes massively more power. Computers running VMs are usually left on. My 5900X idles pretty high in eco mode if I have even one vm active. Every watt of heat generated is another 2 watts of ac required to remove it

 

[TheELF]

I question the value part. Intel consumes massively more power. Computers running VMs are usually left on. My 5900X idles pretty high in eco mode if I have even one vm active. Every watt of heat generated is another 2 watts of ac required to remove it

Define massively, because if you use even the highest power that is still under warranty (253W) then the difference is 23W per hour of having your CPU run 100% or in general for most comparable CPUs between 13th and ryzen4 it's about 10% more power(time to complete job) when running at 100% usage.
In this case where the CPU will most probably not be running at full 100% the difference will be even smaller.

And the thing is that if you run single core or idle then intel uses 10W less per hour in both cases, so it all comes down to how many hours you do what workload but it definitely is not a massive difference either way.

Core i9-13900K, i7-13700K & i5-13600K: Gaming-Könige im Test: Leistungsaufnahme und Effizienz

Intel Raptor Lake im Test: Leistungsaufnahme und Effizienz / Leistungsaufnahme in Anwendungen ab Werk / Der Verbrauch steigt

www.computerbase.de

 

[iamgenius]

That's interesting. I didn't know that. I will also be building a machine for virtualization. I will most likely get the AMD Ryzen™ 9 7950X since HEDT is no more . So Vmware or virtualbox will allow you to pick which ccd to select cores from???

May anyone answer this please? Thanks.

 

[TheELF]

May anyone answer this please? Thanks.

Vm will only allow you to select how many virtual cores the VM will have but not where these should run.
If you run multiple VMs you could use shortcuts to give each one an affinity mask

CPU Affinity Shortcut for a Program - Create in Windows

How to Create a Shortcut to Run an Application with a Set CPU Affinity in Windows Processor affinity or CPU Pinning enables the binding and un-binding of a process or thread to a physical CPU or a range of CPUs, so that the process or thread in question will run only on the CPU or range of...

www.eightforums.com