LGA 1150 Haswell Motherboard with proper VT-d support

[zir_blazer]

A bit more than 3 years ago was the first time I heared about IOMMU virtualization. What it can be used for? To do things like this. From that time, I'm bend that my next build has to support it so I can migrate to a potentially more stable and secure, fully virtualized platform, and raise the middle finger to any specific OS limitation. I will be an OS overseer, ruling supreme above them, and I will choose the most appropiate one for a given task.

After several months wandering around Forums and the Xen Mailing List gathering information from both Intel VT-d and AMD-Vi, the sad conclusion is that albeit Hardware support is there and isn't that rare, Motherboard makers SUCK when it comes to implement it. And it still isn't crystal clear what Hardware is the bare minimum to make it work. For example, for Haswell, AsRock said here, that they got Motherboards with four different Haswell Chipsets with VT-d support, while ASUS claims that on any non-Q Chipset, it fails some tests and that is why they don't support it.
However, from my experience googling around, there are MANY Motherboards that got a BIOS with a option to enable/disable this IOMMU virtualization feature (ASUS included, and most prominent), but when enabled, Hypervisors like Xen complains that the BIOS doesn't have a proper ACPI table that is required to make the feature work, so it is as good as useless. That is the reason why I need some hard evidence that a Motherboard will work for this before purchasing. While a mere BIOS upgrade can fix that assuming proper Hardware support, there are some Motherboard makers that don't even care about it to put resources to fix it. Oh well, we're should be used already to see BIOS with incomplete support that are non-compliant because they don't even announce a feature when they should like this and this.


As I have decided that my next build will be Haswell based, with either a Xeon E3-1245 V3 or 1275 V3, and 4 * 8 GB RAM that I already purchased, what I am missing here is a LGA 1150 Motherboard that can make Xen work with VT-d. I don't care if its from the Desktop segment or a more serious Server one with the C200-series Chipsets, for as long as what I win and what I miss is documented and not an after-purchase surprise.
One of the reasons why I'm not confident about a Server Motherboard, is that due its purpose, chances are that I don't get the wide array of BIOS options to tinker with settings to run Hardware out-of-spec, and while I'm not interesed in overclocking (And neither I would be able to, with a Xeon or non-K Haswell), I will surely try to undervolt as low as possible at default Frequency. I also don't know how the C200-series Chipsets compares to the Desktop counterparts in features, Intel Ark seems quite limited when it comes to Chipsets.

 

[Insert_Nickname]

For example, for Haswell, AsRock said here, that they got Motherboards with four different Haswell Chipsets with VT-d support, while ASUS claims that on any non-Q Chipset, it fails some tests and that is why they don't support it.

It most likely depends on the exact BIOS implementation the manufacturer uses. Also BOTH chipset AND CPU has to support VT-d for it to work, and most likely you have to actively enable it in BIOS. Most mainboards default to disable VT-d.

From what I can gather from the manual this Q87 mainboard support VT-d, but you still need a CPU WITH EXPLICIT SUPPORT (i5 and higher, non-K) to enable it. Best way to check is the ark.

As I have decided that my next build will be Haswell based, with either a Xeon E3-1245 V3 or 1275 V3, and 4 * 8 GB RAM that I already purchased, what I am missing here is a LGA 1150 Motherboard that can make Xen work with VT-d. I don't care if its from the Desktop segment or a more serious Server one with the C200-series Chipsets, for as long as what I win and what I miss is documented and not an after-purchase surprise.

The ASUS P9D-WS might be just what you're looking for, combined with a Xeon.

 

[zir_blazer]

It most likely depends on the exact BIOS implementation the manufacturer uses. Also BOTH chipset AND CPU has to support VT-d for it to work, and most likely you have to actively enable it in BIOS. Most mainboards default to disable VT-d.

Intel has this page where they claim that you only need Processor and BIOS support to use VT-d, they don't mention Chipset. Actually, they tell you in what Intel-branded Motherboards VT-d works, and at the very least you have a wide variety of Chipsets to choose from: H61, Q67, X79 for Sandy Bridge, B75, H77, Q77, Z75, Z77 for Ivy Bridge, and B85, Q87 for Haswell. Chances are that pretty much it works on any Haswell Chipset, don't recall hearing anywhere that it is artificially limited.
Chipset had to support it on LGA 775 platform, at that point it was a Chipset feature only. From Nehalem on, I think its a CPU feature only.

From what I can gather from the manual this Q87 mainboard support VT-d, but you still need a CPU WITH EXPLICIT SUPPORT (i5 and higher, non-K) to enable it. Best way to check is the ark.

Processor is already accounted for, I'm staying away from K series.

 

[Insert_Nickname]

Intel has this page where they claim that you only need Processor and BIOS support to use VT-d, they don't mention Chipset.

That's only a list of their own boards (Intel Desktop Boards) that support standard VT-x. If you look further down you'll see the only boards to support VT-d use either B or Q chipsets with the somewhat strange inclusion of a few Z/H/X thrown in. Intel can't say weather a 3rd party board includes the necessary BIOS support and its strictly up to that 3rd party manufacturer IF they'll include support.

 

[zir_blazer]

That's only a list of their own boards (Intel Desktop Boards) that support standard VT-x. If you look further down you'll see the only boards to support VT-d use either B or Q chipsets with the somewhat strange inclusion of a few Z/H/X thrown in. Intel can't say weather a 3rd party board includes the necessary BIOS support and its strictly up to that 3rd party manufacturer IF they'll include support.

I already looked further down and I saw the VT-d part of it - that is why I linked to it, d'oh!
You said that you need Chipset support for VT-d, but coming straight from Intel's mouth, they didn't mention needing a specific one to get VT-d working, just Processor and BIOS support. They also provide that list with Motherboards that supports VT-d, where you can see a wide variety of supporting Chipsets that *MUST* support VT-d if they're there. There was also a guy on Xen Mailing List saying that AsRock claims VT-d support on Haswell H87 and Z87 Chipsets too, that aren't included on Intel list.
I would say that VT-d in post-Nehalem era isn't related to the Chipset at all, or at least, I didn't recall ever hearing that Intel is artificially crippling VT-d support from the Chipset side. They currently do that with the Processor. So as far that I'm aware, you don't need Chipset support for this feature.


The problem with specific Motherboards is that until tested, you don't know if the VT-d/AMD-Vi implementation works or not. This means that just by looking the manual to check if the BIOS got an option to enable it is NOT enough to guarantee that it will work. That is why I asked if someone had "hard evidence", or basically, tried to run Xen or other Hypervisor using this feature to tell if it works or not.
Many Motherboards, many from ASUS, are missing the DMAR (For VT-d) or ACPI IVRS tables (For AMD-Vi) that are required to be in the BIOS for it to work. If they aren't there, you can't use it. So there are many Motherboards that don't work with this feature simply because of that, even if the Hardware side seems to be fine.

 

[Insert_Nickname]

You said that you need Chipset support for VT-d, but coming straight from Intel's mouth, they didn't mention needing a specific one to get VT-d working, just Processor and BIOS support

I think we're misunderstanding each other. You don't need specific "chipset support" as such, but this feature is (usually) only implemented/enabled on B/Q series based mainboards. If you want 100% guarantee that VT-d works, you need to pay the premium for server grade hardware.

 

[phrosty]

Virtualization is hosed on Gigabyte's G1.Sniper 5. The current BIOS will actually mask off a lot of the new CPUID bits when you enable VT-d, and it's impossible to enable the virtualization features (hypervisor, vt-x) all at once. Pretty maddening, but hopefully a future BIOS revision will fix it. I'd be curious to know if all of Gigabyte's Z87 boards have the same issues.

Pick carefully, make sure you've got a strong confirmation that it works before buying.

 

[ShintaiDK]

VT-D/AMD-Vi doesnt make sense for most people. Hyper-V (and most hypervisors) for example dont even support VD-D/AMD-Vi.

So look if you actually need it. Alot of time people pick/buy hardware on the "what if..." rationale.

 

[zir_blazer]

If I'm specifically asking for a Motherboard for it, is because I'm going to use it. Chances are I'm going to go with Xen, unless I hear that other Hypervisors do VGA passthrough better.

 

[PingSpike]

I understand your frustration on this since I have just begun my side-grade to get this feature working along with gaining the option of intel SRT. I have a H67 motherboard and a 2500K. The biostar board has worked great and even has an option in the bios to enable vt-d but I can't find any information regarding anyone having used it. Given how many have broken implementations there are from supposedly high end companies like ASUS its hard to be to confident in it. I made the wrong bet buying a 2500K instead of a 2500 way back. I didn't know they had yanked an awesome feature from it like vt-d and wasn't 100% sure I was done with overclocking at the time. I now don't care about overclocking and am very interested in different virtualization options. But I made these purchases over 2 years ago and my knowledge and needs have changed. With no performance complaints I was looking to upgrade to get a couple of features.

There's not a lot of information out there as you've found:
http://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware

It is bizarre that Asrock seems to have a lot of representatives in the list. It seems they are fairly proactive with these kinds of features or perhaps the developers are just fans and have thus done more testing.

Since I was looking at swapping a CPU anyway I decided to get a motherboard that was at least somewhat tested with vt-d. Mine *might* have worked but I always wanted iSRT so I had a couple of reasons to change boards. I ended up bagging the haswell route. I probably would have been OK but there are no tested boards for the platform because it is so new. Boards that feature the layout of expansion slots I want were more expensive than the Asrock z77 Pro3 I went with. I am basically giving up some performance for a small amount of money but with no performance complaints I sort of had to remember why I was "side grading" in the first place. The performance improvements of haswell weren't compelling. So now I'm just trying to decide whether I should try to find a 2500 non-K to replace my K or just buy a i5-3470.

If I were you and I was married to haswell I'd either go with an intel branded board that they state has chipset support on their own website or roll the dice with one of Asrock's offerings. Both options are potentially problematic, but the truth is this passthrough virtualization stuff still seems to be in its infancy and is just prone to problems in general. I am trying to do things on a budget otherwise I might have done the same.

 

[zir_blazer]

*BUMP*

but the truth is this passthrough virtualization stuff still seems to be in its infancy and is just prone to problems in general. I am trying to do things on a budget otherwise I might have done the same.

I first heared about the IOMMU 3 years ago, around the time than the AMD 890FX Chipset supporting it was released. It is not in its infancy anymore, is just that no one cared about spending development time to scale it down to Desktop.




Still looking for a Motherboard. There is this list of current LGA 1150 Server Motherboards. However, here comes the issue. After reading a bit, I discovered that ONLY the Intel C226 Chipset allows you to use the Haswell integrated GPU. This means that on C222 or C224, you can't use it and need a discrete Video Card (Or some lame, low end GPU embedded on the Motherboard), and that's a waste considering that Intel finally has a useful GPU. So these are discarded.
I checked some Manuals from Supermicro Motherboards (Server, Desktop) and they claim VT-d support more prominently than other brands. Due to Supermicro being mostly Servers, I suppose they're serious. C7Z87-OCE and X10SAT seems the best bets (If photos are accurate, I would assuming is pretty much the same Motherboard with just different Chipset. X10SAT is missing something on the center-left), but they're a bit too expensive for my tastes, at over 200 U$Ds. I can break the piggy bank, but...

Besides, I don't know the exact features that you win/lose between C226 and Z87. That comparison is far from complete or accurate, as it says that Z87 doesn't support VT-d but I'm sure it does. There is no mention to overclocking/out-of-spec capabilities, where on Z87 you had the most freedom. So I don't know what is the exact tradeoff.
Also, there is the issue that Server Motherboards usually has very lame BIOS options to give you full control, like the C226 based Supermicro I linked earlier compared to the Z87 one. While in neither I would be able to overclock a Multiplier-Locked Xeon via Base Clock more than some minimal headroom, I may want to have fun undervolting it to increase power efficiency, which on the C226 one it seems I can't as the BIOS doesn't even have an option for it.

So basically, what Motherboard could be the closest to be cheap, high quality, have a proper ACPI DMAR table for VT-d to work and be full featured with a BIOS that is missing no option to let you tweak it to your tastes? And maybe I can also ask for Motherboard with only PCIe slots and no legacy PCI slots wasting space, and none of those extra controller chips that adds SATA ports or USB headers that I'm not going to use anyways but adds a potential layer of complexity/issues when virtualizing (Slots that you can't use because the controller chip isn't supported, bla bla bla).

 

[mutantmagnet]

I've been looking for a motherboard with vt-d as well.

Intel makes it very clear when it comes to Haswell you need a Q87 motherboard and they all are problematic lacking many features of consumer boards.

Intel's decision to bundle vPro and Vt-d together is pretty frustrating.

I tried looking at AMD's website to see if there would be alternatives but their site is a complete mess when it comes to finding information on hardware accelerated virtualization.

[edit]
Looking over that page you linked to earlier they mention B85 chipset has Vt-D even though it says otherwise here http://ark.intel.com/products/75019/Intel-DH82B85-PCH

You'll have to contact a sales or customer rep to sort that out.

 

[soulbadguy]

I've been looking for a motherboard with vt-d as well.

Intel makes it very clear when it comes to Haswell you need a Q87 motherboard and they all are problematic lacking many features of consumer boards.

Intel's decision to bundle vPro and Vt-d together is pretty frustrating.

I tried looking at AMD's website to see if there would be alternatives but their site is a complete mess when it comes to finding information on hardware accelerated virtualization.

[edit]
Looking over that page you linked to earlier they mention B85 chipset has Vt-D even though it says otherwise here http://ark.intel.com/products/75019/Intel-DH82B85-PCH

You'll have to contact a sales or customer rep to sort that out.

Bump.

Same Here also, looking a good Q87 board.

 

[vailr]

There doesn't seem to be any full-size ATX Q87 boards.
This mATX Intel board is $138.37:
INTEL BLKDQ87PG
http://www.avadirect.com/product_details_parts.asp?PRID=27753
Newegg lists only two Q87 boards:
ASRock Q87M vPro
http://www.newegg.com/Product/Produc...82E16813157397
ASUS Q87M-E
http://www.newegg.com/Product/Produc...82E16813131988
No idea whether any of these three boards would include the feature being discussed. The Intel board would seem like a good possibility, however.

 

[zir_blazer]

All the previous "Q87 only" info is wrong. While I don't know of first hand experience having VT-d working on a Z87, there do are people that got VT-d working on previous generations non-Q Chipsets. For example, you have Z68 that according to Intel Ark does NOT support VT-d, then you have this. I didn't EVER read that Intel removes VT-d on Chipsets, and I already proved that Intel Ark isn't always right, and is more often that not wrong with regard to VT-d. It all boils down to bad BIOS support from manufacturers. And as I have stated several times already, Intel themselves doesn't mention that you need an specific Chipset for VT-d, just BIOS and Processor support.
I did asked an ASUS rep and he shielded himself saying that only Q series Chipsets passes all the tests, and that they don't want to claim pseudo-support. However, with NO info about what these test are about, we don't know exactly what the other Chipsets are missing. I just know that even on Z series Chipsets from previous generations VT-d works as expected for as long as there is BIOS support.
Besides, between a Q87 and a C226, I think I would go for the latter and build a proper Server instead of a business Desktop.

 

[soulbadguy]

All the previous "Q87 only" info is wrong. While I don't know of first hand experience having VT-d working on a Z87, there do are people that got VT-d working on previous generations non-Q Chipsets. For example, you have Z68 that according to Intel Ark does NOT support VT-d, then you have this. I didn't EVER read that Intel removes VT-d on Chipsets, and I already proved that Intel Ark isn't always right, and is more often that not wrong with regard to VT-d. It all boils down to bad BIOS support from manufacturers. And as I have stated several times already, Intel themselves doesn't mention that you need an specific Chipset for VT-d, just BIOS and Processor support.
I did asked an ASUS rep and he shielded himself saying that only Q series Chipsets passes all the tests, and that they don't want to claim pseudo-support. However, with NO info about what these test are about, we don't know exactly what the other Chipsets are missing. I just know that even on Z series Chipsets from previous generations VT-d works as expected for as long as there is BIOS support.
Besides, between a Q87 and a C226, I think I would go for the latter and build a proper Server instead of a business Desktop.

Interresting... How easy it to check support for vt-d on specific board ? i am thinking of asking people on here to check on a board to board basis.

So which side are you leaning on ? I am thinking right now that the safest route as you said is a C226 and a Xeon processor. The haswell xeon line is actually pretty cheap, and their even have a i5 equivalent with hyperthreading . My main concern software compatibility and driver issue (and linux drivers) as i have no idea how well is the C226 supported on windows

 

[vailr]

Google search of "Motherboard, Intel C226 Chipset" found these three boards:
Supermicro X10SAE-O:
http://www.superbiiz.com/detail.php?p=MB-X10SAE
ASUS P9D, & Axiomtek IMB210.
I'll assume that an Intel branded C226 board also exists?

 

[PingSpike]

I agree with the generally idea that vt-d *shouldn't* be in its infancy, but I'm just saying practice that it still isn't a set it and forget it feature.

The Xen hardware list has more information but many of the boards in this list weren't actually tested. Its not unusual for the implementation to be broken.
http://wiki.xen.org/wiki/VTdHowTo

Intel's website is not very good about explaining what chipsets do and do not work with vt-d, but from what I've gathered its quite possible to add the support to the bios of pretty much any sandybridge+ supporting chipset board (whether that was intended or not by Intel remains to be seen) but ultimately the CPU has to support it, which is easier to determine.

At the very least, if you download the motherboard manuals they will indicate whether the option is available in the bios. From reading around it seems Asrock is better than expected with supporting this considering they are a budget board, or at least Xen users are big enough fans to have tested more of them. I'm sure the Supermicro boards are good candidates as well, but the Xen list indicates they have had problems in the past so even that isn't a perfect route to go. And they are kind of expensive.

 

[soulbadguy]

I agree with the generally idea that vt-d *shouldn't* be in its infancy, but I'm just saying practice that it still isn't a set it and forget it feature.

Sadly, true.But for exemple virtualbox requiere vt-d extension to use a 64 bit guess OS. And In my experience virtualbox is much faster (for the ubuntu unity) and more stable with vt-d enable.

The Xen hardware list has more information but many of the boards in this list weren't actually tested. Its not unusual for the implementation to be broken.
http://wiki.xen.org/wiki/VTdHowTo

Intel's website is not very good about explaining what chipsets do and do not work with vt-d, but from what I've gathered its quite possible to add the support to the bios of pretty much any sandybridge+ supporting chipset board (whether that was intended or not by Intel remains to be seen) but ultimately the CPU has to support it, which is easier to determine.

At the very least, if you download the motherboard manuals they will indicate whether the option is available in the bios. From reading around it seems Asrock is better than expected with supporting this considering they are a budget board, or at least Xen users are big enough fans to have tested more of them. I'm sure the Supermicro boards are good candidates as well, but the Xen list indicates they have had problems in the past so even that isn't a perfect route to go. And they are kind of expensive.

God this is a mess... I went through some of asrock motherboard manual as you suggested and some of them indeed have support for vt-d.The question is now how robust are those implementation vs an official supported chipset (c226 or Q87)... My guess is that those have some kind of intel validation process that guaranty a minimal level of functionality. It would be nice to know if asrock boards (an any other manufacturer) who claims vt-d supports on z87 chipset went through the same certification process

I am really tempted to jump on one of those asrock. The Z87 board have many interesting feature (better sounds wifi ac etc...) vs the peace of mind provided by an official chipset

 

[rekd0514]

I was looking at the X10SAT for a virtualization ESXi build as well, but I don't see anywhere you can actually buy it. It seems like the board to get for what I want to do. I would like to run an all in one box with pfsense, WHS or Server 2012, and a Win 7/8 WMC to record TV, and possibly more if I can think of something.

I was wanting to get a Xeon chip with the integrated graphics as well for the recording TV aspect. I need vt-d to pass through a raid card for the server and the NICs as well.

 

[vailr]

I also don't know how the C200-series Chipsets compares to the Desktop counterparts in features, Intel Ark seems quite limited when it comes to Chipsets.

Wild guess: the C224, C226, etc. chipset's main function is probably to add support for EC RAM.
Intel's chipset drivers actually use the same description for the "pci.sys" driver for both of the Ivy Bridge series chipsets (desktop & server), as listed in Device Manager/System Devices:
Intel(R) 7 Series/C216 Chipset Family PCI Express Root Port 1 -1E10
Whether or not one of the server chipsets is mandatory for supporting the VT-d feature? Is outside my area of expertise.

 

[mutantmagnet]

I have some good news if price is no object for you.

Supermicro makes a motherboard with the Z87 chipset that still happens to support Vt-D as long as you have the appropriate CPU.

Considering their pedigree I doubt this is a mistake in their manual.

http://www.supermicro.com/products/motherboard/Core/Z87/C7Z87-OCE.cfm

Good to hear that Asrock supports VT-d on other chipsets. I'll look more carefully at their manuals to see which ones suite me best.

 

[zir_blazer]

I mentioned that Supermicro Motherboard earlier. I saw it for 251 U$Ds at AcmeMicro, a bit too expensive for my taste, considering that as with a Xeon I wouldn't be able to overclock, I don't need something capable of being pushed hard. At that price, it competes with ASUS SaberTooth Z87, ASRock Fatal1ity Z87 Professional, MSI Z87 MPOWER and Gigabyte GA-Z87X-UD5H. Not counting VT-d support, that theorically all them should be able to support with a proper BIOS, I don't know how it compares to those others to justify its price tag.
Supermicro X10SAT is still unreleased and looks pretty much a sister to the C7Z87-OCE as they look to be based on the same PCB, but will possibily be even more expensive. To be honest, what doesn't convinces me of a C226 is missing all the BIOS features to run out-of-spec. While I will not be able to overclock, Haswell should be capable of a healthy undervolt, which on a Server Motherboard you may not be able to do.

 

[PingSpike]

Sadly, true.But for exemple virtualbox requiere vt-d extension to use a 64 bit guess OS. And In my experience virtualbox is much faster (for the ubuntu unity) and more stable with vt-d enable.

I wasn't aware of any direct performance improvements that vt-d actually offered. I believe direct access to non-virtualized hardware in the guest is the long and the short of what this feature provides. Are you sure you aren't talking about VT-x/AMD-V which are much more commonly supported? A quick peak at the virtualbox docs seems to indicate that is a requirement when running 64-bit guests under a 32-bit host, but doesn't mention VT-d that I could see.

 

[soulbadguy]

I wasn't aware of any direct performance improvements that vt-d actually offered. I believe direct access to non-virtualized hardware in the guest is the long and the short of what this feature provides. Are you sure you aren't talking about VT-x/AMD-V which are much more commonly supported? A quick peak at the virtualbox docs seems to indicate that is a requirement when running 64-bit guests under a 32-bit host, but doesn't mention VT-d that I could see.

I will give a try on my laptop tonight , you might right i might be confusing vt-x and vt-d