Some preliminary testing of PCI passthrough with FM2+ socket

[PingSpike]

I'd been planning on trying out AMD-Vi (intel vt-d) IOMMU passthrough to virtual machines for awhile.

AMD doesn't offer a lot for performance these days but one area they do seem to have a niche IMO is if you're trying to do PCI passthrough on the cheap. Intel processors require purchase of an i5 (non-K only as well) to get the feature which even for a used 1155 processor I was looking at $130 for the cpu. I actually already have one of those but since I might use this box as a low end server role and I didn't want to mess with my main machine first this was appealing to me. Unfortunately my plan to buy a cheap used FM2 processor used never really worked out, I ended up with a A4-6300 for $40 which is OK.
So as a toy I recently picked up an FM2+/FM2 board, Gigabyte GA-F288X-D3H:
http://www.newegg.com/Product/Produc...82E16813128656

Being inexperienced in Linux in general I decided to do testing with esxi since it seemed a lot more straight forward. After some short struggles with getting the removed realtek network drivers included in 5.5 update 2 I got that up and running.

About the board for this task:
There is little information about working passthrough combinations in general, and there is near to nothing about it on the FM2 socket. That is really why I'm posting, to put a record out there. One thing that really sucks based on research and so far confirmed with this board is that, at least boards based on the newer (Bolton) chipsets, none of the onboard(chipset) devices seem to be enabled for passthrough. Nothing onboard showed up in the list except the onboard Realtek LAN. There was a user hunting around that whose trail I followed through google who noticed this with an asrock board and wondered if the previous chipset 85X (Hudson) boards had the same issue. One passing antecdote I saw seemed to indicate they did not but these boards are not easy to acquire cheaply anymore so I chanced on gigabyte. This gigabyte board is sparse on added on features, the other user that had a 88x board at least seemed to have an additional USB3 controller onboard that was available for passthrough. I knew that no onboard devices showing up was the most likely outcome, I selected the gigabyte hoping that was an asrock only problem (it wasn't) and because the board has a lot of slots so if the problem wasn't asrock only at least I had a lot of ports to work around the problem with.

So even though no onboard devices worked I hope at least add on ones do. I added a 4850, Sound Blaster Live! Value, and a cheap ass USB3 controller to the board last night and all devices appeared available. I passed them through to an XP virtual machine and the OS found them all and Windows update installed the drivers for the live! at least. I didn't get a chance to test if anything worked but it didn't crash on boot.

I think my next step with be to do some basic sound tests, then to try and get the 4850 working.

 

[PingSpike]

I didn't have much time last night, but I got vmware tools installed, windows updates finished and installed AMD display drivers for the 4850. I think it was version 14.4.

I still had the vmware virtual video card installed, but after changing the primary monitor to the one attached to the 4850 I got video out of it. I tried running 3dmark05 1.2 that I just happened to have downloaded, but it seemed to lock up on the loading screen. I don't know if that has anything to do with passthrough though, its old as shit now. I also quickly ran Jedi Knight (even more ancient) and managed to enable 3D acceleration but it looked garbled. Again, I'm not sure that works on modern hardware minus the passthrough. No crashes. I'll try again tonight with something newer hopefully.

 

[VirtualLarry]

Interesting.

 

[PingSpike]

So, not much new last night. I installed Mount and Blade: Warband and ran around in the tutorial. It seemed to play fine with no graphical glitches. I didn't play for that long. I installed 3dmark06 and ran it, there was a crash report for something taking system info on startup but after that it continued loading and then ran and completed. I don't actually think the crash had to do with passthrough really, I suspect it just didn't handle the virtualized system well for some reason. The application itself seemed to run fine.

I'm not sure if 05 locked up because of the same issue or what. Jedi Knight...well, thinking back I'm pretty sure that game has serious to minor issues on all modern hardware so that's just not a good test.

I should probably install a modern game or something but with little time at night I didn't feel like waiting for steam to download something. I think I have Skyrim or Oblivion on DVD though.

Right now the virtualized video card (SVGA II or something) is still "installed" on the guest VM. I'm not sure if there are any problems with this going forward. I think you can remove it from the hardware in the VM configuration now that things are up and running but perhaps there would be issues with trying to get into the bios.

Also of note is that I did not install the Catalyst Control Center. I heard that can cause issues so no attempt was made for initial test run.

Up next I should test sound, I suspect there are no issues there.

Then I'd like to attempt a few crazy things just for my own knowledge. I want to try to pass through an x1xxx adapter. All info on this says it doesn't work, you need HD series or better. But those reports are also a few years old at this point, perhaps something has changed.

Then I'd like to try and passthrough some older PCI video cards. Mostly for shits and giggles. There is a working report of some one passing through a Matrox g200e, which it seems the drivers have not been updated for since 2006. I'd like to try an old FX series nvidia card or a MX series, unfortunately I have neither in my possession so this won't happen. I'm also curious if my canopus pure3d voodoo1 would pass.

I won't waste to much time on the above. I ultimately want to get an old capture card passthrough for what I'll actually use this setup for.

 

[DrMrLordX]

Just to clarify, when you say that none of the onboard hardware was available for passthrough, did that include the iGPU? Or are you using that as your primary display device which (probably) makes it unavailable for passthrough?

 

[PingSpike]

The iGPU is not available, but I am using that as the esxi display adapter anyway. However, esxi is using the onboard nic as well and that did show up on the list. I'm actually not sure how much luck people have passing the onboard video through anyway, but I'd prefer to pass the igpu through and use one of the older boards I have lying around doing nothing as the esxi console but it doesn't seem to be an option.

 

[zir_blazer]

PCI/VGA Passthrough is a pain to get right. Also, ESXi is a bad choice for this purpose. While it has Passthrough capabilities, they seem to be inferior to what is possible with Xen or QEMU KVM VFIO. The problem that you have is that Xen itself is a hit-or-miss, some people may get it working flawlessly in an specific version while its impossible on others, so you may end up installing different versions just for test what works for you.
The other problem is that for a first time guy with no Linux experience, you should try to get a distribution which has both a Xen package ready, and decent instructions on how to install it. Same if you want to use QEMU KVM VFIO. Arch Linux has both, but be ready for a warm welcome in a console shell.

 

[imagoon]

I would take a guess your system doesn't have ACS or the video card is behind an nonACS controller. You also may need to map and and all PCI bridges between the cpu and the card slot to get it to be stable.

The fact that the entire PCI table isn't appearing in the list actually suggests that you have a unsupported or poorly supported bridge or the system board lacks the required parts to actually support VT-d (or what ever AMD's name for the same thing is).

Also Radeons are known for issues because the cards don't seem to announce all of their memory space until the driver is talking to them which requires .vmx tweaks to and PCI hole values.

 

[PingSpike]

zir_blazer: Yeah, I'm not married to ESXi as a final solution, but it got me off the ground with very little effort. I just wanted to try passing though graphics cards to start and it seems to have gotten me there. As you said, other setups are still maturing as well. Its possible the missing onboard devices have to do with ESXi since I believe the other user with this chipset was using it.

I'm doing a little more testing today. I'm going to attempt a surely ill fated passthrough of a x1300 adapter. I do not expect that to work but I have a couple reasons I'd like to at least try it.

I haven't done extensive testing but the 4850 passthrough seemed to be working fine. I retested JK on my main machine and it actually works worse than the passthrough test machine. It simply crashes on startup. So I'm included to say that game simply isn't compatible with modern hardware/drivers ATM, but my main machine needs a driver update.

 

[PingSpike]

The x1300 crashes windows on reboot after the drivers are installed. Not unexpected, the legacy drivers for it are from 2010.

 

[PingSpike]

So, nothing much new lately. Been busy. I installed a PVR-150 card and it passed through (it was actually automatically passed to the VM that already had a passed PCI device once I added it to the table since all PCI devices can only be passed to the same VM). Drivers installed and no crash. I'll install the crappy software that goes with that card and try to do a quick capture.

I'm just going to add a post that sswift (the user whose trail of bread crumbs I followed to start) posted on tweaktown forums about his attempts with XenServer instead because its interesting and I dug it out of google cache. The only question he had that I've really answered is that its not specific to Asrock's implementation.

Re: Asrock FM2A88X Extreme6+ : Partial ESXI/Passthrough support?

Well, Just tried XENSERVER 6.2 and managed to successfully to pass-thru the embedded GPU to a windows VM (with HDMI sound as well). It looks like that ESXi is not properly recognizing the A88X (Bolton D4) chipset for pass-thru. It could also be a BIOS issue or the implementation of the A88X chipset on the specific Asrock Board, as all other PCI devices are passing through properly - I don't know if the embedded devices in Bolton-D4 chipset (SATA3/USB3/GPU) are behind a proper PCI bridge.

Note: In XENSERVER 6.2, the integrated GPU (8570D) does not passthrough without issues, though. When restarting or shutting down/powering up the Windows VM, the GPU is not recognized and therefore there is no output on the screen. The solution is to reboot XEN host...

Currently I am testing stability and performance stuff in XENSERVER. I will try again in ESXi , as it has a couple of advantages that don't want to lose (like RDM, more powerful UI - Vsphere Client-).
If this a no-go, I am thinking of trying the Extreme6 (A85X) board to see if it properly works on ESXi, as I have to rebuild all of my VM's if I stick with XENSERVER.

I would love to know, if any of you guys managed to pass-through the embedded GPU (particularly) in ESXI, on any other Asrock FM2/FM2+ board. ..

Last edited by sswift; 01-24-2014 at 07:43 PM.

 

[imagoon]

That post seems to at least partly confirm that the board doesn't have everything it needs to do PCI pass through properly. Just having it on the CPU isn't enough.

 

[zir_blazer]

The DomU issue is common, it may not be Motherboard related at all. No Video Card that I'm aware of supports FLR (Function Level Reset), they need to be resetted by Software, as Drivers on DomU boot would typically expect to have it in a fresh status instead of initialized. So some people requires a full Dom0 reboot because the GPU only works on DomU the first time it is started. Workaround was to use the Windows "Safely Remove Hardware" to eject the Video Card from inside the DomU.
On my case (Supermicro X10SAT, Xeon E3-1245V3, Sapphire Radeon 5770 Flex Edition, Arch Linux with Linux Kernel 3.16.4 and Xen 4.3.3), it works properly every other boot, with no need to reboot Dom0 at all. May want to read this.

I have a guide in the works for installing Arch Linux and Xen. However, it is in spanish and will have to translate it. Also, Passthrough success is heavily dependant on Xen version. I think my system should be among the top in functionality with Xen 4.3.3, but on 4.4.1 Passthrough doesn't work properly.

 

[PingSpike]

Actually my take based on that post and zir_blazer's is it seems possible that there something in ESXi in particular that keeps it from being able to use the onboard devices for passthrough. It wasn't clear if he was able to pass through other devices besides the onboard video but it sounds like the the issue he had occurs on all radeons including discrete?

I'll have to keep in mind the pci hole thing. For my tests I neglected to allocate a large amount of ram so that issue may simply be obscured at the moment.

I saw a lot of people had trouble passing through on another thread but some of the users that didn't seem to follow the path I took...entirely by happenstance. I created a VM machine with no discrete video card allocated to it, installed windows and booted it. I may have then installed VMware tools or perhaps I did it last. I know I installed the video card physically at this point, marked it for passthrough and then rebooted ESXi. Then I added it to the existing windows VM, booted and installed the drivers. Its possible I then installed VMware tools after rebooting again for the drivers. At some point I changed the primary display to the read monitor, setting the virtual VM SVGA II device to the secondary screen. At that point I began testing a few 3D applications. I don't know if that series of events was required.

 

[PingSpike]

PCI/VGA Passthrough is a pain to get right. Also, ESXi is a bad choice for this purpose. While it has Passthrough capabilities, they seem to be inferior to what is possible with Xen or QEMU KVM VFIO. The problem that you have is that Xen itself is a hit-or-miss, some people may get it working flawlessly in an specific version while its impossible on others, so you may end up installing different versions just for test what works for you.
The other problem is that for a first time guy with no Linux experience, you should try to get a distribution which has both a Xen package ready, and decent instructions on how to install it. Same if you want to use QEMU KVM VFIO. Arch Linux has both, but be ready for a warm welcome in a console shell.

Are any of the Ubuntu derivatives good candidates to try?

 

[zir_blazer]

May want to read this guide for Ubuntu. You have too this one for Linux Mint. And this one for Fedora.
At least on Xen, the PCI Hole while doing VGA Passthrough is supposed to have been fixed a year or two ago. I have a WXP x64 DomU with 24 GB of RAM and a Radeon 5770 working good.
Also, you made me remember a thing. I saw introduction of some patchs for Intel VGA Passthrough because the integrated GPU seems to have some issue with the PCI Address or something like that (Drivers always assuming it is 00:02.0 or something around that line), and they reserve a PCI Address for it if you're passing it. Maybe it also applies to AMD integrated ones.

I have a distaste for Ubuntu as Dom0. Because the idea is that Dom0 is just merely a administrative VM, I wouldn't put a complete, bloated distro for that purpose, when something slim like Arch Linux could cut it as well. However, if it is easier for you, go for it.
I need to translate my guide ASAP...

 

[DrMrLordX]

If bloat is an issue for Ubuntu, there is always the option of using Lubuntu instead. Same repos as Ubuntu, but the install size and memory footprint are much smaller.

 

[PingSpike]

Yeah, I was thinking Lubuntu anyway. I prefer minimalist designs.

Could you not make Dom0 your main work OS or does it not work like that? One thing I'm struggling with thinking ahead to another project is that ESXi uses up a bunch of onboard devices making them unavailable for passthrough. And you can't do anything with ESXi as an OS itself, its just a slim bare metal hypervisor. Ideally, I guess I'd like a Linux install that functioned in a traditional way and for a second Windows install to be virtualized under that with only a few devices like a discrete graphics card passed to it. But maybe that is a silly way to do it.

 

[PingSpike]

I got to play with this for awhile over the holidays. I made it a point to take good notes on my tests.

One thing I like about esxi is that it has a fairly simple gui for configuring passthrough. I think I encountered a bug with it when swapping cards in and out which resulted in nonsensical entries that did not work when passed through. In the end this problem seemed to go away after I yanked all the cards at one point.

I had a purple screen of death and other hypervisor lockups in early testing.

Unfortunately, I never got my hvr-1850 card to work. The card appears for passthrough fine and the drivers install fine in XP and Win7. However, in XP viewing anything from the analog s-video input resulted in the video crapping out pretty quickly and the same happened with recordings. There would be chunks missing from the .ts file and the file always failed to record the total time. Windows 7 actually would record the appropriate time but it has a different nasty issue. The tuner would become inaccessible after successfully recording and nothing short of a reboot of the guest would fix it. Is the driver locked up? You could not record or view the s-video. Neither of these scenarios provided an acceptable solution.

I tried it in probably every pci-e slot while trying to get it to work with xp.

In bare metal XP there were no issues. I have not yet tested Windows 7 bare metal to see if the inaccessible tv tuner issue occurs there. Its possible its a general driver issue.

Maybe this card works on a linux guest, I have not tested.

Maybe Xenserver could work.

I also strongly suspect this card is the cause of most if not all of the hypervisor lockups I had. I think it was involved in all of them.

I do have my older pci hauppauge pvr-150 working though. This card seems to have no problems despite being ancient. I did have trouble detecting during my card swapping craziness that may have been do to poor seating because I recall not being able to detect it in a bare metal install then either. I've done a fair number of recordings and am pretty confident in this one.

I never got my NEC chipset Renesas card to work, it gets a driver failed to load error. Other people seem to have gotten this card to work. I ordered another card with a different chipset. I probably should try to find updated drivers to try again.

I have a cheap pci-e c-media sound card that works with the exception of a code 10 on the game port...which the card doesn't have. Sound seems to work fine with testing the playing of a few mp3s. I didn't notice any crackling or anything. XP install.

Sound blaster live! value pci works fine, windows installs drivers and its good to go. Sound test of several mp3s was fine. I didn't notice any crackling or anything. XP install.

HD 3450 testing is...ongoing. I get memory cannot be read errors on the install on this sometimes when navigating the CCC. I thought they had disappeared after installing VMware tools sans the video driver but I saw a single one pop up. I use UltraVNC to view this (because the video card prevents output to console in vsphere) and it was spiking the cpu to %100 while running it, then yesterday it didn't do that. As this is a cheap used card off amazon and I never actually ran it baremetal...its possible this is simply a bad card. Or there could be something wrong with the OS install. I'm running it through the 16x 4x electrical port because its sole purpose is to provide an s-video output. I don't recall these kinds of problems with the hd4850.

And finally...Canopus pure 3D 3dfx voodoo1. This card...mostly works? If you know your old ass video cards, you'll know the voodoo1/2 cards were 3D cards only. They used a vga passthrough cable and an electronic switch turns on video output when a 3D application is launched with another graphics card required to perform 2D functions. I initially thought testing this would be complicated by the necessary addition of another physical card to control the windows desktop before launching a 3D application. Then I realized that a real card was not required. VMware console works as usual but when you launch a game the console video simply stops being updated and video it output from the output port of the voodoo card as per usual.

I only played Unreal Tournament 99, because it was the only glide title I had lying around. Unfortunately, there didn't seem to be any XP/2000 drivers for direct3D (any maybe opengl) for the voodoo1. There are some for the voodoo2 which came out just as 3dfx was going defunct.

The bug this has is similar to the one people have with other graphics cards and is actually fairly tolerable if you really need this weird setup. I get garbled output when the application launches after rebooting the guest. It seems rebooting again (not sure how many times? Not more than 2 I'd have to check my notes) clears it up. The first boot always works, or at least it seemed to in my tests. Not enough testing to say for sure but I believe it only going out to lunch after the guest is shut down and then restarted. I couldn't unmount/remount from the taskbar, it says the hardware didn't support it.

 

[DrMrLordX]

Mad props for testing the voodoo1. A nitpick: 3dfx kept afloat as long as it did specifically because of the Voodoo2. It was the Voodoo Rush, Voodoo3, and then finally the Voodoo4/5 that finally did them in.

 

[PingSpike]

Sorry, I was a bit vague there. I ended up reading a bit about 3dfx history in my quest for full blown voodoo1 drivers. I meant that voodoo2 direct3D drivers were released as the company was going out of business, but it wasn't their final product. I think the voodoo2 was originally released in 1998? And the company went out of business in 2000-2002? I guess. So the voodoo1/voodoo2 predate the release of XP or 2000. It looks like they never felt the need to develop voodoo1 drivers for 2000/XP. There are glide drivers though and I believe openGL drivers. The lack of direct3D drivers for 2000/XP probably has a lot to do with the fact that the NT codebase predecessor to those OSes, NT4, did not even have Direct3D support.

As a side note, I thought about buying a cheap voodoo2 off ebay but it seems they have fallen into collectors item status now and go for $40-50 at least. That's to much money when I actually couldn't think of an actually use case for myself. I tested because I already had the hardware and liked the idea of having a VM that could run old early 3D games...even though I never actually do that.

 

[DrMrLordX]

Oh, interesting bit about the voodoo2 direct3d drivers. It seems bizarre when you consider how many games back then supported GLide. D3d was a poor-performing afterthought.

I can see that having Voodoo1s and 2s with passthrough would be must-have for old Win95/98 VMs running stuff like Quake 2. Them were the days . . .

 

[PingSpike]

Well, I did get that hvr-1850 card to work. I added pcipassthru0.msiEnabled=FALSE to the VM config and it seems to work alright now. I'm kind of frustrated that I did not know about that option since I wasted a fair amount of time trying other things and apparently a lot of people use that to get hardware working. What I didn't find out was what that option does exactly.