question re: PCIe bandwidth limitations & using multiple PCIe x16 GPUs

[Sunny129]

Often times i'll see a crossfire or SLI capable motherboard advertised as being able to run a single PCIe x16 GPU @ x16 bandwidth in a particular slot, but that it will drop the bandwidth down to x8 when running in crossfire (or SLI) with a PCIe x16 GPU in another slot.

what if i have 2 GPU's running simultaneously, but doing their own tasks/instructions (i.e. not in crossfire or SLI)? are those PCIe x16 GPUs still limited to x8 bandwidth each, or do they each get the full x16 bandwidth? in other words, is it the literal act of running multiple GPUs in crossfire (or SLI) that invokes those kinds of PCIe bandwidth limitations?...or is it the mere physical occupation of those specific slots by GPUs that invokes the PCIe bandwidth limitations?

it is assumed that motherboard manufacturers are referring to crossfire and SLI usage when they talk about PCIe bandwidth limitations on their websites or in their literature...and that makes sense, given that the biggest selling point for multiple PCIe x16 slots was crossfire and/or SLI capabilities. but i don't know that i've ever seen a single motherboard manufacturer specifically state that running multiple GPUs not in crossfire or SLI won't result in the same PCIe bandwidth limitations that affect crossfire and SLI...i wonder if it's b/c they lack attention to detail in their specifications/documentation, or b/c PCIe bandwidth truly isn't limited under such multi-GPU, non-crossfire/SLI configurations...

TIA,
Eric

 

[(sic)Klown12]

When you put a second card in, you drop to x8. Doesn't matter if it's in Crossfire/SLI or not. These motherboard have a set number of lanes to offer for graphics, so when the second card is inserted and powered they have to be divided up.

 

[Sunny129]

thank you for the response...it would be nice if mobo manufacturers included a disclaimer on their sites and in their documentation that stated that PCIe bandwidth limitations not only affect multiple GPUs in crossfire or SLI, but that they affect multi-GPU configurations period. well at least now i know...thanks again.

 

[Elfear]

That's one of the reasons I haven't bought a Z68 board yet and jumped on the Sandy Bridge bandwagon. Going from x16 to x8 doesn't make much difference now at normal resolutions, but I'd hate to invest in a new system and then be limited when the new GPUs hit the market.

 

[Sunny129]

actually i'm coming from a minority perspective. you see i'm not concerned with display resolutions, 3D rendering capabilities, etc. i'm actually going to be using multiple GPUs strictly for distributed computing projects...hence my concern about PCIe bandwidth while not utilizing crossfire or SLI. while having to jump down to x8 PCIe bandwidth makes little difference in games at normal resolutions, the same cannot be said about many DC projects - cutting that bandwidth in half can take a significant toll on GPU crunching efficiency...and now that i know that the same PCIe bandwidth restrictions that affect crossfire and SLI setups will affect multi-GPU setups that are not in crossfire or SLI, i have a more specific question...

i have a motherboard in my possession but is not currently in use. it is an MSI 790FX-GD70, and it has 4 PCIe x16 slots on it. i believe slot 1 has full x16 PCIe bandwidth so long as slot 2 is not occupied, and slot 3 has full x16 PCIe bandwidth so long as slot 4 is not occupied. if slot 2 is occupied, it and slot 1 get restricted to x8 PCIe bandwidth, and if slot 4 is occupied, it and slot 3 get restricted to x8 PCie bandwidth. so at best i could utilize two slots (1 and 3) at full x16 PCie bandwidth, and one of the GPUs would have to run the display in addition to crunching DC work b/c this mobo does not have an integrated GPU. if i wanted to have a 3rd GPU just to run the display while the other 2 GPUs do nothing but DC work, 1 cruncher could operate at x16 PCIe bandwidth while the other cruncher would have to operate at only x8 PCie bandwidth (b/c it would have to share bandwidth with the display GPU). if i wanted to make up for that cruncher's loss in bandwidth by adding yet another GPU for DC work only, making for a total of 4 GPUs, all 4 GPUs would have to operate @ x8 PCie bandwidth (since PCIe slots 1 through 4 would all be occupied, and slots 1 and 3 share bandwidth with slots 2 and 4 respectively).

at the end of the day, there aren't many mobos out there that don't sacrifice PCIe bandwidth in ways that don't appeal to me. i have yet to find a mobo out there that has enough PCIe bandwidth/lanes to support 3 GPUs at full x16 PCIe bandwidth. in fact, i'm finding it difficult to find a mobo that provides full x16 PCIe bandwidth to just 2 slots with a 3rd GPU present. take my first example for instance (the MSI 790FX-GD70 i have in my possession) - the only way to have 2 GPUs @ full x16 PCIe bandwidth is to make one of them run the display. otherwise, in order to devote those 2 GPUs solely to DC work, a 3rd GPU must be installed to run the display, forcing one of the dedicated crunchers to drop down to x8 PCIe bandwidth operation. another example is the ASUS Crosshair V Formula mobo, which also has 4 PCIe x16 slots on it. it can also support 2 GPUs at full x16 PCIe bandwidth, but once the top 3 PCIe x16 slots are occupied, only the top slot gets full x16 PCIe bandwidth - slots 2 and 3 get limited to x8 PCIe bandwidth. so if i wanted 2 dedicated crunchers and a dedicated display GPU that has at least x8 PCIe bandwidth, one of my crunchers would not get full x16 PCIe bandwidth using this mobo (b/c it would share bandwidth with the disaplay GPU). i suppose i could alternatively use the bottom PCIe x16 slot for the dedicated display GPU even though its limited to x4 PCIe bandwidth, b/c its occupation does not limit the ability to run dual GPUs with full x16 PCIe bandwidth in slots 1 and 2.

...and then there are mobos like the AsRock 890FX Deluxe4 and 890FX Deluxe5, which both limited to only 3 PCIe x16 slots, but seem to have constant/unchanging PCIe bandwidth configurations (PCIe x16 slot_1/slot_2/slot_3 always seem to have x16/x16/x4 PCIe bandwidth respectively, regardless of which, or how many, slots are occupied). the only downside seems to be that slot 3 is limited to x4 PCIe bandwidth...but such a slot would have enough bandwidth for just display purposes, i would think. and that leaves PCIe x16 slots 1 and 2 for 2 GPUs to run at full x16 PCIe bandwidth. so to sum up my extremely long lead-in, i'm seeking confirmation that either of these AsRock mobos (or the ASUS above) are in fact my solution to being able to run a dedicated display GPU along side 2 dedicated crunching GPUs at full x16 PCIe bandwidth.

thanks again,
Eric

 

[greenhawk]

it is a limitation of the chipset. How the motherboard manufactures make use of them is what makes the difference. And that is assuming the limiting of the bandwidth makes a big enough difference to matter in the over all cost of things (ie: two GPUs at x8 is proberly cheaper than 2 separate systems at x16).

if you want more PCI-E slots to get the extra bandwidth, then you are forced to look at some of the server boards that do have the extra lanes (or the high end cpus like s1366 or the newer s2011) or even look at dual cpu boards (running Xeons or Opterons).

Even the high end boards that do support more bandwith to each slot (electrically and not just phically) sometimes use PCI-E switchs, so in effect, you are still limited ot 16x for the two slots, but if one needs full speed while the other is ideling, it gets the extra PCI-E lanes for a time. While this covers some issues of not having 16x lanes, under load it is still the same as having 2 8x slots.

Only advice seeing as you mentioned you have one of the 16x or 8x/8x boards is to test to see what sort of difference (or search for the information) it makes. If 10-20% then I would live with it. If it is as you fear and is a 50% performance loss, look at buying individual machines instead (ie: something with integrated GPU and a single 16x slot, adding in whatever cheap cpu you can find to run it).