4U chassis for six GPUs?

[frowertr]

Trying to put together a test rig built similar to how Tyan and Supermicro have 6+ GPUs installed into a 4U chassis for their Tesla computers. I don't want to spend the $6G+ for those pre-built system (nor do I need their quad CPU, 32 DIMM slots, etc...) so I am looking to build one myself. But I'm not having much luck finding parts on my own.

What I am looking for is a 4U chassis that will accept an ATX motherboard which will contain 6 GPUs. I'll need a PSU of around 1800W+ for this build so the chassis will need to accept either a dual ATX PSU (~1000W) OR dual 1U/2U 1000w OR one 1U/2U of at least 1800W. Power input will need to be 240V AC which is standard anyway for all PSUs.

Of course the PSU(s) will need to have a total of 6 PCI-E 6+2 pin connectors AND 6 PCI-E 6 pin connectors.

I am not sure if what I need exists outside of the Tyan/Supermicro builds. Building a 4 GPU 4U chassis is simple. Lots of companies sell those chassis ready to go. It the extra two GPUs that are throwing me through a loop because of their extra power requirements.

Ideas?

 

[Topweasel]

http://www.newegg.com/Product/Produc...sPowerSearch=1

This is what I came up with doing a power search on newegg (14 expansion slots).

Edit: Crap didn't consider it would have no IO options. Not sure if you are going to find a straight forward option. I mean the 10 (or 3 extra) slots on ATX cases are already pushing past common uses cases by a lot.

Width is a major concern as well. Think of it as a tower on its side. You have a max height that Corsair, CM, and everyone else can ignore. So it's going to be pretty damn hard to fit an ATX IO panel and 12+ slots on a rack width.

You should look at Tower options. This being the only 12 slot one I found on newegg. http://www.newegg.com/Product/Produc...82E16811112387

Again the big concern is ATX boards. If you could find a board with no IO panel to use for one of the cases from the first search and you should be fine.

 

[frowertr]

While those cases would be able to fit 6 GPUs, how would the power be handled?

That really seems to be the problem I am running into. I need more than 1600W (most powerful ATX PSU) of power for 6 GPUs so I can't just use one PSU to power the entire server. I need either a server 1U/2U 1800W+ PSU with the appropriate PCI-E connectors or dual ATX PSUs.

 

[frowertr]

Oh, I see what you mean when you say no IO panel. I didn't even think of that...

Crap.

Here is the product I am looking at for a 4 GPU setup: http://usa.chenbro.com/corporatesite/products_detail.php?sku=162

It is a tower chassis that can also be rack mounted.

 

[Topweasel]

Oh, I see what you mean when you say no IO panel. I didn't even think of that...

Crap.

Here is the product I am looking at for a 4 GPU setup: http://usa.chenbro.com/corporatesite/products_detail.php?sku=162

It is a tower chassis that can also be rack mounted.

4GPU's is a piece of cake. Because that fits within most ATX guidlines in terms of expansion slots and they can pull it off on a rack mount setup. But That's the problem here a rack has a limited width that is the same everywhere and just by looking at the 14 slot cases you can see that anything more than about 9 starts cutting into the space for the IO plane.

About the only hope would be case with a server PSU that ran over the top of the mobo and then used the space for the PSU for the mobo. I am guessing most HPC and other Tesla servers that support 6 GPUs are basically barebone extenders that just have 1 data cable that plugs into a primary system that has less and has the IO ports.

 

[frowertr]

Here is the back of one of the Tyan 8 GPU Tesla servers:

I didn't realize, until you brought it up, that there is no room for a standard ATX IO panel. Looks like they are using some type of backplane setup to get very basic IO port access.

Here is the inside of that same server:

It does look like the PSU sits above that motherboard.

Well damn. I guess I am out of luck using an ATX MB in a server chassis to run 6 GPUs. I'd have to use some type of other MB form factor.

 

[SithSolo1]

They dont make server racks but caselabs has a few cases with at least 10 pcie slots, atx i/o cutouts, and multiple psu support. You could always use a riser cable on the last card and mount it without a pci bracket.

 

[Topweasel]

They dont make server racks but caselabs has a few cases with at least 10 pcie slots, atx i/o cutouts, and multiple psu support. You could always use a riser cable on the last card and mount it without a pci bracket.

That's the point. Power and max slots isn't the issue. Max width of a Server rack is the issue. Another solution would be to get something squarish but Tall (like the caselabs) and put it onto a rack tray. But then it would be much much much taller than 4U.

 

[frowertr]

As topweasel points out, its the width of the rack server that I wasn't taking into account when requesting a rackmout ATX case with 12 single width PCI-E slots or 6 dual width PCI-E slots. There simply isn't enough room (width wise) in a server rack to accomodate more than 8 PCI-E slots in the ATX form factor.

And after dong more research, there isn't an ATX motherboard made with more than 7 or 8 single width PCI-E slots anyway.

I'm going to head over to the motherboard sub-forum and ask if there are any server form factor motherboards that will have the amount of PCI-E slots I need.

Thanks for the help guys.