8x GTX TITAN Build

[blueberryjudy]

In my current research lab, we use machines that have 4x or 8x NVIDIA K20 cards. The motherboard they use are supermicro/tyan motherboards (which can support the 4x/8x PCIE cards).

They work and they are great... however I want my personal dev machine for home usage as well. I don't need the K20 features, so I can do with 8x GTX TITANs; which are faster as well (e.g., higher clockspeed and more cores).

I tried searching on newegg.com for CPU + motherboard combinations, but I can't find any that suit my needs? I don't know what to buy. Do I really need to go with tyan? I can't even find the model my lab uses. I was hoping Gigabyte or ASUS would have 8x PCIE slots for me? What about powersupply? What kind would I need as well?

Cost is more "important" than and stability (e.g., I am using my personal funds). I imagine I would use this machine for 1.5 years before needing to upgrade again (e.g., whenever NVIDIA comes up with new chips we upgrade).

Much thanks all.

 

[SKORPI0]

. I don't need the K20 features, so I can do with 8x GTX TITANs; which are faster as well (e.g., higher clockspeed and more cores).

You'll need a motherboard that is at least PCIE 3.0 16X for the GTX Titan ( GK110 GPU).

 

[vailr]

What is the model number of the Supermicro motherboard (at your workplace) that has the 8 video card slots? If I'm understanding what you're trying to say?
However, AFAIK GTX Titan cards each require 2 back panel openings, so any more than 4 video cards in a system is not likely to be possible.

 

[blueberryjudy]

What is the model number of the Supermicro motherboard (at your workplace) that has the 8 video card slots? If I'm understanding what you're trying to say?
However, AFAIK GTX Titan cards each require 2 back panel openings, so any more than 4 video cards in a system is not likely to be possible.

One of our boards is Tyan S7015

 

[vailr]

The motherboard they use are supermicro motherboards (which can support the 8x PCIE cards).

Model number is?

 

[blueberryjudy]

Model number is?

See previous post, my mistake-- we're actually using Tyan for the 8x Tesla servers.

 

[lehtv]

You said cost important... can u state exactly how much you're looking to pay

 

[Knavish]

Since this problem is apparently very parallel, why not put 4X titans in two different computers -- this would make it much easier to acquire the computer parts. It looks like titans can consume 250-300W a piece. Add this to the computer and you need a power supply that is rated for over 2kW. You have to buy a big, expensive (and possibly loud) server chassis to support this. I don't know too many people who want one of these at home.

Are you doing high frequency trading? Normally when I hear about research labs I start thinking of graduate students, but I haven't met too many graduate students who want to spend $8000 on video cards so they can work from home...

 

[blueberryjudy]

Since this problem is apparently very parallel, why not put 4X titans in two different computers -- this would make it much easier to acquire the computer parts. It looks like titans can consume 250-300W a piece. Add this to the computer and you need a power supply that is rated for over 2kW. You have to buy a big, expensive (and possibly loud) server chassis to support this. I don't know too many people who want one of these at home.

Are you doing high frequency trading? Normally when I hear about research labs I start thinking of graduate students, but I haven't met too many graduate students who want to spend $8000 on video cards so they can work from home...

No.. I want them all on one machine, unless you want to buy me InfiniBand‎ which is even more expensive. Computation is parallelized, but latency is very important between cards as well.

Basically I guess my question is, what is the cheapest (Haswell? Sandy Bridge?) 8x PCIE motherboard on the market that supports GTX Titans (e.g., double slots) and recommend me a power supply and chassis to go with it.

 

[Knavish]

No.. I want them all on one machine, unless you want to buy me InfiniBand‎ which is even more expensive. Computation is parallelized, but latency is very important between cards as well.

Basically I guess my question is, what is the cheapest (Haswell? Sandy Bridge?) 8x PCIE motherboard on the market that supports GTX Titans (e.g., double slots) and recommend me a power supply and chassis to go with it.

I do not believe a single-socket motherboard exists with 8 physical PCIe-x16 slots. When you think about it, you need room for 8 double-width cards. This would not even fit on an ATX motherboard since ATX does not have room for 16 single-width cards.
(edit)
Looking at the Wikipedia Article on ATX, various motherboard companies have developed HPTX and XL-ATX to accommodate 4x double-width video cards. A standard ATX board has room for 7 slots. These new sizes have room for up to 8-9 slots, and are only equipped with 7. This allows the board to fit 4x dual-slot video cards. A board that can take eight double width cards would be quite bizarre looking since it would literally be twice as "wide" as a normal ATX motherboard. This may introduce EMI and latency issues since the motherboard traces would be twice as long as normal.
(/edit)

I briefly looked at Supermicro's GPU servers and I don't think any of them support more than 6 slots, unless you go with a blade system, which isn't a single computer.

One question: does your application require full x16 bandwidth on each of the titans, or can you run at x8 bandwidth? Even the X79 boards with seven physical PCIe x16 slots will only run four of them at x16 bandwidth.

 

[blueberryjudy]

I do not believe a single-socket motherboard exists with 8 physical PCIe-x16 slots. When you think about it, you need room for 8 double-width cards. This would not even fit on an ATX motherboard since ATX does not have room for 16 single-width cards.

I briefly looked at Supermicro's GPU servers and I don't think any of them support more than 6 slots, unless you go with a blade system, which isn't a single computer.

You are correct, we use dual socket motherboards. Our 8x GPU servers uses Tyan, our 4x GPU servers uses supermicro.

 

[Atreidin]

You can get a quote from these guys. I'd be interested in the pricing.

http://www.renderstream.com/products/high-performance-computing/

 

[Knavish]

You can get a quote from these guys. I'd be interested in the pricing.

http://www.renderstream.com/products/high-performance-computing/

Lol the system pictured on their site looks identical to the Tyan S7015 system that blueberryjudy mentioned earlier in the thread. I wonder if anyone makes a similar system with updated hardware. I haven't seen a new system listed that supports more than 4x dual-slot GPUs. I wonder if everyone is moving toward blades with an integrated infiniband backbone...

 

[blueberryjudy]

Lol the system pictured on their site looks identical to the Tyan S7015 system that blueberryjudy mentioned earlier in the thread. I wonder if anyone makes a similar system with updated hardware. I haven't seen a new system listed that supports more than 4x dual-slot GPUs. I wonder if everyone is moving toward blades with an integrated infiniband backbone...

I am guessing from the responses that I would need to go through these system builders and I can't buy my own motherboard + CPU from newegg?

Edit: What are my options for 4x GPUs then?

 

[monkeydelmagico]

Asus rampage IV extreme

 

[aigomorla]

No.. I want them all on one machine, unless you want to buy me InfiniBand‎ which is even more expensive. Computation is parallelized, but latency is very important between cards as well.

Basically I guess my question is, what is the cheapest (Haswell? Sandy Bridge?) 8x PCIE motherboard on the market that supports GTX Titans (e.g., double slots) and recommend me a power supply and chassis to go with it.

THERE IS NO PCI-E 3.0 which can handle the total IO output of 8x titans in parallel...
There is no PSU which can handle more then 4 titans at one time...
You would need 3 PSUs.... 2 x 1200W for your videocards... and 1 just for the system.

And then u run into the problem with current at your house and the breaker
(welcome to the pains of insufficient power capacity which i just ended up getting out of, because of insufficient planning... DOE!)

Having a system pull 2400W+ underload will POP your house curcuit breaker unless u have something greater then 20amps!
Did u even calculate down to the pull such a system would cause on your house electrical?
That means to even maintain this system on a ON, you would need to tie it to a 30AMP breaker on a 120V US standard from the house.... or have 2 different circuits feed into that 1 PC.
And u cant just replace the breaker switch either as the wires have to be raited to handle 30amps or you risk a major fire hazzard.... its a big mess once u get into power draw of that nature.
I hope your considering putting a 220V converter to help you with the load if ur thinking about more then 1 PC of this type of nature.

Also there is no case for this type of system either... you need a total of 16 PCI slots which i cant think of ANY case that offers it including server racks.

Im sorry to break it to you but your not going to get away cheap on ANYTHING u asked for, and it wont be smooth as ur expecting it to be because your not using original hardware designed for the job ur seeking.

Also correct me if im wrong.. but what ur trying to do is make a gaming card into a workstation card.... and if im also not mistaken.. Nvidia prohibits u from doing that without a flash modification on the titan, which then u risk the chance of killing your titan.
(reference the many headaches people ran into flashing geforce -> Quadro as your trying to do the same thing...)

does your software work with titans and nvidia geforce drivers?
I would verify this b4 even thinking of a system on this scale.


Basically...

1. Find out of software works for Geforce Titans.
2. Find out what the total amp circuit breaker is to the room/office your locating the pc at.

These 2 should be cleared before you even think about parts for the system.

 

[Knavish]

THERE IS NO PCI-E 3.0 which can handle the total IO output of 8x titans in parallel...
There is no PSU which can handle more then 4 titans at one time...
You would need 3 PSUs.... 2 x 1200W for your videocards... and 1 just for the system.

Also there is no case for this type of system either... you need a total of 16 PCI slots which i cant think of ANY case that offers it including server racks.

The Tyan S7015 system listed earlier in this thread does support 8X dual slot cards. It must be a ridiculous looking system -- I don't think it could even fit in a standard rack.

 

[aigomorla]

The Tyan S7015 system listed earlier in this thread does support 8X dual slot cards. It must be a ridiculous looking system -- I don't think it could even fit in a standard rack.

i was under the impression that the titan gtx utilized all 16x bandwith.

The tests which i think we all are thinking about were based on a 680GTX.
The Titan GTX is 50% faster then a 680GTX i believe... also has way more process handlers, hence will show a bigger bottle neck on PCI-E 8X.
^ i am guessing... please dont take me on fact...

I could be wrong... but if i had to boil it down.... if forced to used 8X id rather use 780GTX over a full bloated titan as it would be a smaller PCI-E 8X vs 16X gap then a titan.. and 8X performance should almost be only tiny bit lacking.

And u know what i stand corrected.. i blame u darn bitcoin miners for making things like this:

More examination tells me its a prefab'd package from Tyan so im going with specialized layout due to its customized power distribution node... meaning u cant build it... The box / board / PSU should come as one package, and u add the cpu + ram + gpu.

It does look nice... and OP it maybe your only option.... u still need to verify your office/house breaker will be handle 20Amp+ or have a certified electrician install a 30Amp to feed the computer SOLO.

 

[blueberryjudy]

Hi guys, thanks for all the responses. Seems like 8x GTX Titan system may be out of reach for my personal machine, I am now going to aim for 4x GTX Titan system.

To answer some questions:

No I don't need the full 16x PCIE 3.0 bandwidth, x8 bandwidth is quite fine.

No, I can not and will not split across multiple machines-- the reasons are quite complex (related to my research) which I can not explain.

Yes, the GTX Titan w/ CUDA drivers will run my CUDA code-- there is no need for flashing.

So I guess the question is now, what motherboard should I get for the 4x GTX Titan system, what power supply and case?

Thanks!

 

[vailr]

Price: $4,739

Tyan B7059F77AV6R-2T 4U Server DP Xeon E5-2600 LGA2011 Register DDR3 6x2.5"/4x3.5" SATA Hot-Swap 10GbE PCIe R2400W 80PLUS

http://www.acmemicro.com/Product/13...kValue=6|2&Crits_CheckValue=31|4&pager_index=

 

[blueberryjudy]

Price: $4,739

Tyan B7059F77AV6R-2T 4U Server DP Xeon E5-2600 LGA2011 Register DDR3 6x2.5"/4x3.5" SATA Hot-Swap 10GbE PCIe R2400W 80PLUS

http://www.acmemicro.com/Product/13...kValue=6|2&Crits_CheckValue=31|4&pager_index=

No cheaper option out there?

The machine will probably be tossed after 1.5 years to upgrade to the latest GPU/CPU.

I was hoping newegg.com had some 4x PCIE motherboards easily available, but I guess I need to go full fledge?

 

[mfenn]

You can get Haswell or Ivy Bridge-E motherboards that support 4 GPUs. Does your application need a ton of CPU cores as well or is a quad enough?

 

[monkeydelmagico]

No cheaper option out there?

Sort of. Comparing business class equipment with enthusiast gear is apples and oranges but here's an option:


ASUS Rampage IV Extreme $429.-
http://www.newegg.com/Product/Produc...82E16813131802

Intel Core i7-4820K $309.-
http://www.newegg.com/Product/Produc...82E16819116940

G.SKILL Ripjaws Z Series 64GB $600.-
http://www.newegg.com/Product/Produc...82E16820231508

Lepa G 1600 power supply $300.-
http://www.newegg.com/Product/Produc...epa%20g%201600

Corsair Obsidian Series 900D case $300.-
http://www.newegg.com/Product/Produc...82E16811139019

Thats about $2,000. total. Then there is the 4 Titans at $1k apiece. wow, I'd love to be there when all those boxes arrive! Best christmas ever.

NOTE: Your answer to Mfenn's question above could make my suggestion a no go if the workload needs more cores. Also went to max on memory but your requirements may not need all that. Some cost savings could be had with a 32gb kit instead.

 

[Golgatha]

i was under the impression that the titan gtx utilized all 16x bandwith.

The tests which i think we all are thinking about were based on a 680GTX.
The Titan GTX is 50% faster then a 680GTX i believe... also has way more process handlers, hence will show a bigger bottle neck on PCI-E 8X.
^ i am guessing... please dont take me on fact...

I could be wrong... but if i had to boil it down.... if forced to used 8X id rather use 780GTX over a full bloated titan as it would be a smaller PCI-E 8X vs 16X gap then a titan.. and 8X performance should almost be only tiny bit lacking.

And u know what i stand corrected.. i blame u darn bitcoin miners for making things like this:

More examination tells me its a prefab'd package from Tyan so im going with specialized layout due to its customized power distribution node... meaning u cant build it... The box / board / PSU should come as one package, and u add the cpu + ram + gpu.

It does look nice... and OP it maybe your only option.... u still need to verify your office/house breaker will be handle 20Amp+ or have a certified electrician install a 30Amp to feed the computer SOLO.

Probably would want to drop a 240V line instead. Better efficiency, dedicated circuit, and gives you some headroom for the system. That would be a beast!

 

[monkeydelmagico]

Probably would want to drop a 240V line instead. Better efficiency, dedicated circuit, and gives you some headroom for the system. That would be a beast!

If the OP goes with a 4x titan setup this would be unnecessary. Most current households use 20 amp systems. They should double check first. Assuming they have 20 AMPS x 120 volts = 2400 watts. You can only load up to 80% of the stated rating for extended periods so 1920 watts. Thats plenty of power as long as it's the only thing plugged into the outlet.