quad-GPU cruncher, mark 2

[VirtualLarry]

Thinking about possibly setting up another quad-GPU rig. Unsure whether I'll overhual my existing one, or build a new one to run in parallel.

(For those that don't know, I built a quad-GPU cruncher around a K9A2 Platinum mobo from MSI, a dual-core AM2 CPU, and four PCI-E 9600GSO 96SP cards, in an Antec 300 case full of fans.)

The old cruncher isn't exactly a slouch, but the GTX460 cards are getting cheap (their last hurrah?), and they are like 3x faster than the 9600GSO cards.

Here's the parts list:

MSI 890FXA-GD70 AM3+ AMD 890FX SATA 6Gb/s USB 3.0 ATX AMD Motherboard
http://www.newegg.com/Product/Produc...82E16813130274
$189

This board has 5 PCI-E x16 physical slots, and is in an acceptable layout for using four dual-slot graphics cards like the GTX460.

There is a newer board, the 990FX-GD80, which doesn't have the right PCI-E slot layout for four GPUs. Avoid that one.


AMD Phenom II X6 1055T Thuban 2.8GHz Socket AM3 125W Six-Core Desktop Processor HDT55TFBGRBOX
http://www.newegg.com/Product/Produc...82E16819103851
$150

Hex-core, to feed the quad-GPUs, and have some CPU left over for CPU crunching projects.
The potential to overclock it is there too.

PNY VCGGTX4601XPB GeForce GTX 460 (Fermi) 1GB 256-bit GDDR5 PCI Express 2.0 x16 HDCP Ready SLI Support Video Card
http://www.newegg.com/Product/Produc...82E16814133428
$130 + 7 ship

Need four of these cards.

Rosewill BLACKHAWK Gaming ATX Mid Tower Computer Case, come with Five Fans, window side panel, top HDD dock
http://www.newegg.com/Product/Produc...82E16811147107
$100

This case has support for eight expansion slots, which is a requirement for four dual-slot GPUs.
Feel free to substitute other cases that also support 8 slots.
ThermalTake Armor
some Lian-li
Antec 100

Edit: Antec 100 case seems to be discontinued at Newegg. That sure was a short lifespan. I was really excited when that case came out, because it had the cooling of an Antec 300, but with eight expansion slots, for four GPUs.
Sigh.

 

[somethingsketchy]

Impressive array of parts you got. I'll throw in the recommendation of a Rosewill Thor (gaming) case. I just got it for Christmas and so far I'm liking the additional space. It's spec'ed to an XL-ATX format with 10 PCI slots.

Currently I only have eight slots (mainly because of the ATX motherboard), with only one GPU, so I can't verify how well it does with multiple GPUs. That said, there is already a 230mm fan, or the possibility of 4x 120mm/140mm fans to be installed.


http://www.rosewill.com/products/1699/productDetail.htm - manufacturer website

 

[blckgrffn]

That sounds pretty swanky to me!

Power supply? I am curious at what it is going to take to keep that beast fed...

That also seems to be the best price on the 460 I have seen in a while...

 

[Uppsala9496]

Would take something like this to power it?
http://www.newegg.com/Product/Produc...82E16817151109

 

[Uppsala9496]

Oh, and for the record, the 1055t overclocks nicely. With no effort whatsoever I have overclocked 2 of them to 3.5. I could go higher I am sure if I tried. I just don't have the patience right now.

 

[blckgrffn]

Would take something like this to power it?
http://www.newegg.com/Product/Produc...82E16817151109

Yikes at the price, but yeah, that is what I was wondering.

 

[Rudy Toody]

I use this in my new build.

It is excellent! It should handle your new rig without missing any beats!

 

[blckgrffn]

I use this in my new build.

It is excellent! It should handle your new rig without missing any beats!

I demand details about your new build.

Short of details, I demand extremely vague hints. Or bumps to random threads that detail said build one component at a time.

 

[Sunny129]

nice Larry...i'm excited to see this build come together!

I demand details about your new build.

Short of details, I demand extremely vague hints. Or bumps to random threads that detail said build one component at a time.

...or a build thread like VirtualLarry's would be nice

Fred, i think many of us would definitely be interested to see what went into that build!

 

[ZipSpeed]

That build is a beast! I have the older board (790FX-GD70) in my cruncher 3 rig which is capable of running 4 GPUs but I never did build it with that many video cards. I wouldn't know where to put that box with the amount of heat it would generate. Perhaps one day I will update the rig if I can find a cool place short of sticking it outside.

 

[Rudy Toody]

I demand details about your new build.

Short of details, I demand extremely vague hints. Or bumps to random threads that detail said build one component at a time.

1) Enermax MaxRevo 1350W psu
2) ASRock 890GX Extreme4 R2.0
3) Corsair 16GB DDR3 RAM
4) OCZ 60GB SSD
5) Antec 900 case
6) AMD 1100T x6

I was building toward the release of Bulldozer. Went with the 1100T instead.

I haven't decided if I should put 3 gpus into the box.

 

[Sunny129]

That build is a beast! I have the older board (790FX-GD70) in my cruncher 3 rig which is capable of running 4 GPUs but I never did build it with that many video cards. I wouldn't know where to put that box with the amount of heat it would generate. Perhaps one day I will update the rig if I can find a cool place short of sticking it outside.

i have the 790FX-GD70 in one of my rigs as well, though i didn't get it with the intentions of crunching with 4 GPUs on it. rather i got it with the intention of running 2 or 3 crunching GPUs and 1 diaplsy GPU. right now i'm only crunching with a single GTX 560 Ti, while the display is being run by a GT 430. i'll add a 2nd GPU cruncher soon, and i'll probably have to replace the GT 430 dedicated display GPU with a true single slot GPU if/when i add a 3rd GPU cruncher.

1) Enermax MaxRevo 1350W psu
2) ASRock 890GX Extreme4 R2.0
3) Corsair 16GB DDR3 RAM
4) OCZ 60GB SSD
5) Antec 900 case
6) AMD 1100T x6

I was building toward the release of Bulldozer. Went with the 1100T instead.

I haven't decided if I should put 3 gpus into the box.

i assumed that decision had been made by the time you decided to go with a 1350W PSU lol...i don't know if it'll break the bank, but on behalf of all TeAm members, please load her up with GPUs :biggrin:.

btw, have you used a kill-a-watt meter to see how much power the system draws in its current configuration? i would imagine that without GPUs, the system may be drawing less than 20% of the PSU's max output (even with the CPU crunching under full load), and as such, is probably not achieving its 80 Plus Gold efficiency rating at that low a level of power consumption.

 

[blckgrffn]

i assumed that decision had been made by the time you decided to go with a 1350W PSU lol...i don't know if it'll break the bank, but on behalf of all TeAm members, please load her up with GPUs :biggrin:.

btw, have you used a kill-a-watt meter to see how much power the system draws in its current configuration? i would imagine that without GPUs, the system may be drawing less than 20% of the PSU's max output (even with the CPU crunching under full load), and as such, is probably not achieving its 80 Plus Gold efficiency rating at that low a level of power consumption.

I second this! Might as well get a couple of GTX 570s!

Sunny, what do your 1090t's pull on just CPU crunching? I've got a new build at home (which I'll spin up a thread for at some point - seems like I have enough going here right now to be borderline spamming) with a 1090t and I am curious

 

[Sunny129]

I second this! Might as well get a couple of GTX 570s!

Sunny, what do your 1090t's pull on just CPU crunching? I've got a new build at home (which I'll spin up a thread for at some point - seems like I have enough going here right now to be borderline spamming) with a 1090t and I am curious

i'll PM you the details so as not to hijack this thread any further (when i figure it out)...

 

[blckgrffn]

i'll PM you the details so as not to hijack this thread any further (when i figure it out)...

Sounds good - but a baseline on the x6 power consumption could be valuable even for this thread

What do you mean I could go read a review? What fun would that be?

http://www.anandtech.com/show/4048/...455-phenom-ii-x2-565-and-phenom-ii-x6-1100t/6

Hmmm... that doesn't make a lot of sense to me. Was the 1100t a different stepping than the 1090t or something?

http://www.cpu-world.com/CPUs/K10/A... Edition - HDE00ZFBK6DGR (HDE00ZFBGRBOX).html No? Maybe just different conditions for Anandtech's test?

Anyway, account for ~150 Watts from the wall when looking at a stock clocked system with an x6 - I would guess they stay under 200W when OC'd. Add GPUs on top of that and you should be set. A reasonable x6 + 2 GPU setup should be good to go with a ~750W power supply, eh?

And I would appreciate that PM in any case

 

[Sunny129]

forget about the PM - i completely misunderstood what you were asking me...i thought you wanted a PPD figure. now i realize we're still talking about power draw. you're absolutely right that it could be valuable information here.

at any rate i lost that information unfortunately in a recent reformat, and to make things worse i have a GTX 560 Ti in the rig now that uses power whether its idle or loaded...and i'm too lazy to yank it. nevertheless, i'll shut her down sometime this evening to connect the kill-a-watt meter and measure a few scenarios, including the one you asked for

btw, i find those 1100T power consumption figures odd too...or is it the 1090T figures that are strange?

 

[Sunny129]

Anyway, account for ~150 Watts from the wall when looking at a stock clocked system with an x6 - I would guess they stay under 200W when OC'd. Add GPUs on top of that and you should be set. A reasonable x6 + 2 GPU setup should be good to go with a ~750W power supply, eh?

that's about right in line with what i found. but before i post the numbers, consider this disclaimer: neither the GTX 560 Ti GPU nor the GT 430 GPU were removed from the system while testing the CPU's power draw under load. the GTX 560 Ti GPU is a dedicated 24/7 cruncher, while the GT 430 is a dedicated display GPU. so without further ado...

- System Idle: 116W
- CPU crunching only (100% load, 6 Einstein@Home S6GC or FGRP1 tasks): 207W
- GPU crunching only (~85% load*, 3 Einstein@Home BRP4 CUDA tasks): 240W
- CPU & GPU crunching (4 Einstein@Home S6GC or FGRP1 tasks, 1 Test4Theory@Home task, and 3 Einstein@Home BRP4 CUDA tasks): 320W

*while the GPU averages only 85% load while crunching 3 tasks simultaneously, it unfortunately cannot handle a 4th task. while there's headroom with the compute units, my GPU memory is limited to 1GB. if each task used a max of ~250MB of memory, i could get away with running a 4th task...but b/c my GPU's memory usage often spikes just north of 800MB while running 3 tasks, i'm sure i'd be asking for trouble if i tried to run a 4th.

now based on the last figure, i could probably add a GTX 460, maybe even another GTX 560 Ti, and be fine with my 650W PSU. but my other rig, which has an HD 5870 GPU dedicated to crunching and uses onboard graphics to run the display, i believe uses almost 100W more (i'll have to double-check that figure, as i'm just recalling it off the top of my head). if i were to add another 5870 or a 6950/6970 to that machine, i'd be foolish not to upgrade its PSU to a 750W unit.

 

[VirtualLarry]

Before you folks get too excited, that build is what I was dreaming up. Not ordered. Still working out the possible funding.

And the PSU I already have, it's an ABS Silver 1050W PSU. List price is an insane $350, but I picked it up when Newegg was clearing them out for $130. It's supposedly a single-rail version of the Enermax 1050, which got really good reviews from JonnyGuru.

It has mostly modular cables, and I think 6 PCI-E power cables. I would of course need 8, so to fix that, it appears that there is a secondary 8-pin EPS12V cable (for use with server boards with dual CPUs). I can get a cable (at least, I've seen them in the past), that convert between EPS12V, and dual 6-PIN PCI-E.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817814010

 

[brownstone]

It seems that it always comes down to that pesky funding bit.

 

[VirtualLarry]

I had some alternative ideas, for a combo cruncher/gamer/main rig.

Phenom II X6 1045T 3GHz Six Core Socket AM3 Boxed Processor
http://www.microcenter.com/single_product_results.phtml?product_id=0382798
$130 + tax, in-store only

(Would actually be picking up two of these if I can, one to go into my existing quad-GPU 9600GSO cruncher, which currently has an AM2 dual-core.)

GIGABYTE GA-990XA-UD3 AM3+ AMD 990X SATA 6Gb/s USB 3.0 ATX AMD Motherboard
http://www.newegg.com/Product/Product.aspx?Item=N82E16813128510
$140

Has three PCI-E x16 slots physically, which are PCI-E 2.0 x8/x8/x4 electrically. Supports both SLI and Crossfire.

I have two Gigabyte GTX460 1GB cards, and I picked up a Galaxy GTX460 768MB card at BestBuy for $112 + tax.

I already have this case:
Rosewill BLACKHAWK Gaming ATX Mid Tower Computer Case, come with Five Fans, window side panel, top HDD dock
http://www.newegg.com/Product/Product.aspx?Item=N82E16811147107
(Currently $10 off, with free shipping, this is a great case and that's a good deal!)

Has 8 expansion slots, so I can fit a double-wide graphics card into the last physical slot on an ATX mobo. Also has front-panel USB3.0, which that motherboard supports.

Already have this PSU:
ABS SL series SL1050 1050W Continuous @50°C ,80 PLUS SILVER Certified,Modular Cable Design,Single 12V Rail,ATX12V v2.3/ EPS12V v2.92,SLI Ready,CrossFire Ready,Active PFC"Compatible with Core i7, i5" Power Supply
http://www.newegg.com/Product/Product.aspx?Item=N82E16817814010\
Cost me $130.

Has three pairs of PCI-E 6-pin power connectors. Single-rail (which I am slightly nervous about, I would feel safer with a multi-rail PSU if the **** ever hit the fan electrically).

Already have plenty of 4GB DDR3-1333/1600 DIMMs, 1.5v.

So I figure I can build this rig, with three GTX460 cards, two 1GB models in SLI for gaming, and a third 768MB model for PhysX, and hopefully all three can also run as CUDA cards. I will initially be putting in 16GB of RAM, but might bump that up to 32GB, as the budget and prices of 8GB DIMMs allow.

I figure this will be a decent gaming rig, IF I can overclock the hex-core CPU to 4.0Ghz, since single-threaded speed is still dominant in games. Multi-threaded stuff like BOINC should immediately benefit from the hex-core.

I do about 99% distributed computing, and about 1% gaming. So this should be a decent rig hopefully.

I'm working on getting the funds together, and scheduling a trip to Microcenter. I already have someone lined up who wants to buy the free motherboards that you can get in a bundle with the 1045T.

In the future, if Piledriver improves IPC compared to Bulldozer, and it becomes an affordable 8-core CPU, then I might upgrade to it. The mobo is AM3+.

 

[VirtualLarry]

Hmm, I have some Win7 Pro licenses, was thinking of using one of them on this machine, esp. if I upgrade to 32GB of RAM, I will need 7 Pro.

Which opens up the possibility of remote desktop usage of my machine. I probably won't expose RDP directly over the internet, I already have VPN support via my router.

HOWEVER, last time I tried that, on XP Pro, with Folding@Home running in the background, once you switch over to RDP, the Windows user workstation switches to emulated video and sound drivers, which for some reason cuts Folding@home off from the CUDA GPUs.

Is this still true in 7 Pro? Or is it now possible to run CUDA apps, and use RDP? I would REALLY like to find the answer to that. I would also REALLY like to have RDP support on my main desktop, if I need it.

Edit: Since I have the discs, I suppose I could do a trial install and test it out.

Edit: BAD NEWS. Remoting in causes BOINC to lose access to the GPU. I consider this to be a defect in the NV driver model. Surely there must be a way to provide CUDA services through an independent interface, and not via the video driver. Or I could be wrong about that.

So scratch that, I cannot compute as well as remote in. One or the other.

 

[somethingsketchy]

I would be very careful and verify the 1045T can actually support and utilize all 32GB of RAM. As I have sadly discovered earlier this week (researching a LGA2011 build) that even the top end $999 processor doesn't support more than 16-24GB of RAM.

I haven't seen any info refuting or confirming my suspicions (http://shop.amd.com/US/All/Detail/Processor/186973), but nonetheless, proceed with caution.


EDIT: I stand corrected. The 3960X DOES support 32GB (http://ark.intel.com/products/63696...rocessor-Extreme-Edition-(15M-Cache-3_30-GHz)). I'm thinking the X6 might as well. If not, there's always the six core bulldozer

 

[blckgrffn]

Hey -

VNC. Windows Live. Both of these are valid ways to remote in and respect the GPU. Windows Live is super-de-duper slick IMHO, you can get to it anywhere with a browser.

Because of how RDP works it kills any graphics drivers access to hardware - this is by design and will not be resolved via drivers.

 

[VirtualLarry]

http://www.youtube.com/watch?v=X-5OTdwMbX4

Video review/overclock of an FX CPU @ 5Ghz, on a 990XA-UD3. He says that there is some sort of throttling issue with this motherboard, that prevents using stress-test programs like LinX, otherwise it clocks way, way down.


That doesn't sound right to me, for a board with VRM heatsinks, and a supposed 8+2 VRM scheme.

Edit: Even worse, later in the video, he shows CPU-Z, and it lists the TDP for that CPU at 95W.

Edit: He later has it at 285Mhz base clock, which is good news. I calculated you would need 296Mhz base clock to run the 1045T @ 4.0.


This link describes the problem, seems to be related to a power cap on AMD's TurboCore CPUs.
http://www.amdzone.com/phpbb3/viewtopic.php?f=532&t=138895

This seems to say that it was fixed in the F10g BIOS?
http://www.overclockers.com/forums/showthread.php?t=689657&page=2


Edit: I found a comparable Asrock mobo for $110.
ASRock 970 EXTREME4 AM3+ AMD 970 SATA 6Gb/s USB 3.0 ATX AMD Motherboard with UEFI BIOS
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157262

Supports SLI and CF. Has three physical PCI-E x16 slots, PCI-E 2.0 x8/x8/x4. Unfortunately, if you use double-wide GFX cards in both primary PCI-E x8 slots, it blocks both PCI slots. There is no free PCI slot if you are going to do SLI.

Hmm, this power limit throttling seems to be a theme among 9xx-series AMD mobos.
http://www.overclockers.com/forums/showthread.php?t=696708

Now I'm wondering if I should go with my original choice, the Gigabyte 990XA-UD3, if they found a patch for the throttling.

Edit: Recent thread about Asrock board here. It appears the issue is not solved yet for the Asrock board?
http://forums.tweaktown.com/asrock/...120-keeps-throttling-im-not-overclocking.html

 

[Uppsala9496]

I was unable to get the 1045T from microcenter the other day with the free mobo. Kept adding all of the combinations into the cart and it never put in the mobo for free.
Settled on the 960T for $118 after tax.
My bios shows 6 cores, however ubuntu only shows 5. So, 1 extra core unlocked. I did oc it to 3.6ghz. Right now my ram is a little weak so I will have to get some better stuff to see if that helps the overclock.