Dual-cpu system for fractal rendering (integer)

[Dinkydau]

Hello everyone.

I was thinking of buying the parts below and already contacted shops trying to get the motherboard and CPUs, in the Netherlands, which makes that kind of difficult. I tried to keep the cost around €2000, for which you can usually get the same here as for $2000 in the USA. Here's the list:

motherboard: Asus KGPE-D16, 2× g34 socket
CPU: 2× AMD Opteron 6272, 16-core
RAM: 4× Kingston DDR3, 1333 MHz, 4GB kits (2×2048 MB)
GPU: (my current) nVidia 8800 GTS, 512 MB
storage: (my current) 750 GB HDD and 2 TB HDD
sound card: some cheap sound card should be enough
cooler: noctua nh-u9DO a3, amd opteron, socket g34
power supply: Corsair 750 watt ax750 gold
case: Cooler master HAF932

I will be using this system for fractal rendering in fractal extreme and apophysis, fractal software for windows. Basically:

• apophysis: a lot of floating-point operations
• fractal extreme: a lot of integer operations

I'm planning to use fractal extreme more than apophysis and the calculations are much more intense at high zoom levels, so integer performance is more important here.

The CPU that got my interest immediately when I began looking for parts was the AMD opteron 6272, the least expensive of the three 16-core versions of the interlagos CPUs, which have 16 integer cores and 8 float cores. I wish they had 16 float cores as well, but can't have everything for $540 on newegg.com, a very good price. No other high-end server CPU is that cheap and it does perform well in benchmarks by anandtech. Intel appears to be insanely expensive, so I didn't spend much time on that. I don't think the faster opteron 6274 or 6276 are worth the extra money for the small performance gain.

I'm currently rendering an image that uses 960-bit calculations, to give you an idea of the precision required for deep fractal renders. AVX-instructions could speed-up this kind of work in the future, which the interlagos CPUs support, so they're future-proof.

There are 2 people I know who have respectively an intel 2600k and an amd fx-8150 (same bulldozer architecture as interlagos) who wanted to help me, and the programmer of fractal extreme did a benchmark with his intel 3930k, all in fractal extreme. The benchmarks in this particular program:

• intel i7 2600k as reference
• amd fx-8150 bulldozer is 35% faster
• intel 3930k is another 10% faster

Intel wins at speed here, but not at price, which isn't different for server CPUs (see anandtech benchmarks). Amd opteron 6272 doesn't have fractal extreme benchmarks yet, but considering intel is just a little better for much more money in the desktop CPU and server CPU competition, the amd opteron 6272 is probably the best deal, and amd wins.

Then with the CPU in mind comes the choice of the motherboard. It needs to have 2 g34 sockets. Asus kgpe-d16 is cheapest for $430 on newegg.com and everyone uses it instead of the alternatives tyan or supermicro, so I guess it must be good.

What do you all think of the CPU-motherboard combination for this purpose?

I am aware that the kgpe-d16 motherboard's bios doesn't always support the AMD opteron 6200 series processors. Information on this:
http://dlcdnet.asus.com/pub/ASUS/mb...enual_QVL/KGPE-D16_Interlagos_Support_new.pdf
I am aware that the speed per core is low, but that's no problem because there's always enough to render.

If you have any ideas, negative or positive, please post.
If you have a 64-bit CPU and OS you can help by running this fractal extreme benchmark program for only a few seconds and post the results:
ftp://ftp.cygnus-software.com/pub/InfprecPerf.exe
I've left out the results of the CPUs already tested in this program but if needed I'll post them later.

Sorry for writing so much

Also if you want to know more about what I do, visit my deviantart and youtube accounts:
http://www.youtube.com/user/Dinkydau00
http://dinkydauset.deviantart.com/

 

[mfenn]

Main problem I see with your plan is that the 932 can't handle the huge EEB board. Something like this Chenbro isn't as sexy as the 932, but it will handle the mobo.

 

[Dinkydau]

Thank you for the reply

According to the cooler master website, the case has room for an EATX motherboard:
http://www.coolermaster.nl/product.php?product_id=5363
Does this make a difference?

(Sexyness is not required, cooling is.)

ATX 305 mm × 244 mm
EATX (Extended) 305 mm × 330 mm (35% wider)

I have modified a picture of the case with an installed ATX motherboard, found here, to see how much room there is:

Seems to just barely fit in terms of size, but not clearly too big.

 

[heymrdj]

Thank you for the reply

According to the cooler master website, the case has room for an EATX motherboard:
http://www.coolermaster.nl/product.php?product_id=5363
Does this make a difference?

(Sexyness is not required, cooling is.)

ATX 305 mm × 244 mm
EATX (Extended) 305 mm × 330 mm (35% wider)

I have modified a picture of the case with an installed ATX motherboard, found here, to see how much room there is:

Seems to just barely fit in terms of size, but not clearly too big.

SSI EEB is not EATX. They are the same size (12x13) but they have different mounting holes. SSI EEB is to the core a server board. EATX is a large ATX board, many times server built, but can also just be large enthusiast boards. SSI EEB gernally already has a back plate mounted, so you're not going to be able to use odd exotic coolers. EEB's are generally built with a front to back air flow in mind to channel air with a shroud over the memory, the processors, the chipsets, and possible any RAID chipsets/VRMS. EATX generally rely on third party coolers for these jobs. The chassis that accept these boards normally reflect this design. (AKA large slow fans in an EATX case and high speed not very quiet high cfm fans on SSI EEB chassis).

 

[Dinkydau]

Thanks for that. I'm going to look into this. Do you, by any chance, have any suggestions on SSI EEB cases with good cooling? I mean, kinda like the Cooler master HAF932. I found the Cooler Master Ultra Tower Cosmos II, but it's hella expensive. Maybe it's a good idea to just use the HAF932 and not use all of the screws, appears everybody else is already doing that anyway (check the cases mentioned in these reviews on newegg.com if you don't believe me).

 

[heymrdj]

I don't have much experience outside of the server side of SSI so I can't really say sorry. Typically It's a Supermicro mainboard with a Supermicro chassis.

 

[mfenn]

Thanks for that. I'm going to look into this. Do you, by any chance, have any suggestions on SSI EEB cases with good cooling? I mean, kinda like the Cooler master HAF932. I found the Cooler Master Ultra Tower Cosmos II, but it's hella expensive. Maybe it's a good idea to just use the HAF932 and not use all of the screws, appears everybody else is already doing that anyway (check the cases mentioned in these reviews on newegg.com if you don't believe me).

I suppose you could do that, but like heymrdj said, an ATX case is not designed to provide the laminar flow that a server board expects. IMHO, if you're going to do it, you might as well do it right.

 

[heymrdj]

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100008044%20600029569&IsNodeId=1&name=Pedestal Here's workstation chassis, you can select any one of them that says SSI EEB for the mainboard support.

Your motherboard link confirms that the main board is EEB, not EATX, so you'll have to use a rackmount server case, or one of those pedastal (tower convertible to rackmount) chassis. As you can see, they all have hotswap high flow front to back fans, along with included shrouding or channeling to ensure the high speed air flows over the board properly.

 

[Dinkydau]

All the EEB-cases on newegg have reviews stating that the cooling is loud. I also doubt it's actually better than all those coolers in the HAF932. I don't mind if the case doesn't officially support the form factor. As long as it works it's good, so what I'm worried about at the moment is the screws. How many of the screws can be used, and which?

 

[heymrdj]

All the EEB-cases on newegg have reviews stating that the cooling is loud. I also doubt it's actually better than all those coolers in the HAF932. I don't mind if the case doesn't officially support the form factor. As long as it works it's good, so what I'm worried about at the moment is the screws. How many of the screws can be used, and which?

Normally only 3 or so screw holes work, the rest you need a tapper of the correct size to do the mother board plate with. Sometimes you get lucky because of the crappy language conversion and a chassis that supports EATX is already drilled out for the most part and sloppily for SSI EEB.

And yes SSI EEB is not quiet. Do you want performance or not? You're sticking two xeons in very close to each other which can dissipate 100W+ of heat a piece. You don't point a 120mm fan at that and call it a day.

EDIT: I see you have opterons sorry. That's 115W a piece when fully loaded, which you said you intended to do. Beyond benchmarking, I can't think of any other load than crunching that will get a processor to TDP. 24/7 230W of heat to disipate just for the processors, let alone ram and chipsets. And that's another thing, dual proc chipsets don't get by with passive cooling unless they have cool ambient temps and strong chassis air flow. Otherwise you're going to need a fan on the chipset cooler. If a board says it is SSI EEB, that's because they are building it in the mindfulness that the chassis will provide forced front to back air flow. The SSI standard goes as far as to dictate where the processor goes to standardize thermal behavior: http://en.wikipedia.org/wiki/SSI_CEB

If you're building a work station, please realise these machines are not your quiet dell off the shelf system (though they are quieter than people first think). These machines are cooled well because the heat is real. OEM's of off the shelf systems build desktops with thermals in mind that show approximately 30% duty cycle. Work stations are built for 80-100%, and have much more thermal capacity than an average desktop. It's like trying to compare the cooling capacity of a Honda Ridgeline to an F-550.

 

[Dinkydau]

Those cases on newegg have in fact only one 120mm fan, while the haf932 has:

140mm Fans
1 x 140mm rear fan
230mm Fans
1 x 230mm front red LED fan
1 x 230mm top fan
1 x 230mm side fan

http://www.newegg.com/Product/Produc...82E16811119160

Then the CPU coolers, according to Newegg the CPU coolers can handle 115 watts.

Thanks to six heatpipes, a performance optimized fin stack and its two premium grade NF-B9 92mm fans, the compact NH-U9DO A3 cools Opteron processors up to a thermal design power (TDP) of 115W at minimum noise levels.

http://www.newegg.com/Product/Produc...82E16835608022
So no problems with those either.

I don't want to bash on EEB-cases, but I really don't understand how their cooling is so superior.

 

[heymrdj]

Those cases on newegg have in fact only one 120mm fan, while the haf932 has:

140mm Fans
1 x 140mm rear fan
230mm Fans
1 x 230mm front red LED fan
1 x 230mm top fan
1 x 230mm side fan

http://www.newegg.com/Product/Produc...82E16811119160

Then the CPU coolers, according to Newegg the CPU coolers can handle 115 watts.

http://www.newegg.com/Product/Produc...82E16835608022
So no problems with those either.

I don't want to bash on EEB-cases, but I really don't understand how their cooling is so superior.

I'm not trying to beat you over the head about this, you're simply not understanding the difference between consumer level and professional level.

First of all, those coolers, you're not hanging two of those off that main board. Do you understand how much weight that is? The boards are designed for small efficient low profile copper coolers (1u/2u) or larger aluminum light weight coolers that relay on air flow from from a nearby chassis fan.

Now on to that point. Look at this: http://usa.chenbro.com/corporatesite/products_detail.php?sku=104 You see where those fans are? Right in front of the CPU towers. Those are Delta fans. 120x38mm to be exact. Very heavy airflow. Very directed airflow. You can't throw ambient air flow at a workstation. You can take those 25mm LED fans all you want in the HAF, but they don't direct and push air. They just waft it around. Bit-tech shows the case temps here: http://www.bit-tech.net/hardware/cases/2008/10/16/cooler-master-haf-932/4

That's just 1 130W processor. You're pushing double that. And double that temperature is no where near acceptable for 24/7 use.

I have experience in this because i mounted dual 5500 Xeons into an Antec 1200. The thermals were worthless. The CPU coolers whined like hell because all I was doing was bringing ambient temps down, I wasn't forcing air where it was needed. The chipset overheated from the I/O load for the 3 graphics cards number crunching. In the end the chassis was a hot box, no where near enough air moving in and out to cool the components. Moved to a Supermicro and all was well.

Of course I work with these things for a living so that helps too. Speaking of that I have an R810 to attend to.

 

[Dinkydau]

Okay, I get it now. The picture shows clearly that there's much better airflow. It has 3 fans there, so I assume they need to be bought separately. I'm taking this into consideration. Now that makes me wonder what those noctua coolers are actually supposed to do. If they're too heavy, why do they even exist?

 

[heymrdj]

The fans typically come with the chassis. As I said alot of times in the workstation field you buy a board/fan/chassis/power supply all together. They work as a unit, with drivers provided to monitor sensors all over the chassis. There are, however, tyan, supermicro, chenbro, and a few other brands of barebones that come with just PSU, chassis, and fans. Also remember that the AX750 PSU you have outlined cannot sustain EPSx2 and PCI-Ex2. The second 8pin EPS connector takes up the second PCI-E cable. You'll have to use a modular cable and then a 2x modular to 1x PCI-E adapter to make it work.

The coolers exist mainly for single proc systems. Dual processors changes the game. While it would probably work as long as you're using a chassis that allows you to get all the screws in, you're running a risk by being out of the weight bearing specification (those two coolers together would be almost 3lbs!).

My recommendation is the Supermicro Super Workstation lineup, specifically their AMD G34 lineup A+ models.

http://www.google.com/products/cata...a=X&ei=MMd8T5a5MoegtweytPH0DA&ved=0CCkQ8wIwAQ#

This is a 1200$ barebone, coming with dual socket mainboard, chassis, fans, and 920W 94 efficient power supply (servers are where high efficiency PSU's are really seen, it's not uncommon for 1U servers to have 98%+ efficient PSU's).

For about 1800$ you can get the quad socket model with 1400W gold power supplies.

http://www.google.com/products/cata...a=X&ei=5cd8T8rkF8GItwfv-KH8Bw&ved=0CGIQ8wIwAQ#

Of course that's over budget, but throwing it out there anyways.

 

[Dinkydau]

They may all be doing it wrong, but on overclock.net (a fairly big site), non-EEB cases are being suggested, including this haf932, and the coolers as well:
http://www.overclock.net/t/1212613/...t-to-finish-along-with-ppd-benchmarks-gallery

I wonder what kind of temperatures those people are getting, taking into consideration that they are using a lot of float for their folding calculations.

Also remember that the AX750 PSU you have outlined cannot sustain EPSx2 and PCI-Ex2. The second 8pin EPS connector takes up the second PCI-E cable. You'll have to use a modular cable and then a 2x modular to 1x PCI-E adapter to make it work. .

So as long as I don't use 2 GPUs there should be no problem? I don't intend to use multiple GPUs.

 

[mfenn]

Now that makes me wonder what those noctua coolers are actually supposed to do. If they're too heavy, why do they even exist?

They are bolt-through mount coolers, which support all of the weight on the PCB as opposed to the socket. That's fine for ATX boards which are designed for that mounting solution, not so fine for SSI EEB boards that aren't.

SSI EEB boards are designed to work with the chassis for cooling in space-constrained rackmount servers. The typically mount low-profile passive coolers and use the chassis fans to direct air over them.

TL;DR There are different design considerations for server boards than for desktop boards. What works for one doesn't work for the other and vis versa.

 

[heymrdj]

They may all be doing it wrong, but on overclock.net (a fairly big site), non-EEB cases are being suggested, including this haf932, and the coolers as well:
http://www.overclock.net/t/1212613/...t-to-finish-along-with-ppd-benchmarks-gallery

I wonder what kind of temperatures those people are getting, taking into consideration that they are using a lot of float for their folding calculations.


So as long as I don't use 2 GPUs there should be no problem? I don't intend to use multiple GPUs.

A gentleman in that thread says he can't run it in hot weather due to temps. Another runs it out of a window to get the ambient down.

As for some of their coolers they are mounting, yes they are definitely doing it cheap. Those coolers, as mfenn posts, hang off the PCB. If that case would ever take a shock, that board *will* snap in half, destroying whatever processor underneath of it at the same time most likely (crushing the IHS and the dies underneath it). You couldn't even transport the system upright, if it were carried it would have to be moved on its side. Even over time, heat fatigue on the mainboard is going to slowly weaken it to the point that one day the thing will snap.

Another thing that bothers me, the first post says he's using that second supermicro board. Now go to the page and notice it says AIR SHROUD REQUIRED. See in the pic how the VRM's are nude? What performance board does that these days. heatsinks don't fit in a 1u chassis without blocking the cpu coolers. Alot of times cpu heat spreaders in 1/2u chassis are made especially for the mainboard and cover the cpu's and their vrms. But overall the board design, again, is telling you it requires force air cooling.

You can rig this thing up cheap. That's feasible, and doable, those people in that thread prove that by speaking with their wallet instead of common sense. I don't do that. If I can't build it reliably I won't build it at all. The boards ect will not last with that kind of abuse (both weight and heat) and they'll likely take a processor or two out with them. Folding isn't free, it's a donation of someones money into a cause, that money being what it costs to build a high powered cadillac build.

You are more than free to build it how you need to afford it. Sadly I can't help with that because I just don't have any experience with jerry rigging this stuff together to make it work, I've always just bought parts that work together. Still, if you figure it out, make a build thread anyways. :biggrin:

This still has me wanting to make my own 4P/64c rig..

 

[heymrdj]

And to the graphics deal, yeah if you're running one card you'll have no issues. Still I like density. I would personally do G34 quad socket with 4 graphics cards. But no one makes a board because the extra two sockets eat up the room for the extra GPU's. They do make tri card dual G34/xeon setups though. http://www.supermicro.com/Aplus/system/Tower/4022/AS-4022G-6F.cfm

Only 1200$ http://www.wiredzone.com/Supermicro...ower-4U-Socket-G34-f--AMD-6100~10020534~0.htm

Wish it had a 1400W PSU though, 1000W seems to be low for three super graphics cards (Can you say 3 7990's outside of crossfire? My god the PPD..)

 

[Dinkydau]

Okay, I figured some things out.

First of all, those coolers, you're not hanging two of those off that main board. Do you understand how much weight that is? The boards are designed for small efficient low profile copper coolers (1u/2u) or larger aluminum light weight coolers that relay on air flow from from a nearby chassis fan.

The coolers exist mainly for single proc systems. Dual processors changes the game. While it would probably work as long as you're using a chassis that allows you to get all the screws in, you're running a risk by being out of the weight bearing specification (those two coolers together would be almost 3lbs!).

Yeah, that makes sense. I think I could lay the case down on it's right side. I've got enough room in my room to do that. I know I'm stubborn, but you're probably still saving my motherboard!

Bit-tech shows the case temps here: http://www.bit-tech.net/hardware/cases/2008/10/16/cooler-master-haf-932/4

That's just 1 130W processor. You're pushing double that. And double that temperature is no where near acceptable for 24/7 use.

Watch out! That's not correct physics. The temperature doesn't double, instead, the Δ temperature doubles. In this case (double meaning): 43-21=22°C. 2 CPUs would make it two times that, double the watt = double the energy = double the heat. With the ambient temperature of 21°C that is 22*2 + 21 = 65°C. That's an acceptable temperature. Considering my CPUs have less wattage, it will be less than that: (115/130)*22*2 + 21 = 60° C. There's also the effect that the bigger the difference in temperature inside the case and outside, the more efficiently the air cools, although I don't know how big of a difference that makes, probably not thAt much.

I've read reviews on newegg about practical situations with similar opteron CPUs. I assume they aren't using those SSI EEB cases, otherwise they wouldn't have these coolers. Temperatures are looking good there, although I don't trust those reviews too much without more information about those people's systems.

All the EEB-cases on newegg have reviews stating that the cooling is loud. I also doubt it's actually better than all those coolers in the HAF932. I don't mind if the case doesn't officially support the form factor. As long as it works it's good, so what I'm worried about at the moment is the screws. How many of the screws can be used, and which?

According to cooler master, the kgpe-d16 motherboard can be installed in the haf932. The man at the company I'm buying most of the parts from contacted cooler master about this, and that's what they told him.

So, yeah, I'm still going for the haf932 I think.

 

[heymrdj]

Okay, I figured some things out.


Yeah, that makes sense. I think I could lay the case down on it's right side. I've got enough room in my room to do that. I know I'm stubborn, but you're probably still saving my motherboard!


Watch out! That's not correct physics. The temperature doesn't double, instead, the Δ temperature doubles. In this case (double meaning): 43-21=22°C. 2 CPUs would make it two times that, double the watt = double the energy = double the heat. With the ambient temperature of 21°C that is 22*2 + 21 = 65°C. That's an acceptable temperature. Considering my CPUs have less wattage, it will be less than that: (115/130)*22*2 + 21 = 60° C. There's also the effect that the bigger the difference in temperature inside the case and outside, the more efficiently the air cools, although I don't know how big of a difference that makes, probably not thAt much.

I've read reviews on newegg about practical situations with similar opteron CPUs. I assume they aren't using those SSI EEB cases, otherwise they wouldn't have these coolers. Temperatures are looking good there, although I don't trust those reviews too much without more information about those people's systems.


According to cooler master, the kgpe-d16 motherboard can be installed in the haf932. The man at the company I'm buying most of the parts from contacted cooler master about this, and that's what they told him.

So, yeah, I'm still going for the haf932 I think.

I wasn't getting into the physics of it, just going rough estimate because I thought you were going with crunching cards as well, but then I see you're not so that does make a big difference on chassis heat.

Like I said it's your money you can certainly do with it as you please. I would make sure, however, that you take a cheap autozone infrared thermometer and point it at the VRMS after they have been full load for awhile. Since they are not getting direct flow you might need to buy some VRM sinks and apply them with a drop of thermal paste to ensure they stay cool enough. Then again, I recommend these anyway for number crunching rigs.

http://www.allelectronics.com/make-a-store/item/HS-154/31-X-16-X-11MM-HEATSINK/1.html

Has thermal compound on it already. Cheap insurance

 

[Dinkydau]

That's something I had never heard of before. Can you point me where that module is on the motherboard? I looked carefully at this high detailed image and found nothing that I think could be it given the size of that heat sink:
http://static.pcinpact.com/images/bd/news/82151-asus-g34-kgpe-d16.jpg

 

[heymrdj]

That's something I had never heard of before. Can you point me where that module is on the motherboard? I looked carefully at this high detailed image and found nothing that I think could be it given the size of that heat sink:
http://static.pcinpact.com/images/bd/news/82151-asus-g34-kgpe-d16.jpg

You see those 6 little grey blocks that make a small L shape around each socket (one on the short end of the socket, then 5 down the back of the socket). 12 total. Those there provide the voltage for the processors, and depending on who makes them, ambients, ect can get quite hot with flagship wattage processors. I had to provide extra cooling to my MSI 890GXM-G65 mainboard because the VRMs would get hot and make my overclock unstable. My 1090T is a 125W processor.

 

[Mark R]

You see those 6 little grey blocks that make a small L shape around each socket (one on the short end of the socket, then 5 down the back of the socket). 12 total. Those there provide the voltage for the processors, and depending on who makes them, ambients, ect can get quite hot with flagship wattage processors. I had to provide extra cooling to my MSI 890GXM-G65 mainboard because the VRMs would get hot and make my overclock unstable. My 1090T is a 125W processor.

Just to clarify, the grey blocks are notthe key components of the voltage regulators. The key components are MOSFETs and diodes. Look carefully between the grey blocks, and you will see black chips, with 4 pins on one side, and a flat metal pad on the other. Those are the MOSFETs. Immediately next to the MOSFETs, you'll see black plastic bricks, with a grey strip on one end - those are the diodes. It is those components that need very careful cooling. As overheating will cause performance to degrade.

The grey blocks are just coils of wire. They can get hot, and that doesn't matter (as long as you keep the temperature below about 120 C). They're not electronic, they're just wire looped around a ceramic core. Temperature has a minimal effect on their operation.

 

[heymrdj]

Just to clarify, the grey blocks are notthe key components of the voltage regulators. The key components are MOSFETs and diodes. Look carefully between the grey blocks, and you will see black chips, with 4 pins on one side, and a flat metal pad on the other. Those are the MOSFETs. Immediately next to the MOSFETs, you'll see black plastic bricks, with a grey strip on one end - those are the diodes. It is those components that need very careful cooling. As overheating will cause performance to degrade.

The grey blocks are just coils of wire. They can get hot, and that doesn't matter (as long as you keep the temperature below about 120 C). They're not electronic, they're just wire looped around a ceramic core. Temperature has a minimal effect on their operation.

Yes you're right. I see that in tapatalking it cut off my second paragraph. I was going on to say that i used sinks that bridged between the black squares and the grey squares. Thanks for clarifying that for us!

 

[Dinkydau]

Thank you mfenn, heymrdj and Mark R very much for the information, and as well thanks to everyone else who couldn't help but still viewed the thread. I think I don't need further assistance for now. I'll post the results when the computer is finished. For your info, I won't be building it myself to avoid taking the risk of doing something wrong and ruining expensive hardware. I'll only do a few small things, so I can't really take pictures of the progress.