Win a 48-core Magny Cours system

[aigomorla]

Water cooling could be huge for data centers, but it needs to be standardized. Until that happens people are not going to jump in. When the back of a server has a water input and output that are standardized like a USB or ethernet port, it will have a chance.

JF if you ask... they will make it:

Sorry to derail this but you asked:

U4 server rack H2o System:

U2 Watercooling Case:

PCI Passthough:

Basically its all using QuickDisconnects so you would connect it like how you would lan cables. :sneaky:


JF if you guys are ever bored and have extra parts u should seriously give me an opportunity to build something for you guys.

I even custom cool my boards like this:

I would probably redefine the definition of enterprise watercooling, if i had a sample to monkey with.

 

[JFAMD]

OK, it helps if I read the rules. Contest open to US and Canada (excluding Quebec), so if any Canadians want to try, I apologize for the mistake.

 

[Rubycon]

I visited a data center that had a room of water cooled servers. It was in a testing stage. There were 21,000 servers in a room they said if they were air cooled the room would catch on fire!

I did not see the specifics other than they were definitely quick connected with parker type fittings. The water supply and return was glycol based and there was a 700 ton chiller in the basement with a cooling tower on the roof.

On our ship we have a supply of chilled water that's maintained around 8 degrees centigrade. I'd LOVE to be able to tap this and water cool my computers but am met with great resistance every time I ask. I still do though. It's good to shake the can every now and then. It keeps the good things from settling to the bottom. :biggrin:

 

[SunnyD]

You know, I keep trying to come up with novel ideas to enter with, but I just can't come up with anything realistic.

 

[cbn]

I visited a data center that had a room of water cooled servers. It was in a testing stage. There were 21,000 servers in a room they said if they were air cooled the room would catch on fire!

I did not see the specifics other than they were definitely quick connected with parker type fittings. The water supply and return was glycol based and there was a 700 ton chiller in the basement with a cooling tower on the roof.

On our ship we have a supply of chilled water that's maintained around 8 degrees centigrade. I'd LOVE to be able to tap this and water cool my computers but am met with great resistance every time I ask. I still do though. It's good to shake the can every now and then. It keeps the good things from settling to the bottom. :biggrin:

Do you know what kind of refrigerant they used in the basement chiller?

Looking under Wikipedia I am seeing Ammonia, sulfur dioxide, Co2 and methane listed as possible refrigerants. I wonder if there is potential in making any of these as wastes, but then using them in a chiller? (A self contained loop than makes sense without needing Green laws to achieve viablitiy).

 

[Rubycon]

The chiller was based on the absorption principle with the refrigerant R717 or ammonia.

The plant is steam powered from utility steam off the street. The chiller has heat exchangers that circulate the chilled water from the server room. The condenser circulates water that goes to the roof to a forced draft cooling tower. That's what I remember about the system.

 

[alyarb]

it sucks (or for that matter does it blow?) that you can't use CPU heat to push steam through a turbine.

 

[Rubycon]

Well you can if you use fluids with appropriate boiling points. This is exactly what happens with heat pipes. Ten cpus easily put out over 1 horsepower worth of heat.

 

[nyker96]

As to what I'd do if I won this montster, probably use it to cook my morning breakfast while encoding all my hi def home video into x264, I mean all of them at the same time!!

 

[cbn]

Well you can if you use fluids with appropriate boiling points. This is exactly what happens with heat pipes. Ten cpus easily put out over 1 horsepower worth of heat.

When I looked up cooling towers I noticed there were two different kinds.

One uses an evaporative principle and the other apparently just uses air to cool the water to dry bulb temperatures.

My guess is that the amount of engineering/cost required to recapture some of the evaporated water back as electricity could outweigh any benefits. For example, this forced draft cooler you mentioned looks quite large in pictures. I can only imagine the engineering/construction to harness steam from such an enormous tower would be astronomical.

However if another liquid with a lower boiling point (like you said) could be used then maybe the size of these structures could be scaled back to the point where it was do-able (from an efficiency standpoint). But then how to replenish the liquid being evaporated? Could this liquid be produced from another process? In contrast, water is so cheap and available.

 

[Rubycon]

Well do remember the laws of conservation always apply. At least in the 21st century they do.

These systems were designed for uptime, and processing density. There's other ways to save money too. One installation does that by making ice and running the compressor at night when electricity is cheaper. The ice keeps up with the heat load until about 1800 the following evening when the compressors engage repeating the process.

There was a northbridge cooler made that actually used a tiny stirling engine to turn a fan to cool itself! Does anyone remember that?

 

[JFAMD]

http://www.youtube.com/watch?v=tNrBavjByfw

A look inside the system

 

[cbn]

There's other ways to save money too. One installation does that by making ice and running the compressor at night when electricity is cheaper. The ice keeps up with the heat load until about 1800 the following evening when the compressors engage repeating the process.

What you are mentioning here sounds really interesting. Not only is that compressor using electricity when it is cheapest, but it is also making heat at night.

Maybe it is the heat produced at night that leads to even greater gains in cost reduction? Is there another process that is even more profitable than simply heating the building when all the employees are at home?

Obviously I don't know the answers to these questions. However, I am sure someone out there has ideas and expertise to implement a strategy that increases competitiveness.

 

[Rubycon]

What you are mentioning here sounds really interesting. Not only is that compressor using electricity when it is cheapest, but it is also making heat at night.

Maybe it is the heat produced at night that leads to even greater gains in cost reduction? Is there another process that is even more profitable than simply heating the building when all the employees are at home?

Obviously I don't know the answers to these questions. However, I am sure someone out there has ideas and expertise to implement a strategy that increases competitiveness.

I don't recall the specifics of that system but it's possible that heat of compression could be used for other things besides being dumped into the cooling towers on the roof.

 

[cbn]

http://www.youtube.com/watch?v=tNrBavjByfw

A look inside the system

Thanks for the video JFAMD.

P.S. How big is the Magny Cours die? I would assume Global Foundries is at the point where making bigger chips on 45nm is much cheaper than it used to be.

 

[alyarb]

magny cours is two istanbul dies in MCM (346 sq mm times 2).

 

[cbn]

magny cours is two istanbul dies in MCM (346 sq mm times 2).

So we are talking what is essentially two Opteron 2400 dies linked together but each is using only 57.5 watts each?

But doesn't each Istanbul die have enough L1/L2/L3 cache to efficiently support clock speeds higher than 2.1-2.2 Ghz? What about the increased number of memory channels on Opteron 6100 as well as the increased bandwidth from DDR3?

It would seem to me Magny Cours is capable of much higher performance, but the core clocks are holding it back.

I think linking two smaller mainstream chips together to make a single high end product is a tremendous cost saving idea. Is there potential to take this modular concept one step further?

Could water cooling preserve the capability to run higher clock speeds on these dual die chips while maintaining the necessary undervolting to achieve the target TDP?

P.S. Forgive me if I am using any of these terms wrong. Like I have said before I am not an IT or hardware guy.

 

[cbn]

As to overclocking, that is not supported on servers. Overclocking is a consumer activity and enterprise customers do not see the value in it. Performance is much lower down the list behind stability and power consumption. Overclocking will increase something of lower value and negatively impact two things of higher value.

I'm sure that Tyan Mainboard doesn't support undervolting either does it?

Maybe undervolting a Phenom II x6 under H20 would be more telling of any power savings possibilities?

 

[IntelUser2000]

*snip*

It would seem to me Magny Cours is capable of much higher performance, but the core clocks are holding it back.

*snip*

The power usage figures suggest otherwise, but power is the sole reason these massive chips run at low clock speeds.

 

[cbn]

Here we are seeing water conduits built directly into the PCB

How well could something like this work with modular chips?

Watercooling (with channels going in between the separate dies) for the high core count SKUs vs. air cooling for the cheaper lower core count mainstream offerings?

The power usage figures suggest otherwise, but power is the sole reason these massive chips run at low clock speeds.

57.5 watts per die is not a lot of power. How sensitive is the efficiency of these chips to heat? (I have read that as heat increases so does xtor leakage)

 

[JFAMD]

magny cours is two istanbul dies in MCM (346 sq mm times 2).

Magny Cours is two Lisbon dies, not two Istanbul dies. There are feature differences to those.

 

[JFAMD]

The power usage figures suggest otherwise, but power is the sole reason these massive chips run at low clock speeds.

You'll see power usage for these 12 core processors being roughly equivalent to our 6-core Istanbul today.

Istabul is generally the same power consumption as Nehalem today.

Westmere fits into the same platform as Nehalem.

My guess is that intel is going to lean into the thermals for westmere to get as much performance as possible, so while it is 32nm, it will still be in the same TDPs.

So, if you use the rough mathematical formula, you get:

Istanbul = nehalem = westmere = magny cours

Today 2P systems are roughly equivalent:

Opteron 2435 (258 @ 100%):
http://www.spec.org/power_ssj2008/results/res2009q3/power_ssj2008-20090602-00168.html

Xeon X5570 (245 @ 100%): http://www.spec.org/power_ssj2008/results/res2009q2/power_ssj2008-20090407-00143.html

So if you assume that we both end up in the same power/thermal range, then you end up with the 6 cores of Westmere vs. the 12 cores of Magny Cours.

This of course assumes that Intel pushes the limits to get the highest clock speeds and does not drive to lower clocks and lower power consumption. History has proven that they take this path 100% of the time.

The biggest piece of misinformation that they are spreading is that magny cours will be hot. The reality is that it will generally be the same power/thermals as Istanbul (and actually lower at idle.)

When you combine that power profile with the fact that it will be up to 2X the performance of Istanbul, you start to get a pretty good view of where performance per watt lands.

Low clock speeds? Let's just say that at clock to clock comparisons, MC is a lot faster than Istanbul. And our 8-core and 12-core products will have higher clocks than beckton.

 

[JFAMD]

Here we are seeing water conduits built directly into the PCB

How well could something like this work with modular chips?

Watercooling (with channels going in between the separate dies) for the high core count SKUs vs. air cooling for the cheaper lower core count mainstream offerings?




57.5 watts per die is not a lot of power. How sensitive is the efficiency of these chips to heat? (I have read that as heat increases so does xtor leakage)

Well, don't forget that there wll also be low power 12 core processors as well

 

[dalauder]

Yes, in my price range GPU is more cost effective.

Did you get in on those Galaxy GTS 250 deals? With some legwork, they can be found for 21 bucks? ($119 - $80MIR -15%Bing CB ~ $21)

As to uses for the 48 core rig, I know my brother could use it for number crunching complex weather simulations from home instead of the lab. Environmental science research needs all the super computers it can get. Right now, weather simulations often analyze the 20 miles thick of atmosphere like it's 2 dimensional across the earth. So if you could get 20 of these rigs, you might be able to process some limited 3D simulations in a couple weeks.

 

[taltamir]

What would you do with 48 cores

I would sell it.
honestly, I don't think my house's wiring can provide enough power to run it.