Quiet Cluster Cooling

[Minotaar]

Im a grad student preparing plans for my prof for a new 50-100 node duallie cluster in our lab. Currently, we've got a 16 node 1900 MP cluster, each in a mini chassis. Its LOUD. CRAZY loud.

Yes, there is a possibility that we can put the new cluster in another building, but that sets up administration and accessibility issues. For the sake of this discussion, lets assume this cluster isnt going to be in a machine room. But we would ALSO like to keep the labspace holding the cluster as useable as possible: Unfortunately, with 200 cpu fans and 400 case fans whirlling, we'll all be deaf as Kiss. So heres my problem:

I would like to get your input on ways to QUIETLY cool 100 rackmounted duallie boxes. Money is not really an issue; Im just askin for ideas.

In particular, I've been trying to see if it is possible to centrally watercool the cluster, by running all the waterstreams through one big-a$$ exchanger. Has anyone got any experience with this? One of the problems would be the pump: 100 half inch tubes moving water off CPUs is a lot of water per second. It would have to be a hardcore pump.

There are also failure issues: if the pump dies, we dont want to lose 100 machines. Is there a way to get a pump to start up if water pressure drops? Are there waterblocks that can dissipate enough heat with no water that the CPU doesnt die before auto-shutoff?

A) how would you watercool rackmounted 200 opterons or 200 MPs? And would it even be that quiet? I think it could be made quiet with central cooling (i.e. giant exchanger, 1 giant pump)

or

B) how would you cool it otherwise?

Constraint: No downclocking/volting

 

[MDE]

Air condition the room to about ~60 degrees (or whatever is the lowest allowed by your local building code) and run the biggest fans possible in the cases and on the heatsinks.

 

[Minotaar]

we're going to be using 1U cases, for space reasons. The room has AC, but no cieling

Wish i could do that.

The other thing is that the retard staff occasionally turns off the AC so we cant be dependant on that anyway.

If we were using ATX chassis, yeah I would go for that. But we're talking a 16x16 room, 100 machines. need rackmounts.

 

[MDE]

Watercooling sounds way too expensive and too complex for 100 systems, let alone 100 dual CPU systems.

 

[Minotaar]

Its a Challenge!

 

[sparkyclarky]

Watercooling would not work. The chance for failure over that many rigs would be very high. Maintenance would be a PITA for that many rigs too. I don't know if it's possible, but you might want to try to rig up the largest fans possible in the rackmounts, even if it requires some cutting.

 

[myocardia]

The only way I can think of to cool 200 Opterons in a 16'x16' room quietly is with liquid nitrogen.

edit: Besides, 100 computers, even if they're in 1U cases, is going to consume an entire 16x16 room anyway. There isn't going to be any room left for conversation!

 

[Minotaar]

You can put them in 2 48U racks.

we got 16 antec small ATX boxes on a metro-rack thing in there, and it takes up about a 8x3 foot area of space. (three levels)
Two racks would be about the same, I think.

I might be a little wrong about the dimensions; it might be closer to 20x20, but even so, its s small square room.

I totally understand what the guy is saying about the failure rate if we go with water cooling. But how can we do air cooling
quietly?
I was thinking of getting a "sheet" of 120mm fans and throwing them accross the front and back, but Im not sure if that would be enough.

The real concern for me is that each mobo is the same, and thus the CPUs will be in the same position vertically, making one giant hot spot.

 

[Cerb]

Don't worry about the hotspot as they will be. That's why there's tons of fans in those little guys!

However...I don't think you can cool anything better than a socket PIII quietly in a 1U case.
Athlon MPs, Opterons, whatever, will use around 20-70w. It's got to be pulled by the HS, and pushed somewhere. Period. Add to that that AMD specs 89w for them, so the fans used will be pretty beefy.

I seriously don't think there's a way to do it without custom building multi-node cases from the 1U cases (say, four of them,most of the front and back removed, 120mm fans in the front pushing back, duct through the CPU HSs, having the only major vent be the exit hole with the CPU HSs). But then I couldn't be sure that it would push enough air to do the job right.

Now, if you could decide on hardware (Opterons will be the most power-friendly, though MPs will use a bit less going full-bore), and buy ten or so, you could experiment.

 

[zodder]

Ummm, I know this is OT, but have you considered stopping into the Distributed Computing forum and joining one of our great TeAms? It makes me sad to see so much horse power going untapped.

 

[Minotaar]

How about this:

you help me figure out what I need to do to quietly cool this cluster, and if the grant for the cluster is approved, come over here and help me set it up, we'll put a cron process up that runs SETI@home or your favorite number cruncher in the background. That'll benchmark the box pretty well, and put some big iron on your team

 

[Rastus]

1U cases are so thin, the fans are small and high speed, and the air is being forced through such a small space that the noise is going to be a problem. You might have to get thicker cases for your farm if you want less noise.