Our immersion cooled extreme gaming rig experiment

pm

Elite Member Mobile Devices
Super Moderator
Jan 25, 2000
7,420
0
81
#1
My co-workers and I do a demonstration overclocking rig every year for Intel Colorado LANFest which is a charity event which benefits the Larimer County United Way here in northern Colorado. Each year we scrounge up parts from Intel, vendors like Asus, Mushkin and EVGA, and buying some stuff ourselves and put together a demo rig for the event and try to overclock it to beat the world records on HWBot.com for various. The demo rig is actually part of a contest – we challenge anyone to beat our system at 3DMark and Linpack (need to win both) using a closed-loop cooling system (no LN, no dry ice) and the prize is usually a top-of-the-line Extreme Edition CPU (value ~$1000). In seven years of doing it, we haven’t ever lost – although, with one notable exception, no one has ever really tried that hard. The whole point is for a bunch of engineers to geek out playing with a lot of crazy cooling equipment and the latest-and-greatest hardware and see if we can beat some world-class benchmarks in the progress - and it's all for charity.

Last year, 2014, we decided to go one step further than liquid cooling using a refrigeration system. We had always been impacted by frost and condensation build-up on the liquid cooling lines and we wanted to try and go colder than our glycol-based refrigeration system was capable of driving. So we decided to build an immersion cooled machine and set a goal of hitting -100C with it. The fluid we chose was 3M Novec fluorinated liquid (http://multimedia.3m.com/mws/media/...ectronic-liquid-fc-770-product-info-sheet.pdf ) and we came up with a three stage cooling system to drive the temperature down. Funny fact, Novec is in the same material family as the liquid that they showed in the movie “The Abyss” and theoretically if you oxygenated it, you could breathe it in. But we didn’t try this ourselves.

The system that we used for this was an Intel 5960X Extreme Edition, an Intel 1TB PCIe SSD, two Radeon 295x2 video cards and two Xeon Phi cards. The two Xeon Phi cards were fun to play with and gave us a jaw-droppingly high Linpack score. On the refrigeration side, we used a ~$20k industrial refrigeration system running glycol to take the 3M Novec fluorinated liquid down to about 7C (close to the recommended low-end limit for the system and coolant) and then used a two-stage Peltier system that had ~1000W per head (2kW total) to drop it down to about -80C.

I uploaded our photos and documented everything on Imgur:
http://imgur.com/a/dlevz

While the system was capable of running down to -80C by itself, when we dropped in the two 295x2s and the two server CPU’s and powered all that up we had a hard time holding it around -55C. If we let the system sit idling, we could get close to -60C, but once we powered it up, it shot up to over -40C under Linpack and 3DMark. The system didn’t have the thermal capacity to hold it much below 0C when it was fully powered up and ran continuously under load, but by bursting benchmarks and then letting it cool, we could run it briefly at -50C to -60C. But you have to pick an overclock point that is at the warmest the system would get, so we could overclock to a much higher frequency when the system was idling (or running super short benchmarks like SuperPI) but under a more sustained load the temperature would rapidly rise.. We overclocked the 5960X to 5.2GHz on all cores and were able to run Linpack and 3DMark06 stable like this. And we pushed the video cards very high – particularly in the memory speed. We could boot as high as 5.6GHz but it wasn’t stable in the physics test. This got us the #7 spot in the world on hwbot.com. On Linpack, we hit just over 4.9Tflops sustained single-precision Linpack – which actually put us on par with the fastest supercomputer in the World in 2000 (Lawrence Livermore Lab’s ASCI White - http://www.top500.org/featured/systems/asci-white-lawrence-livermore-national-laboratory/ )

There were a lot of learnings building this system – we learned that Novec evaporates pretty quickly and –rather terrifyingly - smells good. We learned that it thermally contracts so that you had to add more liquid to the system. And we learned that our Peltier stage was sized too small and we need to at least double or quadruple it next time. This year, 2015, LANFest was pulled in by two months (from October to August) and we didn’t have enough time to get the components needed so we took a year off. But next year we will be back with another round, ideally with the latest Xeon Phi cards, and a pair of whatever is the latest and greatest CPU at the time – and more Peltier TEC modules.

If you happen to be near Colorado and want to join us for LANFest 2016, we doubled the size of the event. I don't know that we have the date exactly nailed down yet but it will be sometime around Sept. 2016.
 
Last edited:
Mar 13, 2006
10,089
48
126
#2
Neat Patrick, thanks for sharing.

Is that you with the ponytail, or are you in the Ralph Lauren shirt?
 
Last edited:

pm

Elite Member Mobile Devices
Super Moderator
Jan 25, 2000
7,420
0
81
#4
Neat Patrick, thanks for sharing.

Is that you with the ponytail, or are you in the Ralph Lauren shirt?
I'm the photographer. I'm not in any of the photos. It's a bit like our family photos... if you looked at them, you might think my children don't have a father. I can stick one in of me if it helps for visualization. :)

The ponytail guy is Warren Parks - our power delivery and liquid cooling guy, and the best guru at soldering things that I have ever seen. The guy with the Ralph Lauren shirt is Eric Fetzer - he's our organizer and the guy who comes up with the ideas and the funding and he's the main overclocker. And the the gentleman in the blue shirt near the end is Steve Poehlman - he is our Linpack and Xeon Phi guru. I'm not sure what purpose I serve exactly except to take photos. The guy in blue in the top photos is Tom Bozic who is an kickass analog design engineer and my lunchtime running partner. Tom's in the photos because he heard we were submerging a computer in the lab and said "oh I have got to see this"
 
Last edited:

pm

Elite Member Mobile Devices
Super Moderator
Jan 25, 2000
7,420
0
81
#6
Ba dum, tsh.

I remember that stuff from The Abyss. Apparently it's called Liquivent. It seems to have some of the same properties as Novec.
I got my info from this wikipedia page - https://en.wikipedia.org/wiki/Fluorinert - and Novec is the successor to Flourinert. But... wikipedia. Either way, it makes for a neat story to tell whenever anyone looks at the stuff... whether or not it's true. I'm Irish and why let a few facts get in the way of a good story.
 
Apr 27, 2000
10,521
373
126
#7
It is a good story. With some more time and money, somebody could probably whip up a similar solution less prone to losing fluid to evaporation and less prone to icing . . . but it would probably look a lot less cool (and/or garish). One-off cooling solutions for events don't need that kind of treatment anyway.

However, if you could dunk a rat in there and keep it alive for awhile, that would be a nice trick (until PETA found out). The rat from The Abyss really did breathe that Liquivent stuff, though they had to cut a lot of footage from the scene since the rat kept freaking out and defecating in the fluid (lulz). It also became terribly prone to infection afterwards, so it needed antibiotics.

What kind of TECs were you using last year? Is everyone still stuck on bismuth telluride? I've heard rumblings of better materials, but I don't know if anything has made it into available products yet.
 

Headfoot

Diamond Member
Feb 28, 2008
4,380
27
126
#8
Totally awesome. Let the engineers run loose with a decent budget and you can get some insanely cool stuff
 

DigDog

Diamond Member
Jun 3, 2011
9,980
102
126
#9

as far as i can tell, it's a PCIE card with 61 cpu cores.
 

pm

Elite Member Mobile Devices
Super Moderator
Jan 25, 2000
7,420
0
81
#13
You might want to correct that ;)
Yeah, you are right. My bad. The 2013 system was dual-CPU. The 2014 system wasn't.

I'll fix it.
 


ASK THE COMMUNITY

TRENDING THREADS