My co-workers and I do a demonstration overclocking rig every year for Intel Colorado LANFest which is a charity event which benefits the Larimer County United Way here in northern Colorado. Each year we scrounge up parts from Intel, vendors like Asus, Mushkin and EVGA, and buying some stuff ourselves and put together a demo rig for the event and try to overclock it to beat the world records on HWBot.com for various. The demo rig is actually part of a contest – we challenge anyone to beat our system at 3DMark and Linpack (need to win both) using a closed-loop cooling system (no LN, no dry ice) and the prize is usually a top-of-the-line Extreme Edition CPU (value ~$1000). In seven years of doing it, we haven’t ever lost – although, with one notable exception, no one has ever really tried that hard. The whole point is for a bunch of engineers to geek out playing with a lot of crazy cooling equipment and the latest-and-greatest hardware and see if we can beat some world-class benchmarks in the progress - and it's all for charity.
Last year, 2014, we decided to go one step further than liquid cooling using a refrigeration system. We had always been impacted by frost and condensation build-up on the liquid cooling lines and we wanted to try and go colder than our glycol-based refrigeration system was capable of driving. So we decided to build an immersion cooled machine and set a goal of hitting -100C with it. The fluid we chose was 3M Novec fluorinated liquid (http://multimedia.3m.com/mws/media/...ectronic-liquid-fc-770-product-info-sheet.pdf ) and we came up with a three stage cooling system to drive the temperature down. Funny fact, Novec is in the same material family as the liquid that they showed in the movie “The Abyss” and theoretically if you oxygenated it, you could breathe it in. But we didn’t try this ourselves.
The system that we used for this was an Intel 5960X Extreme Edition, an Intel 1TB PCIe SSD, two Radeon 295x2 video cards and two Xeon Phi cards. The two Xeon Phi cards were fun to play with and gave us a jaw-droppingly high Linpack score. On the refrigeration side, we used a ~$20k industrial refrigeration system running glycol to take the 3M Novec fluorinated liquid down to about 7C (close to the recommended low-end limit for the system and coolant) and then used a two-stage Peltier system that had ~1000W per head (2kW total) to drop it down to about -80C.
I uploaded our photos and documented everything on Imgur:
http://imgur.com/a/dlevz
While the system was capable of running down to -80C by itself, when we dropped in the two 295x2s and the two server CPU’s and powered all that up we had a hard time holding it around -55C. If we let the system sit idling, we could get close to -60C, but once we powered it up, it shot up to over -40C under Linpack and 3DMark. The system didn’t have the thermal capacity to hold it much below 0C when it was fully powered up and ran continuously under load, but by bursting benchmarks and then letting it cool, we could run it briefly at -50C to -60C. But you have to pick an overclock point that is at the warmest the system would get, so we could overclock to a much higher frequency when the system was idling (or running super short benchmarks like SuperPI) but under a more sustained load the temperature would rapidly rise.. We overclocked the 5960X to 5.2GHz on all cores and were able to run Linpack and 3DMark06 stable like this. And we pushed the video cards very high – particularly in the memory speed. We could boot as high as 5.6GHz but it wasn’t stable in the physics test. This got us the #7 spot in the world on hwbot.com. On Linpack, we hit just over 4.9Tflops sustained single-precision Linpack – which actually put us on par with the fastest supercomputer in the World in 2000 (Lawrence Livermore Lab’s ASCI White - http://www.top500.org/featured/systems/asci-white-lawrence-livermore-national-laboratory/ )
There were a lot of learnings building this system – we learned that Novec evaporates pretty quickly and –rather terrifyingly - smells good. We learned that it thermally contracts so that you had to add more liquid to the system. And we learned that our Peltier stage was sized too small and we need to at least double or quadruple it next time. This year, 2015, LANFest was pulled in by two months (from October to August) and we didn’t have enough time to get the components needed so we took a year off. But next year we will be back with another round, ideally with the latest Xeon Phi cards, and a pair of whatever is the latest and greatest CPU at the time – and more Peltier TEC modules.
If you happen to be near Colorado and want to join us for LANFest 2016, we doubled the size of the event. I don't know that we have the date exactly nailed down yet but it will be sometime around Sept. 2016.
Last year, 2014, we decided to go one step further than liquid cooling using a refrigeration system. We had always been impacted by frost and condensation build-up on the liquid cooling lines and we wanted to try and go colder than our glycol-based refrigeration system was capable of driving. So we decided to build an immersion cooled machine and set a goal of hitting -100C with it. The fluid we chose was 3M Novec fluorinated liquid (http://multimedia.3m.com/mws/media/...ectronic-liquid-fc-770-product-info-sheet.pdf ) and we came up with a three stage cooling system to drive the temperature down. Funny fact, Novec is in the same material family as the liquid that they showed in the movie “The Abyss” and theoretically if you oxygenated it, you could breathe it in. But we didn’t try this ourselves.
The system that we used for this was an Intel 5960X Extreme Edition, an Intel 1TB PCIe SSD, two Radeon 295x2 video cards and two Xeon Phi cards. The two Xeon Phi cards were fun to play with and gave us a jaw-droppingly high Linpack score. On the refrigeration side, we used a ~$20k industrial refrigeration system running glycol to take the 3M Novec fluorinated liquid down to about 7C (close to the recommended low-end limit for the system and coolant) and then used a two-stage Peltier system that had ~1000W per head (2kW total) to drop it down to about -80C.
I uploaded our photos and documented everything on Imgur:
http://imgur.com/a/dlevz
While the system was capable of running down to -80C by itself, when we dropped in the two 295x2s and the two server CPU’s and powered all that up we had a hard time holding it around -55C. If we let the system sit idling, we could get close to -60C, but once we powered it up, it shot up to over -40C under Linpack and 3DMark. The system didn’t have the thermal capacity to hold it much below 0C when it was fully powered up and ran continuously under load, but by bursting benchmarks and then letting it cool, we could run it briefly at -50C to -60C. But you have to pick an overclock point that is at the warmest the system would get, so we could overclock to a much higher frequency when the system was idling (or running super short benchmarks like SuperPI) but under a more sustained load the temperature would rapidly rise.. We overclocked the 5960X to 5.2GHz on all cores and were able to run Linpack and 3DMark06 stable like this. And we pushed the video cards very high – particularly in the memory speed. We could boot as high as 5.6GHz but it wasn’t stable in the physics test. This got us the #7 spot in the world on hwbot.com. On Linpack, we hit just over 4.9Tflops sustained single-precision Linpack – which actually put us on par with the fastest supercomputer in the World in 2000 (Lawrence Livermore Lab’s ASCI White - http://www.top500.org/featured/systems/asci-white-lawrence-livermore-national-laboratory/ )
There were a lot of learnings building this system – we learned that Novec evaporates pretty quickly and –rather terrifyingly - smells good. We learned that it thermally contracts so that you had to add more liquid to the system. And we learned that our Peltier stage was sized too small and we need to at least double or quadruple it next time. This year, 2015, LANFest was pulled in by two months (from October to August) and we didn’t have enough time to get the components needed so we took a year off. But next year we will be back with another round, ideally with the latest Xeon Phi cards, and a pair of whatever is the latest and greatest CPU at the time – and more Peltier TEC modules.
If you happen to be near Colorado and want to join us for LANFest 2016, we doubled the size of the event. I don't know that we have the date exactly nailed down yet but it will be sometime around Sept. 2016.
Last edited:
Facebook
Twitter