Lifetime of a supercomputer

[Phynaz]

Roadrunner is getting turned off - four years after it was the fastest computer in the world it has fallen to 22nd place.

http://arstechnica.com/information-...r-from-09-is-now-obsolete-will-be-dismantled/

 

[lakedude]

It sucks too much juice!

I guess that answers the question about absolute performance vs performance per watt...

 

[grimpr]

Seems they are moving their massive parallel workloads and algorithms to GPUs & Intels Phi.

 

[Maximilian]

Does it gets scrapped now or re purposed in smaller bits or something?

 

[BallaTheFeared]

Jaguar hit 1.76 petaflops to take the title, and it still exists as part of an even newer cluster called Titan. Titan took the top spot in the November 2012 supercomputers list with a speed of 17.6 petaflops.

What a difference the decimal placement makes!

 

[Idontcare]

It sucks too much juice!

I guess that answers the question about absolute performance vs performance per watt...

Yep

While the computer is still one of the 22 fastest in the world, it isn't energy-efficient enough to make the power bill worth it.

For example, in the November 2012 ratings Roadrunner required 2,345 kilowatts to hit 1.042 petaflops and a world ranking of #22. The supercomputer at #21 required only 1,177 kilowatts, and #23 (clocked at 1.035 petaflops) required just 493 kilowatts.

Quite a range in performance/watt.

 

[fixbsod]

Interesting to see perf/w being just as important in supercomputers vs tablets/phones.

 

[pm]

I took a tour of the NCAR "Yellowstone" supercomputer near Laramie, Wyoming, a month or two ago. It's currently #13(http://www.top500.org/system/177827 and http://en.wikipedia.org/wiki/Yellowstone_(supercomputer) ) and It was amazing to me to see how much infrastructure it takes to run a supercomputer. While they clearly built for expansion, even so, it was really surprising to me to see the size of the power delivery, air delivery, filteration, water delivery and cooling infracture in comparison with the comparatively small size of the computer room with the supercomputer in it.

So given the level of infrastructure for power delivery and then cooling, it makes some sense why they are so obsessed with perf/watt.

Look at the size of the cars in the parking lot versus the building... and the only use for the whole building is the supercomputer (ignoring some conference rooms and a interactive exhibit open to the public).

http://maps.google.com/maps?q=ncar+...=ncar&hnear=Laramie,+Albany,+Wyoming&t=h&z=17

 

[Revolution 11]

User likewhat at Arstechnica did a good summary of why they are removing this supercomputer.

Titan (the #1 super computer at the moment) cost $97 million for 17 times the performance, and less than 4 times the energy usage.
​

Based on the cost and performance of Titan, a super computer that would perform similarly to Roadrunner would cost about $6 million today. Roadrunner costs $1.2 million/year in power alone.
​

A new super computer with performance similar to Roadrunner would consume less than $0.3 million/year in power.
​

So taking into account power benefits alone, you would have a ROI in less than 7 years. That's not even taking into account space / staffing / development costs.
​

 

[Phynaz]

Does it gets scrapped now or re purposed in smaller bits or something?

This computer was used for classified purposes, therefore it will be shredded.

 

[Homeles]

User likewhat at Arstechnica did a good summary of why they are removing this supercomputer.

Titan (the #1 super computer at the moment) cost $97 million for 17 times the performance, and less than 4 times the energy usage.
​

Based on the cost and performance of Titan, a super computer that would perform similarly to Roadrunner would cost about $6 million today. Roadrunner costs $1.2 million/year in power alone.
​

A new super computer with performance similar to Roadrunner would consume less than $0.3 million/year in power.
​

So taking into account power benefits alone, you would have a ROI in less than 7 years. That's not even taking into account space / staffing / development costs.
​

That's pretty interesting. I'm actually a bit surprised that super computers cost that much. Thought they'd be more. The joys of buying in bulk, eh?

 

[Charles Kozierok]

Interesting to see perf/w being just as important in supercomputers vs tablets/phones.

Servers and mobile have always been the two areas where this matters; it's only desktop were p/W gets less attention. It's arguably even more important for servers, because not only do you have the power consumption of the electronics to deal with, but each extra watt of heat dissipated is another watt you have to pay remove from the server room.

(I didn't see if they included these secondary power costs in their figures.)

Intel started selling LV and ULV server chips around the same time that they did mobile chips. If I understand correctly, they are planning a significant move for Atom into the server arena as well.

 

[mv2devnull]

The power consumption could not have been much less back in 2009. The power bill had to be acceptable back then, when there was no alternative.

It is not really different from our desktops: I walk to shop to get then best thing ever, return with scrap metal, and begin to wait for the next best thing ever. Luckily, in different scale.

 

[Idontcare]

User likewhat at Arstechnica did a good summary of why they are removing this supercomputer.

Titan (the #1 super computer at the moment) cost $97 million for 17 times the performance, and less than 4 times the energy usage.
​

Based on the cost and performance of Titan, a super computer that would perform similarly to Roadrunner would cost about $6 million today. Roadrunner costs $1.2 million/year in power alone.
​

A new super computer with performance similar to Roadrunner would consume less than $0.3 million/year in power.
​

So taking into account power benefits alone, you would have a ROI in less than 7 years. That's not even taking into account space / staffing / development costs.
​

I feel that way about my $189 FX-8350 that costs me ~$100/yr in power costs. The cost of the hardware is less than the power bill after 2 or 3 yrs.

 

[Charles Kozierok]

Yep. IT people look at TCO, not price tags.

 

[itsmydamnation]

i wonder which one is actually better

http://www.top500.org/system/177975
http://www.top500.org/system/177556

my bet is bluegene, look how much "peak performance" is left on the table for titan on even something like linpack

 

[Idontcare]

my bet is bluegene, look how much "peak performance" is left on the table for titan on even something like linpack

I'm willing to bet that is the inefficiencies of the opterons that is driving that delta between theoretical and realized performance. On paper bulldozer was supposed to be a beast.

 

[Revolution 11]

I'm willing to bet that is the inefficiencies of the opterons that is driving that delta between theoretical and realized performance. On paper bulldozer was supposed to be a beast.

This may be a stupid question but why then did they use Opterons for supercomputers if it is so power-hungry/inefficient and power obviously being a critical component of costs? Did they really take AMD's word for how good Bulldozer would be?

 

[Idontcare]

This may be a stupid question but why then did they use Opterons for supercomputers if it is so power-hungry/inefficient and power obviously being a critical component of costs? Did they really take AMD's word for how good Bulldozer would be?

It was a contract with Cray that was setup long before bulldozer's hype was sanity-checked with reality (silicon).

 

[VirtualLarry]

It was a contract with Cray that was setup long before bulldozer's hype was sanity-checked with reality (silicon).

I heard something about custom interconnect ASICs, that interfaced directly with the Opteron CPUs (HT?). Stuff that would prevent dropping i n a different CPU/mobo on a whim.

Plus, Opterons aren't actually that bad for server workloads. Much better at those than desktop workloads. (Bulldozer was designed as a server chip first, desktop chip second.)

 

[itsmydamnation]

I'm willing to bet that is the inefficiencies of the opterons that is driving that delta between theoretical and realized performance. On paper bulldozer was supposed to be a beast.

Bulldozer can hit very close to its peak flops no problems on workloads more complex then linpack. Bulldozers problem is single thread performance not throughput.

I think you are wrong, GPU's achieve way lower average to peak flops on Linpack compared to bulldozer. Start running more complex workloads with much more complex memory access patterns and watch the GPU drop which was actually my actual point. Bulldozer holds performance very well on those workloads thax to its large amount of cache, you can see this especially comparing 4 core vishera to trinity.

 

[Idontcare]

I heard something about custom interconnect ASICs, that interfaced directly with the Opteron CPUs (HT?). Stuff that would prevent dropping i n a different CPU/mobo on a whim.

Plus, Opterons aren't actually that bad for server workloads. Much better at those than desktop workloads. (Bulldozer was designed as a server chip first, desktop chip second.)

AMD server market share doesn't reflect that, if true.

If you're curious about why Titan uses Opterons, the explanation is actually pretty simple. Titan is a large installation of Cray XK7 cabinets, so CPU support is actually defined by Cray. Back in 2005 when Jaguar made its debut, AMD's Opterons were superior to the Intel Xeon alternative. The evolution of Cray's XT/XK lines simply stemmed from that point, with Opteron being the supported CPU of choice.

http://www.anandtech.com/show/6421/inside-the-titan-supercomputer-299k-amd-x86-cores-and-186k-nvidia-gpu-cores/2

Incidentally, the Opteron processors used in the system are dual-chip CPUs based on the Bulldozer microarchitecture. We asked Sumit Gupta, General Manager for Tesla Accelerated Computing at Nvidia, why those CPU were chosen for this project, given the Xeon's current dominance in the HPC space. Gupta offered an interesting insight into the decision. He told us the contracts for Titan were signed between two and three years ago, and "back then, Bulldozer looked pretty darn good."

http://techreport.com/news/23808/nvidia-kepler-powers-oak-ridge-supercomputing-titan

Bulldozer can hit very close to its peak flops no problems on workloads more complex then linpack. Bulldozers problem is single thread performance not throughput.

I think you are wrong, GPU's achieve way lower average to peak flops on Linpack compared to bulldozer. Start running more complex workloads with much more complex memory access patterns and watch the GPU drop which was actually my actual point. Bulldozer holds performance very well on those workloads thax to its large amount of cache, you can see this especially comparing 4 core vishera to trinity.

Nvidia tells us Titan can achieve over 20 petaflops of peak performance, and over 90% of those flops come from its GK110 GPUs.

http://techreport.com/news/23808/nvidia-kepler-powers-oak-ridge-supercomputing-titan

 

[loccothan]

he he ilike it

 

[itsmydamnation]

Idontcare what you posted doesn't backup your point or contradict what i said. look at something like this http://www.hpcadvisorycouncil.com/events/2012/European-Workshop/Presentations/12_AMD.pdf as you can see the peak flops vs linpack score difference in titan aren't coming from the bulldozer cores.

http://www.top500.org/system/177996
http://www.top500.org/system/10587
im guessing the delta here is caused by those POS intel cores....... they look so good on paper......

 

[Denithor]

This computer was used for classified purposes, therefore it will be shredded.

This is something I've wondered about for a while - with the extreme level of processing power available in these supercomputers - what exactly do they use them for?