Discussion [Tomshardware] EPYC Genoa and Radeon Instinct to Power Two-Exaflop DOE Supercomputer

[DisEnchantment]

EPYC win for AMD for two exaflops exascale supercomputer.
EPYC Genoa + Future Radeon Instinct
Zen4 + DDR5 + Radeon Instinct

Big win for ROCm and users of open source SW.

AMD Wins El Capitan: EPYC Genoa and Radeon Instinct to Power Two-Exaflop DOE Supercomputer

AMD hits another home run

www.tomshardware.com

I hope this gives a big boost to ROCm.
As a Linux user I am really looking forward to ROCm support upstreamed for most frameworks and the community can benefit without having to reverse engineer their ass out to find bugs and root causes if something is not working in the SW.

(Intel is also a big OSS contributor so I hope they win some too)


Update 1
AT Link

El Capitan Supercomputer Detailed: AMD CPUs & GPUs To Drive 2 Exaflops of Compute

www.anandtech.com

However the most interesting claim is that these IF 3.0 device nodes will support unified memory across the CPU and GPU, which is something AMD doesn’t offer today.

I hope it means what it sounds.
That we can write code which treat memory across the GPU as how we treat memory shared with another thread/core on the CPU? Sounds like a dream come true.


Best bit for me

Scott said, “As part of this procurement, the Department of Energy has provided additional funds beyond the purchase of the machine to fund non-recurring engineering efforts and one major piece of that is to work closely with AMD on enhancing the programming environment for their new CPU-GPU architecture.” Work is ongoing by all three partners to take the critical applications and workloads forward and optimize them to get the best performance in the machine when El Capitan is delivered.

Exascale Watch: El Capitan Will Use AMD CPUs & GPUs to Reach 2 Exaflops

HPE and its collaborators reported today that El Capitan, the forthcoming exascale supercomputer to be sited at Lawrence Livermore National Laboratory and serve the National Nuclear Security Administration (NNSA), will […]

www.hpcwire.com

Update 2
It is time to start a new Zen 4 thread

 

[JasonLD]

Hmmm, interesting that it uses Genoa, i would have assumed it will be using Zen 5.
Does it mean Genoa will not come out until 2022 at the earliest?

 

[prtskg]

Yeah, good win. ROCm is also receiving some funding. Honestly though, I was expecting IBM+Nvidia to win this one. One Intel, one AMD and one by IBM+Nvidia. Guess products from two different companies raises cost.

 

[prtskg]

Hmmm, interesting that it uses Genoa, i would have assumed it will be using Zen 5.
Does it mean Genoa will not come out until 2022 at the earliest?

Why do you think it can't come in 2021? DDR5?

 

[Hitman928]

Hmmm, interesting that it uses Genoa, i would have assumed it will be using Zen 5.
Does it mean Genoa will not come out until 2022 at the earliest?

My guess is Zen3 gets released 4Q this year, still on AM4 and then AMD gives a bit more time between generations to allow for the transition to AM5 (or whatever they call it) and the cooresponding server socket which means that Zen4 (Genoa) will be released late 1H 2022. Just a guess though.

 

[amrnuke]

Fairly incredible, even just looking at efficiency. Frontier 30MW for 1.5 EF, El Capitan "fairly substantially under" 40MW for 2 EF means that efficiency is expected to continue to increase substantially.

Hmmm, interesting that it uses Genoa, i would have assumed it will be using Zen 5.
Does it mean Genoa will not come out until 2022 at the earliest?

Frontier will likely be on Zen3 (expected release this year) but is being completed in 2021. However, close customers are getting Zen3 earlier.
Same with Zen4. LLNL could conceivably get Zen4 in 2021, but actually building this massive of a system requires time I'm guessing.

 

[JasonLD]

Why do you think it can't come in 2021? DDR5?

Usually the supercomputer gets the latest or even yet commercially unavailable tech available at the time of its build. It is just my assumption that sole latest EPYC product that AMD will have for 2022 is Zen 4 Genoa. If Zen 5 was available (or close to release), then it would have used that.

My guess is Zen3 gets released 4Q this year, still on AM4 and then AMD gives a bit more time between generations to allow for the transition to AM5 (or whatever they call it) and the cooresponding server socket which means that Zen4 (Genoa) will be released late 1H 2022. Just a guess though.

That sounds possible. Or we might get Milan refresh in 2021 and Genoa in 2H 2022.

 

[DisEnchantment]

I think Zen5 would not leave much wiggle room should there be issues. Besides the system will come online in Early 2023.
I am imagining something like this, Zen 3 Q2-Q3 2020, Zen4 Q3-Q4 2021, Zen5 Q4 2022-Q1 2023. around 14-16 months as mentioned by Lisa in the Earnings call.

And to think of this, Mike Clark was starting work on Zen5 high level architecture in late 2018 already as mentioned in their video with John Taylor !!!

 

[Topweasel]

Usually the supercomputer gets the latest or even yet commercially unavailable tech available at the time of its build. It is just my assumption that sole latest EPYC product that AMD will have for 2022 is Zen 4 Genoa. If Zen 5 was available (or close to release), then it would have used that.



That sounds possible. Or we might get Milan refresh in 2021 and Genoa in 2H 2022.

You have to look at time for implementation, CPU shipments and stuff like that. Not only that but its not like when AMD and Intel are working through ODM's. AMD will be working with Cray who will be then creating this and installing it. Don't know exact numbers but assume this is the 20k of AMD's bestest chips at the time. They could be shipping pre-release Zen 4. After the design is final but hold off shipping out seeds and stuff to other server clients. But 20k of the bestest fastest EPYC configurations would tremendously slow down retail CPU shipping.

My personal guess is this is a 2022 year long project and the server comes online well before 2023 and that 2023 is when the full system is finished.

 

[eek2121]

They left out just enough details to keep us from getting an idea of what Genoa is capable of.

Regarding the launch date: it takes time to source the parts and put together a freakin super computer! 😉

 

[DisEnchantment]

oops ...

 

[Vattila]

Impressive!

So an El Capitan node will look something like this, perhaps, with socketed GPUs with HBM on interposer, and cache-coherent Infinity Fabric 3 interconnect on the motherboard:

 

[uzzi38]

It is time to start a new Zen 4 thread

https://twitter.com/x/status/1235276453467631622

Oh yes it most definitely is. We get details tomorrow.

Squeals

 

[jpiniero]

What would be nice would be for them to put GPU chiplets on the package. And an active interposer...

 

[soresu]

EPYC win for AMD for two exaflops exascale supercomputer.
EPYC Genoa + Future Radeon Instinct
Zen4 + DDR5 + Radeon Instinct

Big win for ROCm and users of open source SW.

AMD Wins El Capitan: EPYC Genoa and Radeon Instinct to Power Two-Exaflop DOE Supercomputer

AMD hits another home run

www.tomshardware.com

I hope this gives a big boost to ROCm.
As a Linux user I am really looking forward to ROCm support upstreamed for most frameworks and the community can benefit without having to reverse engineer their ass out to find bugs and root causes if something is not working in the SW.

(Intel is also a big OSS contributor so I hope they win some too)


Update 1
AT Link

El Capitan Supercomputer Detailed: AMD CPUs & GPUs To Drive 2 Exaflops of Compute

www.anandtech.com

I hope it means what it sounds.
That we can write code which treat memory across the GPU as how we treat memory shared with another thread/core on the CPU? Sounds like a dream come true.


Best bit for me

Exascale Watch: El Capitan Will Use AMD CPUs & GPUs to Reach 2 Exaflops

HPE and its collaborators reported today that El Capitan, the forthcoming exascale supercomputer to be sited at Lawrence Livermore National Laboratory and serve the National Nuclear Security Administration (NNSA), will […]

www.hpcwire.com

Update 2
It is time to start a new Zen 4 thread

I must be confused.... wasn't the whole point of Fusion/HSA a unified memory architecture?

Wasn't that already fully realised as of Volcanic Islands/GCN3?

 

[soresu]

I wonder what the GPU is descended from, Arcturus or RDNA2.....

 

[uzzi38]

Impressive!

So an El Capitan node will look something like this, perhaps, with socketed GPUs with HBM on interposer, and cache-coherent Infinity Fabric 3 interconnect on the motherboard:

Maybe not.

Just a bit of a guess of mine, but I'd say there's a possibility both Frontier and El Capitan utilise SoIC to get both CPU and GPU on custom packages.

 

[Atari2600]

I must be confused.... wasn't the whole point of Fusion/HSA a unified memory architecture?

Wasn't that already fully realised as of Volcanic Islands/GCN3?

Alright Kenobi - says this in the ananad piece:

For Infinity Fabric 3.0, AMD is promising further improvements to inter-chip bandwidth and latency. However the most interesting claim is that these IF 3.0 device nodes will support unified memory across the CPU and GPU, which is something AMD doesn’t offer today. Indeed even Frontier is only slated to offer coherency between the processors which is a step below a true unified memory model.

I do realise the goal of fusion - circa 10 years ago now! - was unified memory architecture - and thought given the introduction of the High Bandwidth Cache Controller on Vega we were getting very close. But then they stepped back from that a bit.

 

[DisEnchantment]

Apparently this is not all there is to this deal.
There will be a smaller scale clone of the El Capitan which will be available for use to the general scientific community

There will be a “smaller clone” of El Capitan that does not have a name that will be available for the broader scientific community. This smaller system should be faster than the current #1 Summit supercomputer.

HPE-Cray and AMD Win Again with El Capitan 2 Exaflop Supercomputer

HPE-Cray Shasta and Slingshot sweep the third US DoE exascale system. AMD Zen4 and AMD GPUs power El Capitan shutting IBM, NVIDIA, and Mellanox out

www.servethehome.com

BTW, GENCI will be announcing an 'exascale class' system from Atos using AMD EPYC. Not sure about the GPU though.

 

[soresu]

Alright Kenobi - says this in the ananad piece:

Finally someone gets the soresu reference!

 

[soresu]

Interesting that one slide says "Next Generation High Bandwidth Memory".

Perhaps the HBM3 spec is finally nearing completion.

 

[Thunder 57]

Finally someone gets the soresu reference!

Please, enlighten the rest of us that don't yet understand.

 

[maddie]

Please, enlighten the rest of us that don't yet understand.

Google is your friend.

 

[Markfw]

Google is your friend.

Well, I googled it, but I will post it here for those that don't want to spend the time.

"Form III, also known as Soresu, the Way of the Mynock, or the Resilience Form, was the third of the seven classic forms of lightsaber combat that was recognized by the Jedi Council prior to and during the Clone Wars "

 

[thesmokingman]

Well, I googled it, but I will post it here for those that don't want to spend the time.

"Form III, also known as Soresu, the Way of the Mynock, or the Resilience Form, was the third of the seven classic forms of lightsaber combat that was recognized by the Jedi Council prior to and during the Clone Wars "

So.., something ike this?