Question EPYC 9124 (GENOA, SP5) heatsink suggestions welcome

[Pisi]

Building a NAS server (mostly data storage, no major computational load envisioned) around the EPYC 9124 (SP5, 200W TDP). To be used in a regular uATX case, i.e. no rackmount, no huge jet-turbine-like datacenter fans involved. Should I attempt a passive solution, or should I go with an active heatsink?

(Does Noctua make CPU coolers for the SP5 socket? I couldn't find any!)

 

[Markfw]

use this

AMD EPYC 9004 CPU 4U SP5 Cooler Fans 6 tubes Nickel plating on copper sp5 | eBay

All products we send are declared on the lowest standard, fees.

www.ebay.com

 

[Pisi]

use this

Looks awesome, thanks! It ships from China though, and I am too impatient to wait that long. What do you think of this one?

Btw, do they come with thermal paste, or should I procure my own?

 

[Markfw]

I don't remember how long it took, but it was not weeks. Maybe a week. All I can day is at least the one that first came up (I have 3 of these) ordered Nov 25, 2023 and delivered Dec 1st 2023. The other was ordered May 26, same year and delivered Jun 1. different providers. I have ordered from this supplier before, and I can't say for sure, but the timing would be the same I guess. That one by supermicro looks like a 92mm fan, and I have one like that and it loud like a jet plane vs 2 120mm fans.

Edit: Thermal paste: all 3 use this methods, and it works. But I had to redo the pase on one, as I had to change CPUs. mine are on 9554's. If you get it, make sure you torque it in order as the spec from the motherboard using the threadripper tool ot to 13.3 inch pounds.

 

[Pisi]

That one by supermicro looks like a 92mm fan, and I have one like that and it loud like a jet plane vs 2 120mm fans.

Yeah, I've been told it might be loud. I will cancel it, and order the one you recommended, thanks.

 

[Markfw]

Yeah, I've been told it might be loud. I will cancel it, and order the one you recommended, thanks.

BTW, on my 9554's with all 128 threads loaded it is fairly quiet and runs as high as 87C, but it can go to 100c. Also, it does 3.5 ghz on all 128 threads ! On a 16 core, damn it should be fast !

 

[Pisi]

@Markfw: Alright, I got everything I need, finally, and just assembled them all together ... only the mobo doesn't power up!!! Not sure I made the right power connections though: mobo has 3x8 pin + 1x4 pin power inputs. I only connected one of the three 8 pin inputs: should I connect them all? The 4 pin is being powered via 18-to-24-to-4 pins cables.

Here's what the PSU looks like:

 

[Markfw]

What mobo is this again ? You need ALL of the 8 pin cpus populated, but I have never seen a motherboard that takes more than 2. and a 4 pin also ????

 

[Pisi]

What mobo is this again ? You need ALL of the 8 pin cpus populated, but I have never seen a motherboard that takes more than 2. and a 4 pin also ????

This one: https://www.asrockrack.com/general/productdetail.asp?Model=GENOAD8UD-2T/X550#Specifications

 

[Markfw]

This one: https://www.asrockrack.com/general/productdetail.asp?Model=GENOAD8UD-2T/X550#Specifications

OK, then plug in ALL the 8 pin and the 4 pin. Clearly they are all power.

"1 (4-pin, ATX PSU signal) w/ ATX 24-pin adapter cable, 3 (8-pin, ATX 12V) support 12V DC-in"

Edit: actually, you don't have a 4 pin power in addition to the 3 8 pin.

Now motherboard ??? Take that one back ??? my

Supermicro Server Motherboard MBD-H13SSL-N Socket SP5​

Only requires 2 8 pin. + 24 power.

 

[Markfw]

If its easier to return the PSU (first see if you can find one with 3 8-pin CPU + one 4 pin) then take the PSU back. one of them has to go.

EDIT: I just thought of this. Call ASRock tech support and tell then you only have 3 6+2 CPU plugs, and ask them what to do.

 

[Pisi]

If its easier to return the PSU (first see if you can find one with 3 8-pin CPU + one 4 pin) then take the PSU back. one of them has to go.

EDIT: I just thought of this. Call ASRock tech support and tell then you only have 3 6+2 CPU plugs, and ask them what to do.

I just bought another 8-pin cable: it should arrive tomorrow morning. In the meantime let me see if the mobo powers up with only 2 of the 8-pins populated.

 

[Pisi]

Just as an update: it turns out that the Epyc 9124 I purchased from eBay was locked for Dell server motherboards ...

As such, I had to abandon the build, and send everything back.

 

[DrMrLordX]

Just as an update: it turns out that the Epyc 9124 I purchased from eBay was locked for Dell server motherboards ...

As such, I had to abandon the build, and send everything back.

Back when AMD started doing this, people speculated that locked parts could wind up on eBay. And wouldn't you know it, they did. What a load of horse manure. Sorry you had to go through all that trouble.

 

[trfyghlijo]

@Markfw Piggybacking here since I am considering both motherboards in this thread for Turin (https://www.asrockrack.com/general/productdetail.asp?Model=GENOAD8UD-2T/X550#Specifications and https://www.supermicro.com/en/products/motherboard/h13ssl-n) - I like the latter for the 12 slots for RAM instead of 8, but like the former's 4 pcie gen5 x16 slots. I eventually might want to do 4x GPUs for LLMs which is the call, but do you think the H13SSL-N with only 3 x16 slots would work for this (meaning one GPU has to run in x8)? I wish something had both 12 ram slots AND 4 x16 slots to not have to decide, but so few turin non-proprietary boards...

 

[Markfw]

@Markfw Piggybacking here since I am considering both motherboards in this thread for Turin (https://www.asrockrack.com/general/productdetail.asp?Model=GENOAD8UD-2T/X550#Specifications and https://www.supermicro.com/en/products/motherboard/h13ssl-n) - I like the latter for the 12 slots for RAM instead of 8, but like the former's 4 pcie gen5 x16 slots. I eventually might want to do 4x GPUs for LLMs which is the call, but do you think the H13SSL-N with only 3 x16 slots would work for this (meaning one GPU has to run in x8)? I wish something had both 12 ram slots AND 4 x16 slots to not have to decide, but so few turin non-proprietary boards...

I am considering the asrock motherboard if I can't get the supermicro up and running., but yes, handicapping the Turin worth only 8 channel ram is hurtful to think of. There is even a benchmark done that proves it is bad.

 

[trfyghlijo]

Your turin still isn't running? Darn I'm sorry, you've been trying a long time.
Maybe I'll just do H13SSL-NT and worry about 4 GPUs issue later...

 

[StefanR5R]

A PCIe x8 interface is not a problem if the data transfers to/from GPU are infrequent: Load data onto GPU, do loads of processing in VRAM only, save data off of GPU.

An 8-channel RAM interface is not a problem if you have only few program threads doing heavy RAM accesses, or have many threads but acting on datasets which fit well into CPU caches.

*** I don't know at all whether or not your workload is going to be of either type. ***

(You might be able to benchmark your application on an AMD AM5 or Intel LGA1700 single-GPU system: Test with x16 and with x8 PCIe, test with 2-channel and 1-channel RAM population.)

An alternative to an EPYC based host could be the Threadripper platform. Caveats: Threadripper 9000 is not announced yet; Threadripper 7000 tops out at 4 RAM channels and has some PCIe restrictions compared to Threadripper PRO 7000; Threadripper PRO 7000 tops out at 8 RAM channels; Threadripper 7000 is expensive compared to many EPYC models, and Threadripper PRO 7000 is even more expensive. On the upside, if your application is sensitive to CPU clocks, Threadripper (PRO) may get you further than EPYC.

(Apropos, you could, or in fact should, benchmark your application on an AM5 or LGA1700 system with different upper limits of the CPU clock, before you choose platform and processor of your quad GPU host.)

Gigabyte aims at model training/ model tuning/ machine learning specifically with this board: TRX50 AI TOP. If you put a Threadripper PRO 7000 in it, you get 8 RAM channels and 4 PCIe5 x16 slots. With a Threadripper 7000, this is toned down to 4 RAM channels and 3 PCIe5 x16 + 1 PCIe4 x16.

PS:
On Windows, you will need more PCIe bandwidth to GPGPU accelerators than on Linux. That's because the Windows graphics driver stack forces more copying from/to VRAM, in order to enable features of the software stack such as driver crash dumps and live driver updates; features which are not present with the leaner and meaner Linux driver stack. At least that's what I last heard about the topic; I don't know if it is still true with Windows 11.

 

[trfyghlijo]

Thanks so much @StefanR5R! No specific workload in mind yet but I tinker with computationally-heavy workloads frequently and would like to run LLMs a year down the road. I'm trying to get an 'all-rounder' board that could work decently well for any variety of future applications.

TRX50 looks very nice, but I will probably go with this EPYC: https://www.ebay.com/itm/176040155214?_skw=genoa&itmmeta=01JA5XTQ8QG5T9TD7P7XNM4HBZ then upgrade to a cheaper used Turin in a few years (it's for more of a homelab than a high-stakes commercial environment so QS seems okay to me). I may just need to pick between 12-channel RAM and 4 x16 PCIE slots as I'm not patient enough to wait for new motherboards with both. 3 2-slot GPUs max may be fine if there ends up being a penalty for running a fourth in one of the in-between x8 slots, in parallel with the three x16s. Will probably put it in this case: https://www.silverstonetek.com/en/product/info/server-nas/RM51/.

 

[StefanR5R]

In typical GPGPU applications, there is just a single software thread doing the heavy lifting (polling the GPU, copying between RAM and VRAM). If that's also true with your applications, 8 RAM channels should be good for a quad GPU system. Each one of the then four "main" threads should be coerced to run on an own exclusive CCD, in order to maximize RAM access bandwidth of the EPYC.

If your applications stress the GPUs' memory controllers most, cooling in a high air flow case with merely 2 slots per GPU should be good. But if your applications pull a lot of power through the GPUs' SMs, then perhaps an open frame mining rig might be preferable.

 

[StefanR5R]

PS, I was about to suggest the Software/Programming subforum to you to ask for what the CPU and GPU bottlenecks in your field are going to be. But from a quick glance, there doesn't seem to be much related discussion. Neither in the Graphics Cards subforum. :-|

 

[StefanR5R]

@trfyghlijo, another consideration *if* lots of RAM bandwidth is desired while only relatively few software threads are emitting the heavy accesses: In such a scenario, an EPYC SKU in which the CCDs are configured to wide GMI would be optimal.

AMD only said in which cases they can wire the CCDs up in wide GMI:
– Genoa SKUs with up to 4 CCDs,
– Turin SKUs with up to 8 CCDs.
However, I am not aware of any lists of which SKUs actually do have wide GMI enabled.

4 CCD Geonas:
9124 (16c), 9224 (24c), 9254 (24c), 9334 (32c)
All other Genoas have 8 CCDs (or 12 CCDs in case of the 84c and 96c models).

 

[Markfw]

@Markfw Piggybacking here since I am considering both motherboards in this thread for Turin (https://www.asrockrack.com/general/productdetail.asp?Model=GENOAD8UD-2T/X550#Specifications and https://www.supermicro.com/en/products/motherboard/h13ssl-n) - I like the latter for the 12 slots for RAM instead of 8, but like the former's 4 pcie gen5 x16 slots. I eventually might want to do 4x GPUs for LLMs which is the call, but do you think the H13SSL-N with only 3 x16 slots would work for this (meaning one GPU has to run in x8)? I wish something had both 12 ram slots AND 4 x16 slots to not have to decide, but so few turin non-proprietary boards...

OK, I ordered that motherboard, but the PSU is odd. I ordered this:

https://www.amazon.com/gp/product/B0CKY88GMT/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

As it has 4 8 pin cpu inputs, that should give me the 3.5 8 pin that are required ? If thats wrong, please link one that will work.

 

[StefanR5R]

Re GENOAD8UD-2T/X550:
I haven't checked the motherboard manual whether or not all three ATX12V connectors need to be populated. If yes, but if your PSU only has got two of the respective cables, then
– check if the PSU manufacturer sells additional cables,
– 3rd party cable manufacturers might make compatible cables,
– PCIe power to ATX12V adapter cables exist (eg. 2x 6pin PCIe to 1x 8pin ATX12V).

("8-pin ATX12V" is the same as "EPS", search for either one.)

Edit, I looked through the manual now and it isn't entirely clear to me. But it seems as if all three ATX12V connectors are meant to be populated.

 

[Markfw]

Re GENOAD8UD-2T/X550:
I haven't checked the motherboard manual whether or not all three ATX12V connectors need to be populated. If yes, but if your PSU only has got two of the respective cables, then
– check if the PSU manufacturer sells additional cables,
– 3rd party cable manufacturers might make compatible cables,
– PCIe power to ATX12V adapter cables exist (eg. 2x 6pin PCIe to 1x 8pin ATX12V).

("8-pin ATX12V" is the same as "EPS", search for either one.)

Edit, I looked through the manual now and it isn't entirely clear to me. But it seems as if all three ATX12V connectors are meant to be populated.

well, we will see when the PSU shows up. and something about "1 (4-pin, ATX PSU signal) w/ ATX 24-pin adapter cable", I hope that means its included. Why not the standard 24 pin and 2 8 pin like all other server mobos ??