Just got a 9654 Genoa ** Motherboard /ram/heatsink/CPU are here !!

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
Torqueing the heatsink is no easy task like the SP3 socket was. DAMN it takes a long time, and a lot of effort.
 

vityan666

Member
Apr 12, 2023
42
31
51
Hi Markfw,
I've just registered after seeing this thread a few days ago, as it touches relatively rare thing of making Epyc workstation in a large 4U rack, as it's mostly end-up being small rack builds.
My case is even more complicated as I'll put it in regular PC case(While it's huge Corsair 1000D SuperTower it's still not server case). So only large 120mm+ fans, and relatively quiet operation.

So, to start on, I'm in the similar boat right now, going through a process to upgrade my R3900X based workstation/server with AiO on a CPU. Now after 10+ years I'll have to go back to air, and even air coolers options are sparse as hell.
I hope new TR's will use SP5 and not SP6 so we will get new air/water AiO coolers from standard manufacturers.

In my case, I choose to go with Epyc 9274F due to it's great base frequency(I care less about boost speeds, and almost only locked my CPU's to single frequency - for old lovely UT99 game).
The mobo is the same Supermicro H13SSL-NT, which took a week to catch for availability on Newegg, but its here.
Also one Micron RDIMM 2R DDR5 4800 memory module arrived(From one seller), 5 more on the way(From another), and today got a deal on eBay for 2 extra used modules for only 120$ for both of them, so instead of originally planned 6 I will go for 8 modules,
thus 8 channels.

I wanted to inform you about the cooler part, as I myself looked hard around the web, and while AliExpress did have only the small variant like you purchased, their parent company AliBaba had the large one which I've ordered in the middle of March and
it arrived 2-3 days after you received your cooler. Fortunately this seller doesn't have min. units requirement and we can order just one, but it takes about 2 weeks for him to ship, so patience is needed. I've attached my two photos with this cooler. The thing is huge,
and not any 4U rack has enough space for it.
Url to the cooler:

Regarding disks, the time is flying fast and industry can change without us even noticing. Just less than 1.5y ago I've bough Seagate Nytro 3.2TB SAS SSD, 3DWPD for ~1500$, and a week ago my good friend told me that he recently bough
Micron 7450 Enterprise SSD 15.36TB 1DWPD U.2 for 1550$ only for a client company locally.
This disk is available on Newegg as well for a nice price, and also can be purchased in lower densities or the 'Max' edition(3DWPD).

I've personally bought its successor Micron 9400 Pro 7.68TB Enterprise SSD U.3 for 1200$ on NewEgg and MCIO->U.2 cable from eBay to connect it to the mobo.
I don't see any reason to waste $ on those consumer m.2 disks then serious stuff became that accessible(9400 costs like Samsung's 980 Pro, while being orders of magnitude better product).
I already had Samsung 970 Evo Plus 2TB starting to show bad sectors within 1 year of use(It was replaced with Seagate Nytro immediately as stability and data consistency is #1 for my server system which shall run 24x7x365). Then I repurposed this drive(M.2) for non important stuff and didn't accessed it for a few months it just 'reinitialized' and became empty. I stopped using it completely since then, nor relying on any M.2 chips.

2 Corsair DDR4 DIMM's failed on me within one year causing a lot of headache(First one was for the 96GB-128GB address range and was hard to catch as it only caused problems with extra VM's I've loaded occasionally).
It was the last strain to my decision for ECC memory and resulted in plans for this build(Threadippers come to late and change socket every generation, while their cost became similar to Epyc's. Genoa's F models broke the 4Ghz barrier on a base which is a great thing by itself).
Consumer products and DDR5 design disallows for fast 2DPC DDR5 + 4 DIMMS operation, because AMD and Intel can't get their shit together and make Quad-Channel memory controllers standard with DDR5. UDIMM-ECC are so rare and then you need to look for a mobo that supports it, Waiting for one faster than JEDEC speed to appear, hoping that BIOS update will not break it, etc, etc. Thus, the only good option left is Server CPU + RDIMM. AMD took case of server RDIMM yearly speed increase(So instead of 2133 forever we had 2666,2933,3200)
and now Intel take this even further with Sapphire Rapids, by introducing overclockable RDIMMS(Took only 30 years...) - Not that our Epycs will enjoy oveclocked RDIMMs in this generation but it's a good start.
The only problem is lack of chipset for USB ports and Sound. USBs is solved with PCI-E cards and those solve the Audio problem with USB sound card.
If only those new boards had 7 PCI-E slots like their predecessors, but I suspect it's because of PCI-E 5 signaling which again shortened the max range from the CPU.
ASrock tries to overcome it with zillion of PCI-E switches in new planned board:


Regarding your first boot problem,
did you try to boot it with minimal set of hardware, i.e 1 DIMM and w/o GPU, using the BMC?
I know it could be a problem with modern monitor, which will require a rare device like this one:

Also, while you tinkering, I'm interested for my own sake,
Can GPU fit into the bottom(5-th) PCI-E slot without pushing/breaking Front-Panel connectors(Power,Reset,LEDS). USB front panel connector is of no interest as I've ordered 90 degree adapter to reroute it oout of extension card way.

If GPU will not fit, I'll put my Areca 1883-IX-12 RAID controller in there and 3-fans extra cooler below it as it(Fans card) doesn't need PCI-e power.
 

Attachments

  • 20230410_035209.jpg
    20230410_035209.jpg
    525.5 KB · Views: 12
  • 20230410_035213.jpg
    20230410_035213.jpg
    526.7 KB · Views: 13

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
Leave it on. Maybe it's the memory training delay. Remember just four DDR5 DIMMs take 7 minutes. You have 12!
I have waited over 1/2 hour. I used BMC and queried the system. It does NOT see the CPU. Then (as suggested) I unmounted the CPU, cleaned the pins and remounted and retried. No change. Here is the socket and the cpu bottom (pics may not be the greatest, but they sure look good to me)20230413_084208[1].jpg20230413_084446[1].jpg
 

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
Hi Markfw,
I've just registered after seeing this thread a few days ago, as it touches relatively rare thing of making Epyc workstation in a large 4U rack, as it's mostly end-up being small rack builds.
My case is even more complicated as I'll put it in regular PC case(While it's huge Corsair 1000D SuperTower it's still not server case). So only large 120mm+ fans, and relatively quiet operation.

So, to start on, I'm in the similar boat right now, going through a process to upgrade my R3900X based workstation/server with AiO on a CPU. Now after 10+ years I'll have to go back to air, and even air coolers options are sparse as hell.
I hope new TR's will use SP5 and not SP6 so we will get new air/water AiO coolers from standard manufacturers.

In my case, I choose to go with Epyc 9274F due to it's great base frequency(I care less about boost speeds, and almost only locked my CPU's to single frequency - for old lovely UT99 game).
The mobo is the same Supermicro H13SSL-NT, which took a week to catch for availability on Newegg, but its here.
Also one Micron RDIMM 2R DDR5 4800 memory module arrived(From one seller), 5 more on the way(From another), and today got a deal on eBay for 2 extra used modules for only 120$ for both of them, so instead of originally planned 6 I will go for 8 modules,
thus 8 channels.

I wanted to inform you about the cooler part, as I myself looked hard around the web, and while AliExpress did have only the small variant like you purchased, their parent company AliBaba had the large one which I've ordered in the middle of March and
it arrived 2-3 days after you received your cooler. Fortunately this seller doesn't have min. units requirement and we can order just one, but it takes about 2 weeks for him to ship, so patience is needed. I've attached my two photos with this cooler. The thing is huge,
and not any 4U rack has enough space for it.
Url to the cooler:

Regarding disks, the time is flying fast and industry can change without us even noticing. Just less than 1.5y ago I've bough Seagate Nytro 3.2TB SAS SSD, 3DWPD for ~1500$, and a week ago my good friend told me that he recently bough
Micron 7450 Enterprise SSD 15.36TB 1DWPD U.2 for 1550$ only for a client company locally.
This disk is available on Newegg as well for a nice price, and also can be purchased in lower densities or the 'Max' edition(3DWPD).

I've personally bought its successor Micron 9400 Pro 7.68TB Enterprise SSD U.3 for 1200$ on NewEgg and MCIO->U.2 cable from eBay to connect it to the mobo.
I don't see any reason to waste $ on those consumer m.2 disks then serious stuff became that accessible(9400 costs like Samsung's 980 Pro, while being orders of magnitude better product).
I already had Samsung 970 Evo Plus 2TB starting to show bad sectors within 1 year of use(It was replaced with Seagate Nytro immediately as stability and data consistency is #1 for my server system which shall run 24x7x365). Then I repurposed this drive(M.2) for non important stuff and didn't accessed it for a few months it just 'reinitialized' and became empty. I stopped using it completely since then, nor relying on any M.2 chips.

2 Corsair DDR4 DIMM's failed on me within one year causing a lot of headache(First one was for the 96GB-128GB address range and was hard to catch as it only caused problems with extra VM's I've loaded occasionally).
It was the last strain to my decision for ECC memory and resulted in plans for this build(Threadippers come to late and change socket every generation, while their cost became similar to Epyc's. Genoa's F models broke the 4Ghz barrier on a base which is a great thing by itself).
Consumer products and DDR5 design disallows for fast 2DPC DDR5 + 4 DIMMS operation, because AMD and Intel can't get their shit together and make Quad-Channel memory controllers standard with DDR5. UDIMM-ECC are so rare and then you need to look for a mobo that supports it, Waiting for one faster than JEDEC speed to appear, hoping that BIOS update will not break it, etc, etc. Thus, the only good option left is Server CPU + RDIMM. AMD took case of server RDIMM yearly speed increase(So instead of 2133 forever we had 2666,2933,3200)
and now Intel take this even further with Sapphire Rapids, by introducing overclockable RDIMMS(Took only 30 years...) - Not that our Epycs will enjoy oveclocked RDIMMs in this generation but it's a good start.
The only problem is lack of chipset for USB ports and Sound. USBs is solved with PCI-E cards and those solve the Audio problem with USB sound card.
If only those new boards had 7 PCI-E slots like their predecessors, but I suspect it's because of PCI-E 5 signaling which again shortened the max range from the CPU.
ASrock tries to overcome it with zillion of PCI-E switches in new planned board:


Regarding your first boot problem,
did you try to boot it with minimal set of hardware, i.e 1 DIMM and w/o GPU, using the BMC?
I know it could be a problem with modern monitor, which will require a rare device like this one:

Also, while you tinkering, I'm interested for my own sake,
Can GPU fit into the bottom(5-th) PCI-E slot without pushing/breaking Front-Panel connectors(Power,Reset,LEDS). USB front panel connector is of no interest as I've ordered 90 degree adapter to reroute it oout of extension card way.

If GPU will not fit, I'll put my Areca 1883-IX-12 RAID controller in there and 3-fans extra cooler below it as it(Fans card) doesn't need PCI-e power.
The GPU can only fit in the 3rd. even then it almost hits the pins at the end.

Edit: I booted with 1 stick of ram, no NVME, no video and not even the usb 3.0 connector. BMC is up, still the same splash screen.
 

vityan666

Member
Apr 12, 2023
42
31
51
1681403065382.png
May be just the light direction, but are the marked spots from socket screenshot are visually fine?

The left-center side of the CPU has white splash on it - Is it scratch or a thermal compound? Anyway, looks it's on unused area and should not harm.
 
Last edited:

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
Here is another show without the flashlight
20230413_084212[1].jpg
 

vityan666

Member
Apr 12, 2023
42
31
51
I booted with 1 stick of ram, no NVME
I'm not yet sure if CPU or Mobo is faulty.
Would be worth to try another DIMM in main slot and alternatively some other slot, to allow hwinit procedures on the mobo to go further in case of bad DIMM(It can sometimes stuck before detecting other hw), while clearing the CMOS to reset it before each test.
The BIOS version is already 1.1a?
 
Last edited:

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
I'm not yet sure if CPU or Mobo is faulty.
Would be worth to try another DIMM in main slot and alternatively some other slot, to allow hwinit procedures on the mobo to go further in case of bad DIMM(It can sometimes stuck before detecting other hw), while clearing the CMOS to reset it before each test.
The BIOS version is already 1.1a?
I have received word from supermicro, but I think even they are wrong. The snoop post code is 22. They say its memory/pcie. I think its probably vendor locked.
 
  • Like
Reactions: phonitor

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
So... I personally think its a vendor locked CPU. So I am returning it, and I have a NEW RETAIL 9654 coming Sunday. So in 2 days we will know if I am right, and the memory is fine. I would says just in time for the primegrid challenge , but it even if the time is right, I can not add primegrid as a project to any boxes right now.
 
  • Like
Reactions: phonitor

H T C

Senior member
Nov 7, 2018
549
395
136
Noctua has a prototype in action for a 700 W 56-core Intel Xeon W9-3495X: if they are developing "such a mammoth" for this socket, perhaps they are also developing "a similar mammoth" for SP5 socket, and we simply ARE NOT AWARE of it, just yet.


After all, 400W is A LOT EASIER to cool down than 700W, no?
 

Markfw

Moderator Emeritus, Elite Member
May 16, 2002
25,439
14,409
136
Noctua has a prototype in action for a 700 W 56-core Intel Xeon W9-3495X: if they are developing "such a mammoth" for this socket, perhaps they are also developing "a similar mammoth" for SP5 socket, and we simply ARE NOT AWARE of it, just yet.


After all, 400W is A LOT EASIER to cool down than 700W, no?
Unless thats not real, my new 400 watt cooler (due in 2-4 weeks) looks just like that. 700 watt would need to be twice that size. Except that 2 140 fans and this has 2 120 fans,
H875d46b863e349fa8f08fa209eff8856S.jpg_960x960.jpg
 

H T C

Senior member
Nov 7, 2018
549
395
136
700 watt would need to be twice that size. Except that 2 140 fans and this has 2 120 fans

8 heat pipes, instead of 6 and 2 150 fans: here are the full specs of this cooler.

Tried searching Noctua's site for socket SP5 products and got zero results ... bummer ... but that doesn't mean they don't have such cooler(s) "in development".
 

Skillz

Senior member
Feb 14, 2014
905
918
136
8 heat pipes, instead of 6 and 2 150 fans: here are the full specs of this cooler.

Tried searching Noctua's site for socket SP5 products and got zero results ... bummer ... but that doesn't mean they don't have such cooler(s) "in development".

The fans on that cooler are only 140mm. The width of the heatsink is 150mm.
 

Skillz

Senior member
Feb 14, 2014
905
918
136
But it uses 2 of these: Noctua NF-A15 HS-PWM.

Are you saying these aren't 150 fans?

That's exactly what I am saying. Look the fans up. It's 140mm round (the fan) but the width is enlarged to 150mm which doesn't make the fan (the blades) any larger. It also utilized 120mm fan hole spacing (105mm apart) so it can mount to a 120mm fan mount.
 

H T C

Senior member
Nov 7, 2018
549
395
136
That's exactly what I am saying. Look the fans up. It's 140mm round (the fan) but the width is enlarged to 150mm which doesn't make the fan (the blades) any larger.

This is 140 fans:

nf_a14_3_7.jpg


And this is 150 fans:

nf_a15_pwm_chromax_black_swap_3_1.jpg


The mounting holes are the same but the size of the blades is not.
 

Skillz

Senior member
Feb 14, 2014
905
918
136
This is 140 fans:

nf_a14_3_7.jpg


And this is 150 fans:

nf_a15_pwm_chromax_black_swap_3_1.jpg


The mounting holes are the same but the size of the blades is not.

All you have to do is look it up.

The "150" fan is 140mm x 150mm. Please tell me how a circle (a fan) can be 150mm wide and 140mm tall? Its a circle. It's the same height as it is wide.

Even Noctua refers to it as 140mm.

2023-04-14 21_21_27-NF-A15 PWM.jpg

Or read the Amazon page: https://www.amazon.com/Noctua-NF-A15-PWM-Premium-Cooling/dp/B00AED7XFI

2023-04-14 21_22_58-Amazon.com_ Noctua NF-A15 PWM, Premium Quiet Fan, 4-Pin (140mm, Brown) _ E...jpg



The fan itself is 140mm. The fan housing is 140mm tall and 150mm wide. The fan, is 140mm.
 
  • Like
Reactions: H T C

vityan666

Member
Apr 12, 2023
42
31
51
Markfw
So... I personally think its a vendor locked CPU. So I am returning it, and I have a NEW RETAIL 9654 coming Sunday. So in 2 days we will know if I am right, and the memory is fine.
I hope the new one will work. My CPU will probably arrive only next month. :confused:

Unless thats not real, my new 400 watt cooler (due in 2-4 weeks) looks just like that. 700 watt would need to be twice that size. Except that 2 140 fans and this has 2 120 fans,
I've also ordered Noctua NF-F12 iPPC-2000 PWM and Noctua NF-P12 redux-1700 PWM, in case I'll decide to replace their stock fans.
Still, if the description is fully correct and stock fans peak at 2200RPM with >80 CFM, then it has static pressure even higher than NF-F12 iPPC-2000, which I doubt until I see it myself. Still they seem to be >40DBA at 100% load.

H T C
Tried searching Noctua's site for socket SP5 products and got zero results ... bummer ... but that doesn't mean they don't have such cooler(s) "in development".
Unfortunately looks like not. I've asked em' on 24.03 about some love for our small and cute SP5 socket, and got this response on 28.03:

My mail:
Subject: Regular tower Cooler for socket SP5
Hi,
Do you plan to manufacture/introduce a socket SP5 (LGA 6096) compatible cooler for AMD Epyc Genoa CPUs?
I've just ordered an outstanding Epyc 9274F CPU to replace my good Ryzen 3900X based build, and Micron DDR5-4800 2R RDIMMs for the board. My build currently using Corsair supertower and Corsair AIO. The only relatively large cooler I managed to find yet is chineese SP5 cooler with two fans, and no known manufacturer has one yet, while most of companies already manufacture cooler for Sapphire Rapids.
So, will SP5 socket get some love as well, before TR-7000 release?

Response:
Greetings from Noctua!



Thank you very much for your enquiry!

Unfortunately, no such mounting kit is currently planned.


Personally I would love if Corsair/Cooler Master,etc will create AiO for this socket with wb fully covering the socket.