Hi Markfw,
I've just registered after seeing this thread a few days ago, as it touches relatively rare thing of making Epyc workstation in a large 4U rack, as it's mostly end-up being small rack builds.
My case is even more complicated as I'll put it in regular PC case(While it's huge Corsair 1000D SuperTower it's still not server case). So only large 120mm+ fans, and relatively quiet operation.
So, to start on, I'm in the similar boat right now, going through a process to upgrade my R3900X based workstation/server with AiO on a CPU. Now after 10+ years I'll have to go back to air, and even air coolers options are sparse as hell.
I hope new TR's will use SP5 and not SP6 so we will get new air/water AiO coolers from standard manufacturers.
In my case, I choose to go with Epyc 9274F due to it's great base frequency(I care less about boost speeds, and almost only locked my CPU's to single frequency - for old lovely UT99 game).
The mobo is the same Supermicro H13SSL-NT, which took a week to catch for availability on Newegg, but its here.
Also one Micron RDIMM 2R DDR5 4800 memory module arrived(From one seller), 5 more on the way(From another), and today got a deal on eBay for 2 extra used modules for only 120$ for both of them, so instead of originally planned 6 I will go for 8 modules,
thus 8 channels.
I wanted to inform you about the cooler part, as I myself looked hard around the web, and while AliExpress did have only the small variant like you purchased, their parent company AliBaba had the large one which I've ordered in the middle of March and
it arrived 2-3 days after you received your cooler. Fortunately this seller doesn't have min. units requirement and we can order just one, but it takes about 2 weeks for him to ship, so patience is needed. I've attached my two photos with this cooler. The thing is huge,
and not any 4U rack has enough space for it.
Url to the cooler:
Newest Amd Sp5 4u Active Lga 6096 Genoa Epyc 7004 Tdp 400w Cpu Cooler Heat Sink Heatsink Welding Nickel Plating Hy Ball Bearing - Buy Cpu Cooler/cpu Heatsink/cpu Heat Sink/amd Cpu Cooler/amd Heatsink/amd Heat Sink/coolserver Sp5/amd Sp5/active Cpu Cooler Fan,Amd Cpu Cooler/amd Processor/cpu...
www.alibaba.com
Regarding disks, the time is flying fast and industry can change without us even noticing. Just less than 1.5y ago I've bough Seagate Nytro 3.2TB SAS SSD, 3DWPD for ~1500$, and a week ago my good friend told me that he recently bough
Micron 7450 Enterprise SSD 15.36TB 1DWPD U.2 for 1550$ only for a client company locally.
This disk is available on Newegg as well for a nice price, and also can be purchased in lower densities or the 'Max' edition(3DWPD).
I've personally bought its successor Micron 9400 Pro 7.68TB Enterprise SSD U.3 for 1200$ on NewEgg and MCIO->U.2 cable from eBay to connect it to the mobo.
I don't see any reason to waste $ on those consumer m.2 disks then serious stuff became that accessible(9400 costs like Samsung's 980 Pro, while being orders of magnitude better product).
I already had Samsung 970 Evo Plus 2TB starting to show bad sectors within 1 year of use(It was replaced with Seagate Nytro immediately as stability and data consistency is #1 for my server system which shall run 24x7x365). Then I repurposed this drive(M.2) for non important stuff and didn't accessed it for a few months it just 'reinitialized' and became empty. I stopped using it completely since then, nor relying on any M.2 chips.
2 Corsair DDR4 DIMM's failed on me within one year causing a lot of headache(First one was for the 96GB-128GB address range and was hard to catch as it only caused problems with extra VM's I've loaded occasionally).
It was the last strain to my decision for ECC memory and resulted in plans for this build(Threadippers come to late and change socket every generation, while their cost became similar to Epyc's. Genoa's F models broke the 4Ghz barrier on a base which is a great thing by itself).
Consumer products and DDR5 design disallows for fast 2DPC DDR5 + 4 DIMMS operation, because AMD and Intel can't get their shit together and make Quad-Channel memory controllers standard with DDR5. UDIMM-ECC are so rare and then you need to look for a mobo that supports it, Waiting for one faster than JEDEC speed to appear, hoping that BIOS update will not break it, etc, etc. Thus, the only good option left is Server CPU + RDIMM. AMD took case of server RDIMM yearly speed increase(So instead of 2133 forever we had 2666,2933,3200)
and now Intel take this even further with Sapphire Rapids, by introducing overclockable RDIMMS(Took only 30 years...) - Not that our Epycs will enjoy oveclocked RDIMMs in this generation but it's a good start.
The only problem is lack of chipset for USB ports and Sound. USBs is solved with PCI-E cards and those solve the Audio problem with USB sound card.
If only those new boards had 7 PCI-E slots like their predecessors, but I suspect it's because of PCI-E 5 signaling which again shortened the max range from the CPU.
ASrock tries to overcome it with zillion of PCI-E switches in new planned board:
Regarding your
first boot problem,
did you try to boot it with minimal set of hardware, i.e 1 DIMM and w/o GPU, using the BMC?
I know it could be a problem with modern monitor, which will require a rare device like this one:
Also, while you tinkering, I'm interested for my own sake,
Can GPU fit into the bottom(5-th) PCI-E slot without pushing/breaking Front-Panel connectors(Power,Reset,LEDS). USB front panel connector is of no interest as I've ordered 90 degree adapter to reroute it oout of extension card way.
If GPU will not fit, I'll put my Areca 1883-IX-12 RAID controller in there and 3-fans extra cooler below it as it(Fans card) doesn't need PCI-e power.