• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Question Genoa builders thread.

Page 4 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Markfw​

I got supermicro confirmed compatible memory. The first server is still working with the micron, so both are just fine !
Click to expand...

I've got the Smicro compatible Micron 32GB 2R RDIMM modules. The first 6 are retail by Micron, the new 3 from Newegg are same model rebranded by SuperMicro(Still waiting for them).
Micron part#: MTC20F2085S1RC48BA1R
I've selected it out of 3 compatible options(By Micron, Samsung and Hynix). Hynix was 1R and uses older manufacturing process, Micron had much more availability via eBay back then, so it was an easy choice 🙂
It wasn't available via Newegg then I've ordered but its there now.

In my case it's was just really a bad module, not a compat issue. Happens. 🙂
 
Yesterday the Smicro rebranded RDIMM modules arrived. Their color is different(swamp green) from the one of the retail Micron modules(normal green). The model name in BMC is the same, slightly newer lot by serial numbers.
Passed 4 memtest passes(w/o Hammer) and 50% of single Hammer run(I didn't had time to wait more - Needed to put the system back online).
Currently counting the uptime 🙂

Meantime my MCIO storage related journey continues.
While someone in reddit wrote that MCIO x8(74p, SFF-TA-1016) can, possibly be compatible with SlimSAS 8i(SFF-8654) physically(but surely not electrically) I put the system offline this midnight to check this 'assumption',
and they are definitely NOT, as MCIO also has two 'holding/pressing/supporting' plastic legs on its sides, while SlimSAS doesn't have any additional side locking/support available, so it just doesn't hold inside anyway and pulls out.

Finding cables for this new standard is almost impossible. I asked microsatacables about MCIO x8 to U.3(As they have MCIO x4 to U.2, which I got beforehand) and they currently not having such cables. They, will possibly look at creating one in the future,
but I'm not building on it(Through, there are diagrams of such cable available on the internet to create one).
I've found additional quite hacky way to connect my U.3 SSD to MCIO port - I've ordered two 'ingredients' from microsatacables for this:
(1) MCIO 8x to 2xOcuLink 4i cable
(2) Small board with OcuLink 4i female port, SATA power connector and U.2 drive connector
Chaining them will allow to connect the drive, hoping the small board will not short onto the case(SSDs are located in side-back in cable-management area, so high chance of touching the case inner metal or closing doors).

I've also ordered new Areca Tri-Mode RAID controller(ARC-1886-4X12I) to, possibly, replace my aging good ARC-1883ix-12 controller. I'll put it aside as backup hardware for rare component or will upgrade to it,
and connect all my drives(SATA,SAS,U.3) using my SlimSAS cables into RAID/JBODS and call it a day, w/o messing with MCIO anymore...

This build's cost by now heavily exceeded my initial cost estimations... At least out of all useless crap I bougth for it I'll get DataVac ESD as useful one 🙂
 

Markfw​

Do you think the HSF on the left will cool better at 400 watts and be quieter ??
Click to expand...
I think it will be quieter at its 100% fan speed, than the second one at 50%, through it's relatively noisy at its pick 2200RPM.
Otherwise its dead silent.
I would prefer the radiator base to be larger and higher but they targeted the whole system to be almost not more than a socket size,
preventing any clearance issues with the DIMM slots in advance. This however forces the airflow direction through the long side of the socket,
w/o 90 degree rotation possibilities.

Meantime I've added side mounted 120mm fans to blow air on the RAID card and got 10 degree reduction on the ASIC and controller. Enclosure temps left the same,
which is far less effective from my classic direct from bottom fans mount(20-25 degrees reduction, incl the enclosure).
This required to connect them to FAN-A or FAN-B and activate Heavy-IO profile(Otherwise Smicro board ran them @550RPM which is too low and didn't make any effect) which put them at ~1200RPM and added slight noise from those.

I've added two additional 120mm fans behind the board to blow air on RAM sticks - This experiment failed, as RAM temps remained at 42-45 degree mark.
This probably has to do with CPU cooler fans blowing the hot air on the top modules, making the central area relatively hot. This is further complicated by current lack of exhaust fans, creating positive pressure.
 
Last edited:
Mark what is your peak sustained power use like? I've been going over my overhaul plans and beginning to consider a Genoa in a 4u setup. it seems like the better and smarter choice. How many kWh a day if you've got the figure?
 
A/// said:
Mark what is your peak sustained power use like? I've been going over my overhaul plans and beginning to consider a Genoa in a 4u setup. it seems like the better and smarter choice. How many kWh a day if you've got the figure?
Click to expand...
I don't have a kill-a-watt on it, but its running 95% load 24/7 and the BMI says its a 400 watt TDP. NO kill-a-watt with 9 computers and 10 video cards, no idea on KWH
 
Markfw said:
I don't have a kill-a-watt on it, but its running 95% load 24/7 and the BMI says its a 400 watt TDP. NO kill-a-watt with 9 computers and 10 video cards, no idea on KWH
Click to expand...
And that load is directly on all 96c of the 9654? I hate to give credit to Igor's inane suggestions but the more I looked into the more it seems a better idea than running 3-4 individual 7950s all day long. I'll have to ask around people doing the same work but my only concern ever was the lower operating speed over the 7950xs.
 
A/// said:
And that load is directly on all 96c of the 9654? I hate to give credit to Igor's inane suggestions but the more I looked into the more it seems a better idea than running 3-4 individual 7950s all day long. I'll have to ask around people doing the same work but my only concern ever was the lower operating speed over the 7950xs.
Click to expand...
So, my 7950x's (5 of them) are set to 142 watt give or take. They run at 4.7 ghz all core 100% loaded. The 9654 runs 2.8 ghz 100 % loaded. and MIGHT take 400 watts. So 6 of the 7950x's would take 900 watt total for 4.7 vs the 9654@400 watt@ 2.8. If you do the math, the 9654 beats them in power/watt 191 to 142 (not sure which way) but the 9654 is more efficient. Remember, the cores are identical. There is the SOC and other overhead on the 7950x's. If you are looking for efficiency, Genoa wins. If you run the 7950x's at stock, they lose by a LOT.

Edit: let me clarify. The 9654 is the same number of cores as 6 7950x's and at exactly the same performance per GHZ. So unlike other scenarios, its easy to calculate which has better performance/watt.

Edit 2: the 2 channel vs the 12 channel should make the memory performance the same, but the 9654 runs at 4800 and the 7950x's can run easily at 6000. Not sure how much that affects the numbers.
 
Markfw said:
So, my 7950x's (5 of them) are set to 142 watt give or take. They run at 4.7 ghz all core 100% loaded. The 9654 runs 2.8 ghz 100 % loaded. and MIGHT take 400 watts. So 6 of the 7950x's would take 900 watt total for 4.7 vs the 9654@400 watt@ 2.8. If you do the math, the 9654 beats them in power/watt 191 to 142 (not sure which way) but the 9654 is more efficient. Remember, the cores are identical. There is the SOC and other overhead on the 7950x's. If you are looking for efficiency, Genoa wins. If you run the 7950x's at stock, they lose by a LOT.

Edit: let me clarify. The 9654 is the same number of cores as 6 7950x's and at exactly the same performance per GHZ. So unlike other scenarios, its easy to calculate which has better performance/watt.

Edit 2: the 2 channel vs the 12 channel should make the memory performance the same, but the 9654 runs at 4800 and the 7950x's can run easily at 6000. Not sure how much that affects the numbers.
Click to expand...
The only concern I have in terms of clocks were whether there's a large difference in completion time of what I'll be putting them through. Speed and end result completion time isn't linear for these tasks which is why I've been given serious consolidation to you and Igor's earlier suggestion. I know a few people who do similar work but they're out of their respective countries at the moe and I haven't been able to get in touch with them. There's more important things while being on an african safari. the benefit to the 4u setup would be nvidia pro cards to crunch the large data as well as accelerators, you're limited on consumer platforms. My june time was pushed back due to unforeseen circumstances due to some incidents with the property and some rentals I own, product dispute with toro. my luck.
 
A/// said:
The only concern I have in terms of clocks were whether there's a large difference in completion time of what I'll be putting them through. Speed and end result completion time isn't linear for these tasks which is why I've been given serious consolidation to you and Igor's earlier suggestion. I know a few people who do similar work but they're out of their respective countries at the moe and I haven't been able to get in touch with them. There's more important things while being on an african safari. the benefit to the 4u setup would be nvidia pro cards to crunch the large data as well as accelerators, you're limited on consumer platforms. My june time was pushed back due to unforeseen circumstances due to some incidents with the property and some rentals I own, product dispute with toro. my luck.
Click to expand...
Once you include PCIE devices/video cards, the Genoa becomes the only choice. It has way more channels. So the question is. do you need faster CPU completion time ? Or more video cars on one box for more efficiency ? In the DC thread, EPYC is king where you need a lot of efficiency in a lot of GPU devices.

If you need more clarification than this, join here, as there are a LOT of experts in this area. Select "boinc-help" forums

https://discord.gg/E5z65jR5Ka
 
Markfw said:
Once you include PCIE devices/video cards, the Genoa becomes the only choice. It has way more channels. So the question is. do you need faster CPU completion time ? Or more video cars on one box for more efficiency ? In the DC thread, EPYC is king where you need a lot of efficiency in a lot of GPU devices.

If you need more clarification than this, join here, as there are a LOT of experts in this area. Select "boinc-help" forums

https://discord.gg/E5z65jR5Ka
Click to expand...
thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?
 
A/// said:
thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?
Click to expand...
Its like "instant messaging", I don't recommend it to everyone, but for the DC crowd (and maybe you ) it makes sense.
 
A/// said:
thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?
Click to expand...
BTW, I ran across some power utilization stats. Now I am eager to get a kill-a-watt on my Genoa boxes. From Phoronix:
"Lastly is a look at the CPU power consumption across all of the benchmarks ran for this article. The Xeon Platinum 8490H in both single and dual socket configurations consumed significantly more power than the prior-generation Ice Lake CPUs and even the EPYC Genoa SKUs. Across all of the benchmarking a 303 Watt average was observed and a peak power consumption of 379 Watts, as exposed via the RAPL sysfs interfaces. Meanwhile the Xeon Platinum 8380 had a 223 Watt average and a 293 Watt peak across the set of benchmarks while the EPYC 9654 also had a 223 Watt average while at a 363 Watt peak."

So 303 watt average for SR's best, and 223 watt for 96 cores over the 60 cores of SR. SR is a power hog. In this review, both of the genoa chips won on average. a couple of apps used built in accelerators, so SR did better on those.
 
Markfw said:
BTW, I ran across some power utilization stats. Now I am eager to get a kill-a-watt on my Genoa boxes. From Phoronix:
"Lastly is a look at the CPU power consumption across all of the benchmarks ran for this article. The Xeon Platinum 8490H in both single and dual socket configurations consumed significantly more power than the prior-generation Ice Lake CPUs and even the EPYC Genoa SKUs. Across all of the benchmarking a 303 Watt average was observed and a peak power consumption of 379 Watts, as exposed via the RAPL sysfs interfaces. Meanwhile the Xeon Platinum 8380 had a 223 Watt average and a 293 Watt peak across the set of benchmarks while the EPYC 9654 also had a 223 Watt average while at a 363 Watt peak."

So 303 watt average for SR's best, and 223 watt for 96 cores over the 60 cores of SR. SR is a power hog. In this review, both of the genoa chips won on average. a couple of apps used built in accelerators, so SR did better on those.
Click to expand...
Not really shocking. SR is a product that needed to come out years ago but due to mistakes it's competing with modern hardware that entered the design phase later than it. SR only makes sense if you need the few distinct advantages Intel may offer over Epyc, I need to do a deep dive to see what those are exactly and I suspect it pertains only to virtual machines. On the lower end of the scale the 3954, or whatever hot potato casserole Intel released only had 1 distinction against TR being the memory channels and board io. still runs hotter and less performance only equal to older tr or epycs. It hadn't ocurred to me that I could look at the older tr performance to get an idea of how the 96 core may perform but that gives you a very brief look because it doesn't take architecture into account, only speed. With the price revelation of the 96 core I can squeeze in an extra 1700 for the xilinx card.
 
A/// said:
Not really shocking. SR is a product that needed to come out years ago but due to mistakes it's competing with modern hardware that entered the design phase later than it. SR only makes sense if you need the few distinct advantages Intel may offer over Epyc, I need to do a deep dive to see what those are exactly and I suspect it pertains only to virtual machines. On the lower end of the scale the 3954, or whatever hot potato casserole Intel released only had 1 distinction against TR being the memory channels and board io. still runs hotter and less performance only equal to older tr or epycs. It hadn't ocurred to me that I could look at the older tr performance to get an idea of how the 96 core may perform but that gives you a very brief look because it doesn't take architecture into account, only speed. With the price revelation of the 96 core I can squeeze in an extra 1700 for the xilinx card.
Click to expand...
Did you mean 9354 ? 32 core.

www.ebay.com

AMD EPYC GENOA SP5 ZEN4 9354 32-Core Processor CPU QS ES | eBay

Model: EPYC 9354 (GENOA Dual-socket). Sockets Supported SP5. 128L PCIe Gen5. 32-Core / 64-Threads.
www.ebay.com

I bet you can get it for less than that. I got a 9554 64 core for less than $2,000. And I turned down offers for $900 for 9354 QS's.
 
Markfw said:
Did you mean 9354 ? 32 core.

www.ebay.com

AMD EPYC GENOA SP5 ZEN4 9354 32-Core Processor CPU QS ES | eBay

Model: EPYC 9354 (GENOA Dual-socket). Sockets Supported SP5. 128L PCIe Gen5. 32-Core / 64-Threads.
www.ebay.com

I bet you can get it for less than that. I got a 9554 64 core for less than $2,000. And I turned down offers for $900 for 9354 QS's.
Click to expand...
No the intel ws processor, 3495x I think it was? it's unrelated to this plan I'm cooking up but if you had to pick it or the tr and needed the board io and didn't want to use an odm device it wins otherwise slow and hot.
 
Markfw said:
The 9654 has been running 68-74C with the fan at 3500. I will report back after the change
Click to expand...
OK, this is just the first temp reading, but its 69, same as the lowest before. But 2 fans at 2100 and I can not hear anything. Remember, I am sort of deaf, but can hear fans pretty well.

Edit: BUT CPU speed is 2821, I think faster than it was before at 2700.
 

Attachments

  • 20230602_204629[1].jpg
    20230602_204629[1].jpg
    470.6 KB · Views: 33
OK, I have the 9554 up. It does not like booting win 10 and auto-updating. But fully loaded ( a 2080TI @100%) and all the cores, runs 615 watts, the same as the 2970 24 cores and the same video card, so It way more efficient, only running about 300 watts for the CPU with all cores@100%. And runs 3.5 ghz with all 128 threads running.
 
Last edited:
Update from my workstation side.

Yesterday I did a mistake which took me hours to recover from.
Supermicro released new BMC (...0.18) and BIOS(1.4) for H13SSL-NT.
While BMC is still same buggy *, the BIOS update ended in a complete disaster, forcing me to disassemble fans and cards(At least not the CPU/Fan and RAM).

This new BIOS is incompatible with my PCI-e USB card and with my GPU, resulting in POST error 91, no metter that. The extra USB card of another brand, worked, but still.
The only PCI-e card compatible with new BIOS is my Areca 1883 RAID controller - It didn't had any issues with any of the BIOSes.
They put the new AGESA microcode in there(1.0.0.6), up from 1.0.0.3 in previous 1.1a.

And ofc, I didn't make a BIOS backup... As Smicro doesn't provide links to previous versions(Whhhy????), I've booted into my system with BMC VGA w/o USB cards and external GPU, then copied the stored 1.1a BIOS into USB to flash from another PC via BMC.

New BIOS doesn't have the per Slot UEFI/Legacy selections in PCI-e configuration - This is one of the visual changes I saw in it.
I tried a lot to force the new BIOS to work with the cards to no avail. So, finally I gave up and downgraded it(And needed to set primary VGA to Onboard, as 'External' option blocked GPU from booting with same code 91).

The * BMC still shows BIOS version as 1.4... They need some 'Rescan Hardware' button ...

Thus, if you have PCI-e cards on those board it's better to skip this update, and if you decided to take a chance backup your BIOS beforehand.
If no PCI-e slots are occupied then no problem should arise(I booted it with RAID card only installed).
 
Last edited:
The new Heatsinks.... CRAP. They do OK temps wise, until you have to use a piece of cardboard between the upper fan and the HSF to keep it from making a horrific noise, and slowing down the fan and upping the CPU temp by 10c. One is OK, the other I had to put the cardboard in. Yes, they were both installed correctly. I have like 12 of the Noctua SP3 ones, and never had this issue. Its about the HSF fin design and the fan.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top