Question Genoa builders thread.

[vityan666]

Markfw​

I got supermicro confirmed compatible memory. The first server is still working with the micron, so both are just fine !

I've got the Smicro compatible Micron 32GB 2R RDIMM modules. The first 6 are retail by Micron, the new 3 from Newegg are same model rebranded by SuperMicro(Still waiting for them).
Micron part#: MTC20F2085S1RC48BA1R
I've selected it out of 3 compatible options(By Micron, Samsung and Hynix). Hynix was 1R and uses older manufacturing process, Micron had much more availability via eBay back then, so it was an easy choice
It wasn't available via Newegg then I've ordered but its there now.

In my case it's was just really a bad module, not a compat issue. Happens.

 

[vityan666]

Yesterday the Smicro rebranded RDIMM modules arrived. Their color is different(swamp green) from the one of the retail Micron modules(normal green). The model name in BMC is the same, slightly newer lot by serial numbers.
Passed 4 memtest passes(w/o Hammer) and 50% of single Hammer run(I didn't had time to wait more - Needed to put the system back online).
Currently counting the uptime

Meantime my MCIO storage related journey continues.
While someone in reddit wrote that MCIO x8(74p, SFF-TA-1016) can, possibly be compatible with SlimSAS 8i(SFF-8654) physically(but surely not electrically) I put the system offline this midnight to check this 'assumption',
and they are definitely NOT, as MCIO also has two 'holding/pressing/supporting' plastic legs on its sides, while SlimSAS doesn't have any additional side locking/support available, so it just doesn't hold inside anyway and pulls out.

Finding cables for this new standard is almost impossible. I asked microsatacables about MCIO x8 to U.3(As they have MCIO x4 to U.2, which I got beforehand) and they currently not having such cables. They, will possibly look at creating one in the future,
but I'm not building on it(Through, there are diagrams of such cable available on the internet to create one).
I've found additional quite hacky way to connect my U.3 SSD to MCIO port - I've ordered two 'ingredients' from microsatacables for this:
(1) MCIO 8x to 2xOcuLink 4i cable
(2) Small board with OcuLink 4i female port, SATA power connector and U.2 drive connector
Chaining them will allow to connect the drive, hoping the small board will not short onto the case(SSDs are located in side-back in cable-management area, so high chance of touching the case inner metal or closing doors).

I've also ordered new Areca Tri-Mode RAID controller(ARC-1886-4X12I) to, possibly, replace my aging good ARC-1883ix-12 controller. I'll put it aside as backup hardware for rare component or will upgrade to it,
and connect all my drives(SATA,SAS,U.3) using my SlimSAS cables into RAID/JBODS and call it a day, w/o messing with MCIO anymore...

This build's cost by now heavily exceeded my initial cost estimations... At least out of all useless crap I bougth for it I'll get DataVac ESD as useful one

 

[Markfw]

OK< This is more like it. Do you think the HSF on the left will cool better at 400 watts and be quieter ?? I bet it will.

 

[vityan666]

Markfw​

Do you think the HSF on the left will cool better at 400 watts and be quieter ??

I think it will be quieter at its 100% fan speed, than the second one at 50%, through it's relatively noisy at its pick 2200RPM.
Otherwise its dead silent.
I would prefer the radiator base to be larger and higher but they targeted the whole system to be almost not more than a socket size,
preventing any clearance issues with the DIMM slots in advance. This however forces the airflow direction through the long side of the socket,
w/o 90 degree rotation possibilities.

Meantime I've added side mounted 120mm fans to blow air on the RAID card and got 10 degree reduction on the ASIC and controller. Enclosure temps left the same,
which is far less effective from my classic direct from bottom fans mount(20-25 degrees reduction, incl the enclosure).
This required to connect them to FAN-A or FAN-B and activate Heavy-IO profile(Otherwise Smicro board ran them @550RPM which is too low and didn't make any effect) which put them at ~1200RPM and added slight noise from those.

I've added two additional 120mm fans behind the board to blow air on RAM sticks - This experiment failed, as RAM temps remained at 42-45 degree mark.
This probably has to do with CPU cooler fans blowing the hot air on the top modules, making the central area relatively hot. This is further complicated by current lack of exhaust fans, creating positive pressure.

 

[TennesseeTony]

I bet it is 4x quieter, minimum, but as for handling the heat, probably 5C to 8C worse.

 

[Markfw]

I bet it is 4x quieter, minimum, but as for handling the heat, probably 5C to 8C worse.

The 9654 has been running 68-74C with the fan at 3500. I will report back after the change.

 

[A///]

Mark what is your peak sustained power use like? I've been going over my overhaul plans and beginning to consider a Genoa in a 4u setup. it seems like the better and smarter choice. How many kWh a day if you've got the figure?

 

[Markfw]

Mark what is your peak sustained power use like? I've been going over my overhaul plans and beginning to consider a Genoa in a 4u setup. it seems like the better and smarter choice. How many kWh a day if you've got the figure?

I don't have a kill-a-watt on it, but its running 95% load 24/7 and the BMI says its a 400 watt TDP. NO kill-a-watt with 9 computers and 10 video cards, no idea on KWH

 

[A///]

I don't have a kill-a-watt on it, but its running 95% load 24/7 and the BMI says its a 400 watt TDP. NO kill-a-watt with 9 computers and 10 video cards, no idea on KWH

And that load is directly on all 96c of the 9654? I hate to give credit to Igor's inane suggestions but the more I looked into the more it seems a better idea than running 3-4 individual 7950s all day long. I'll have to ask around people doing the same work but my only concern ever was the lower operating speed over the 7950xs.

 

[Markfw]

And that load is directly on all 96c of the 9654? I hate to give credit to Igor's inane suggestions but the more I looked into the more it seems a better idea than running 3-4 individual 7950s all day long. I'll have to ask around people doing the same work but my only concern ever was the lower operating speed over the 7950xs.

So, my 7950x's (5 of them) are set to 142 watt give or take. They run at 4.7 ghz all core 100% loaded. The 9654 runs 2.8 ghz 100 % loaded. and MIGHT take 400 watts. So 6 of the 7950x's would take 900 watt total for 4.7 vs the 9654@400 watt@ 2.8. If you do the math, the 9654 beats them in power/watt 191 to 142 (not sure which way) but the 9654 is more efficient. Remember, the cores are identical. There is the SOC and other overhead on the 7950x's. If you are looking for efficiency, Genoa wins. If you run the 7950x's at stock, they lose by a LOT.

Edit: let me clarify. The 9654 is the same number of cores as 6 7950x's and at exactly the same performance per GHZ. So unlike other scenarios, its easy to calculate which has better performance/watt.

Edit 2: the 2 channel vs the 12 channel should make the memory performance the same, but the 9654 runs at 4800 and the 7950x's can run easily at 6000. Not sure how much that affects the numbers.

 

[A///]

So, my 7950x's (5 of them) are set to 142 watt give or take. They run at 4.7 ghz all core 100% loaded. The 9654 runs 2.8 ghz 100 % loaded. and MIGHT take 400 watts. So 6 of the 7950x's would take 900 watt total for 4.7 vs the 9654@400 watt@ 2.8. If you do the math, the 9654 beats them in power/watt 191 to 142 (not sure which way) but the 9654 is more efficient. Remember, the cores are identical. There is the SOC and other overhead on the 7950x's. If you are looking for efficiency, Genoa wins. If you run the 7950x's at stock, they lose by a LOT.

Edit: let me clarify. The 9654 is the same number of cores as 6 7950x's and at exactly the same performance per GHZ. So unlike other scenarios, its easy to calculate which has better performance/watt.

Edit 2: the 2 channel vs the 12 channel should make the memory performance the same, but the 9654 runs at 4800 and the 7950x's can run easily at 6000. Not sure how much that affects the numbers.

The only concern I have in terms of clocks were whether there's a large difference in completion time of what I'll be putting them through. Speed and end result completion time isn't linear for these tasks which is why I've been given serious consolidation to you and Igor's earlier suggestion. I know a few people who do similar work but they're out of their respective countries at the moe and I haven't been able to get in touch with them. There's more important things while being on an african safari. the benefit to the 4u setup would be nvidia pro cards to crunch the large data as well as accelerators, you're limited on consumer platforms. My june time was pushed back due to unforeseen circumstances due to some incidents with the property and some rentals I own, product dispute with toro. my luck.

 

[Markfw]

The only concern I have in terms of clocks were whether there's a large difference in completion time of what I'll be putting them through. Speed and end result completion time isn't linear for these tasks which is why I've been given serious consolidation to you and Igor's earlier suggestion. I know a few people who do similar work but they're out of their respective countries at the moe and I haven't been able to get in touch with them. There's more important things while being on an african safari. the benefit to the 4u setup would be nvidia pro cards to crunch the large data as well as accelerators, you're limited on consumer platforms. My june time was pushed back due to unforeseen circumstances due to some incidents with the property and some rentals I own, product dispute with toro. my luck.

Once you include PCIE devices/video cards, the Genoa becomes the only choice. It has way more channels. So the question is. do you need faster CPU completion time ? Or more video cars on one box for more efficiency ? In the DC thread, EPYC is king where you need a lot of efficiency in a lot of GPU devices.

If you need more clarification than this, join here, as there are a LOT of experts in this area. Select "boinc-help" forums

https://discord.gg/E5z65jR5Ka

 

[A///]

Once you include PCIE devices/video cards, the Genoa becomes the only choice. It has way more channels. So the question is. do you need faster CPU completion time ? Or more video cars on one box for more efficiency ? In the DC thread, EPYC is king where you need a lot of efficiency in a lot of GPU devices.

If you need more clarification than this, join here, as there are a LOT of experts in this area. Select "boinc-help" forums

https://discord.gg/E5z65jR5Ka

thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?

 

[Markfw]

thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?

Its like "instant messaging", I don't recommend it to everyone, but for the DC crowd (and maybe you ) it makes sense.

 

[A///]

Its like "instant messaging", I don't recommend it to everyone, but for the DC crowd (and maybe you ) it makes sense.

I'll have to read up on it. Too many sketchy web sites out there these days.

 

[Markfw]

I'll have to read up on it. Too many sketchy web sites out there these days.

Its not a website, its an app, and one for the Anandtech DC community linked here: https://forums.anandtech.com/threads/team-anandtech-boinc-and-dc-discord.2610072/

 

[Markfw]

thanks. I definitely have to go over my current plans and weigh which makes sense financially and time. what is discord gg?

BTW, I ran across some power utilization stats. Now I am eager to get a kill-a-watt on my Genoa boxes. From Phoronix:
"Lastly is a look at the CPU power consumption across all of the benchmarks ran for this article. The Xeon Platinum 8490H in both single and dual socket configurations consumed significantly more power than the prior-generation Ice Lake CPUs and even the EPYC Genoa SKUs. Across all of the benchmarking a 303 Watt average was observed and a peak power consumption of 379 Watts, as exposed via the RAPL sysfs interfaces. Meanwhile the Xeon Platinum 8380 had a 223 Watt average and a 293 Watt peak across the set of benchmarks while the EPYC 9654 also had a 223 Watt average while at a 363 Watt peak."

So 303 watt average for SR's best, and 223 watt for 96 cores over the 60 cores of SR. SR is a power hog. In this review, both of the genoa chips won on average. a couple of apps used built in accelerators, so SR did better on those.

 

[A///]

BTW, I ran across some power utilization stats. Now I am eager to get a kill-a-watt on my Genoa boxes. From Phoronix:
"Lastly is a look at the CPU power consumption across all of the benchmarks ran for this article. The Xeon Platinum 8490H in both single and dual socket configurations consumed significantly more power than the prior-generation Ice Lake CPUs and even the EPYC Genoa SKUs. Across all of the benchmarking a 303 Watt average was observed and a peak power consumption of 379 Watts, as exposed via the RAPL sysfs interfaces. Meanwhile the Xeon Platinum 8380 had a 223 Watt average and a 293 Watt peak across the set of benchmarks while the EPYC 9654 also had a 223 Watt average while at a 363 Watt peak."

So 303 watt average for SR's best, and 223 watt for 96 cores over the 60 cores of SR. SR is a power hog. In this review, both of the genoa chips won on average. a couple of apps used built in accelerators, so SR did better on those.

Not really shocking. SR is a product that needed to come out years ago but due to mistakes it's competing with modern hardware that entered the design phase later than it. SR only makes sense if you need the few distinct advantages Intel may offer over Epyc, I need to do a deep dive to see what those are exactly and I suspect it pertains only to virtual machines. On the lower end of the scale the 3954, or whatever hot potato casserole Intel released only had 1 distinction against TR being the memory channels and board io. still runs hotter and less performance only equal to older tr or epycs. It hadn't ocurred to me that I could look at the older tr performance to get an idea of how the 96 core may perform but that gives you a very brief look because it doesn't take architecture into account, only speed. With the price revelation of the 96 core I can squeeze in an extra 1700 for the xilinx card.

 

[Markfw]

Not really shocking. SR is a product that needed to come out years ago but due to mistakes it's competing with modern hardware that entered the design phase later than it. SR only makes sense if you need the few distinct advantages Intel may offer over Epyc, I need to do a deep dive to see what those are exactly and I suspect it pertains only to virtual machines. On the lower end of the scale the 3954, or whatever hot potato casserole Intel released only had 1 distinction against TR being the memory channels and board io. still runs hotter and less performance only equal to older tr or epycs. It hadn't ocurred to me that I could look at the older tr performance to get an idea of how the 96 core may perform but that gives you a very brief look because it doesn't take architecture into account, only speed. With the price revelation of the 96 core I can squeeze in an extra 1700 for the xilinx card.

Did you mean 9354 ? 32 core.

AMD EPYC GENOA SP5 ZEN4 9354 32-Core 3.25GHz Processor CPU QS ES | eBay

Find many great new & used options and get the best deals for AMD EPYC GENOA SP5 ZEN4 9354 32-Core 3.25GHz Processor CPU QS ES at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I bet you can get it for less than that. I got a 9554 64 core for less than $2,000. And I turned down offers for $900 for 9354 QS's.

 

[A///]

Did you mean 9354 ? 32 core.

AMD EPYC GENOA SP5 ZEN4 9354 32-Core 3.25GHz Processor CPU QS ES | eBay

Find many great new & used options and get the best deals for AMD EPYC GENOA SP5 ZEN4 9354 32-Core 3.25GHz Processor CPU QS ES at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I bet you can get it for less than that. I got a 9554 64 core for less than $2,000. And I turned down offers for $900 for 9354 QS's.

No the intel ws processor, 3495x I think it was? it's unrelated to this plan I'm cooking up but if you had to pick it or the tr and needed the board io and didn't want to use an odm device it wins otherwise slow and hot.

 

[Markfw]

The 9654 has been running 68-74C with the fan at 3500. I will report back after the change

OK, this is just the first temp reading, but its 69, same as the lowest before. But 2 fans at 2100 and I can not hear anything. Remember, I am sort of deaf, but can hear fans pretty well.

Edit: BUT CPU speed is 2821, I think faster than it was before at 2700.

 

[Markfw]

OK, I have the 9554 up. It does not like booting win 10 and auto-updating. But fully loaded ( a 2080TI @100%) and all the cores, runs 615 watts, the same as the 2970 24 cores and the same video card, so It way more efficient, only running about 300 watts for the CPU with all cores@100%. And runs 3.5 ghz with all 128 threads running.

 

[vityan666]

Update from my workstation side.

Yesterday I did a mistake which took me hours to recover from.
Supermicro released new BMC (...0.18) and BIOS(1.4) for H13SSL-NT.
While BMC is still same buggy *, the BIOS update ended in a complete disaster, forcing me to disassemble fans and cards(At least not the CPU/Fan and RAM).

This new BIOS is incompatible with my PCI-e USB card and with my GPU, resulting in POST error 91, no metter that. The extra USB card of another brand, worked, but still.
The only PCI-e card compatible with new BIOS is my Areca 1883 RAID controller - It didn't had any issues with any of the BIOSes.
They put the new AGESA microcode in there(1.0.0.6), up from 1.0.0.3 in previous 1.1a.

And ofc, I didn't make a BIOS backup... As Smicro doesn't provide links to previous versions(Whhhy????), I've booted into my system with BMC VGA w/o USB cards and external GPU, then copied the stored 1.1a BIOS into USB to flash from another PC via BMC.

New BIOS doesn't have the per Slot UEFI/Legacy selections in PCI-e configuration - This is one of the visual changes I saw in it.
I tried a lot to force the new BIOS to work with the cards to no avail. So, finally I gave up and downgraded it(And needed to set primary VGA to Onboard, as 'External' option blocked GPU from booting with same code 91).

The * BMC still shows BIOS version as 1.4... They need some 'Rescan Hardware' button ...

Thus, if you have PCI-e cards on those board it's better to skip this update, and if you decided to take a chance backup your BIOS beforehand.
If no PCI-e slots are occupied then no problem should arise(I booted it with RAID card only installed).

 

[Markfw]

The new Heatsinks.... CRAP. They do OK temps wise, until you have to use a piece of cardboard between the upper fan and the HSF to keep it from making a horrific noise, and slowing down the fan and upping the CPU temp by 10c. One is OK, the other I had to put the cardboard in. Yes, they were both installed correctly. I have like 12 of the Noctua SP3 ones, and never had this issue. Its about the HSF fin design and the fan.

 

[vityan666]

Markfw​

Is it 'scratching metal' noise of radiator or vibration noise of Fans over the radiator due to loose springless holders connections?