Discussion EPYC builders thread

[Markfw]

So now that I have 2 boxes, I thought I would start a builders thread, and also pass along what I have learned.

First, They are are cheap, and they are a different animal to build (see below)

OK, so first, I got these 2 7551 ES chips for $300 each, I thought, well that is cheap for a 32 core CPU, and with 8 channel memory they won't be handicapped like my 2990wx's.

Now to the bad... First they were only compatible with ONE motherboard that was available to buy, and with a BIOS that I had to fight with Gigabyte to provide me with ! (not available online)
So I tried several other motherboards and different BIOS's since I could not even find this one online. FINALLY I found the board at $470. I updated the BIOS (odd way, boot into motherboards default stream environment (or something like that) and do a DOS bios update. Well, it worked. Then while trying to install linux mint 19.2, no internet ???? Oh, I did not read the fine print. No ethernet that goes outside the local lan. Only SFP+ (whatever that is), so after some reasearch, I find the adapter for $13 more.

OK fast forward to when I now have an install and updated with ethernet. This 32 core chip only runs turbo all core at 1.6 ghz !!! So I spent almost $500 on the motherboard, and for 128 gig of ECC registered (the cheapest I could find@64 gig on 8 sticks or more) at another $530 and a $300 CPU I have $1330 invested in a 1600 mhz 32 core box... Picture:

And before I got this working, one of the motherboards I tried (dual socket) that would not work with these ES chips, I found 2 7601 used retail chips for $1750 and a $650 motherboard. Now this one needed 16 sticks so that was $1100. So now I am in this one $3500 with no SSD, case or PSU. Well, I get it running, and it only boosts all core to 2.6 ghz. But at least thats a far cry to 1600 mhz.

Here:

Bottom line ? The new 3960x and 3970x are not overpriced. At the same speed they are almost twice as fast. And don't venture into EPYC chips unless you have wads of cash to blow and learn. I bought another motherboard for my 2nd 7551 ES chip, so now I have to build that box too !

But I will soon have 3 server grade ECC memory chip boxes with 128 cores and 256 threads for almost $6000. I can't even get video cards to fit in these, as the memory and heatsinks are in the way ! I have to use the on-board video.

 

[lightmanek]

No, it doesn't. At least not normally. Its consumption depends on load, of course. I currently don't have sensor software which would show it though.

Yes, that 90W figure must be worst case scenario with all PCIe lanes and all memory channels taxed by all cores transferring data in a dual socket system.

 

[Markfw]

Its here ! 128 gig of 2666 ECC memory and a 7452 Rome 32 core, waiting on the motherboard (Thursday ??)

 

[DrMrLordX]

MOAR COARS!!!

 

[Markfw]

MOAR COARS!!!

You bet ! and they are all working to try and cure this covid-19 crap.

 

[DrMrLordX]

Is Oxford benefiting from DC Covid-19 efforts?

 

[Markfw]

Is Oxford benefiting from DC Covid-19 efforts?

Stanford (F@H) and Berkley (BOINC/Rosetta)

 

[DrMrLordX]

Stanford (F@H) and Berkley (BOINC/Rosetta)

Oh, okay. Looks like Oxford might beat them to the punch!

 

[Markfw]

Oh, okay. Looks like Oxford might beat them to the punch!

OK, what is Oxford doing ? Got a link ? All of the DC forum is with the two that I am going.

Well, I found this: https://www.wthr.com/article/oxford...rtedly-shows-success-protecting-monkeys-virus
And I don't care who wins, so long as we kill that freaking virus.

And if its cured, then the same projects go back to cancer research, which is my prime reason for al these CPU's

 

[StefanR5R]

I added figures with cTDP of 180 W (instead of the default 155 W of EPYC 7452) to post #70.

With the gene@home application, increasing the TDP and PPT by 16 % gives...

• 7 % higher clock,
• 8 % higher computational throughput,
• ≈19 % higher host power consumption,
• and hence ≈9 % lower power efficiency.

At the increased thermal output, the Noctua NH-U14Ss still don't break a sweat. T_die is 53 °C, which is 28 K over ambient. Fans are spinning slow, but I don't have RPM readings because I don't have the respective pin connected, as mentioned in post #58. VRM temperature is 68 °C with the 40 mm fan @ 2700 RPM sitting on the modest VRM heatsink.

 

[DrMrLordX]

@Markfw

Jenner Institute's COVID-19 Vaccine Expected to Be Tested in 6,000 People Starting in May | BioSpace

With emergency approval, the Oxford University researchers believe they could have a few million doses of the vaccine available by September if it is proven effective.

www.biospace.com

Oxford is taking the lead, apparently.

 

[Markfw]

I added figures with cTDP of 180 W (instead of the default 155 W of EPYC 7452) to post #70.

With the gene@home application, increasing the TDP and PPT by 16 % gives...

• 7 % higher clock,
• 8 % higher computational throughput,
• ≈19 % higher host power consumption,
• and hence ≈9 % lower power efficiency.

At the increased thermal output, the Noctua NH-U14Ss still don't break a sweat. T_die is 53 °C, which is 28 K over ambient. Fans are spinning slow, but I don't have RPM readings because I don't have the respective pin connected, as mentioned in post #58. VRM temperature is 68 °C with the 40 mm fan @ 2700 RPM sitting on the modest VRM heatsink.

Based on your research, I have the same HSF. You think 200 watt or even 225 watt TDP is OK ? My motherboard is in the state, and supposed to be delivered tomorrow.

 

[StefanR5R]

According to servethehome, 7452's cTDP_high is 180 W (post #50). If somebody could find a public AMD datasheet with cTDP_high, that would be good...

I haven't tried setting a higher value yet. Before doing so, one should find a workload which pushes a very constant power consumption, as a way to check actual power consumption before and after setting an out-of-spec cTDP — to verify whether or not the setting really had the intended effect.
[TN-Grid's gene@home causes too much power fluctuation for this purpose. I saw very constant power consumption from Rosetta@home, but that was during testing with a set of WUs which I fetched within one minute. I don't know if this is still through if WUs are fetched hours or days apart. A repeatable synthetic benchmark would be better for measurements of power consumption relative to cTDP and PPT.]

Another concern is the maximum power draw supported by the mainboard. The Supermicro H11DSi rev 2.00 which I have is specified for up to 240 W (per socket), i.e. this one has got headroom.

BTW, I will keep the 180 W setting for the duration of the upcoming BOINC Pentathlon, but will return to the default 155 W afterwards for higher efficiency ( = less waste heat at home, and lower electric bill).

 

[StefanR5R]

@Markfw, which mainboard did you choose for the new 7452?

The SP3 boards of which I looked up specs so far all have a BMC. This should enable monitoring of temperatures and some other sensors through IPMI LAN access, as an alternative when the operating system doesn't support the sensors yet.

By default, the BMC sets the IPMI LAN address by means of DHCP. On my H11DSi based computer, I reconfigured this in the BIOS to a static IP address, because I have almost all computers in my LAN set to static addresses.

 

[Markfw]

@Markfw, which mainboard did you choose for the new 7452?

The SP3 boards of which I looked up specs so far all have a BMC. This should enable monitoring of temperatures and some other sensors through IPMI LAN access, as an alternative when the operating system doesn't support the sensors yet.

By default, the BMC sets the IPMI LAN address by means of DHCP. On my H11DSi based computer, I reconfigured this in the BIOS to a static IP address, because I have almost all computers in my LAN set to static addresses.

The ASRock Rack EPYCD8-2T, the same as my 7742. I know about the BMC port, I have the address, but thats another cord, and new software and I just know how all that works, its really for use in a data center.

 

[StefanR5R]

BTW, in Supermicro's AMI BIOS, the cTDP and PPT settings are not very intuitively arranged under "Advanced" -> "NB Configuration" (North Bridge Parameters).



Change "cTDP Control" and "Package Power Limit Control" from Auto to Manual. Then, enter the Wattage in "cTDP" and in "Package Power Limit". Typically, set them to the same value.

 

[StefanR5R]

According to servethehome, 7452's cTDP_high is 180 W (post #50). If somebody could find a public AMD datasheet with cTDP_high, that would be good...

I haven't tried setting a higher value yet.

I tried this now, although unplanned. I accidentally entered 280 W for cTDP and for PPT, which was accepted by the BIOS.

The resulting power draw "at the wall" was the same as now after I entered 180 W again, resuming the same Rosetta@home workload. I forgot to make notes of the processor clocks, but given that the system power consumption remained the very same, it's obvious that the excessive custom values were capped at the fused cTDP-high (180 W in case of EPYC 7452).

Edit,
while I had the wrong value of 280 W in there, both the web interface to IPMI and sudo ipmitool sensors showed all of the sensors as not available. Now that I have rebooted with a setting of 180 W, the sensor readings are back. This may – or may not – be related. (Apparently there are other potential reasons why IPMI might see itself unable to read sensors.)

 

[vincjij]

Its here ! 128 gig of 2666 ECC memory and a 7452 Rome 32 core, waiting on the motherboard (Thursday ??)

Hello,

Congratulations on this amazing purchase

I see you are using NEMIX RAM and may I know whether you faced any issues with it ? I am about to pull the trigger on 256GB kit (it's damn cheap compared to other brands) which will be used with EPYC 7451 + ASRock Rack EPYCD8 . Do you have any experience running NEMIX on first-gen EPYC or on EPYCD8?. Any information or advises will be of so much value to me. I live in another country and plan to ship it from NewEgg. So, want to avoid the trouble of returns and all as much as possible.

Thanks in advance.

 

[Markfw]

Hello,

Congratulations on this amazing purchase

I see you are using NEMIX RAM and may I know whether you faced any issues with it ? I am about to pull the trigger on 256GB kit (it's damn cheap compared to other brands) which will be used with EPYC 7451 + ASRock Rack EPYCD8 . Do you have any experience running NEMIX on first-gen EPYC or on EPYCD8?. Any information or advises will be of so much value to me. I live in another country and plan to ship it from NewEgg. So, want to avoid the trouble of returns and all as much as possible.

Thanks in advance.

I have a 7551 ES and dual 7601's, all with nemix ram. No issues. And I have 2 second generation EPYC on the EPYCD8-2T, no issues.

 

[vincjij]

I have a 7551 ES and dual 7601's, all with nemix ram. No issues. And I have 2 second generation EPYC on the EPYCD8-2T, no issues.

Great. Thanks a lot for the quick reply.

 

[vincjij]

I have a 7551 ES and dual 7601's, all with nemix ram. No issues. And I have 2 second generation EPYC on the EPYCD8-2T, no issues.

I would like to ask one more question. How much of a difference do you think a 3200MHz RAM (CL22) make compared to a 2400MHz CL17 one? The primary area of usage is experimenting with SIEM /DataProcessing/BigData Products which does a lot of in-memory processing. Will it make a noticeable difference? I for sure wont be running a production system or ingesting gigabytes of data per second. But I would like to know how it affects the performance in server apps. Would be great if you have any suggestions on this from your experience.

3200 CL22 is like 30-50% more expensive compared to 2400C17. So was wondering if its really worth paying for.

Thank you.

 

[Markfw]

I would like to ask one more question. How much of a difference do you think a 3200MHz RAM (CL22) make compared to a 2400MHz CL17 one? The primary area of usage is experimenting with SIEM /DataProcessing/BigData Products which does a lot of in-memory processing. Will it make a noticeable difference? I for sure wont be running a production system or ingesting gigabytes of data per second. But I would like to know how it affects the performance in server apps. Would be great if you have any suggestions on this from your experience.

3200 CL22 is like 30-50% more expensive compared to 2400C17. So was wondering if its really worth paying for.

Thank you.

Not sure. At the memory size I am buying, I didn't want to spend 50% more for the 3200. Mine is 2666. I have some 2933 coming, and it was a little more, but it will be mixed with the 2666, so it will probably run at that speed.

NOW, mind you this is a guess based on Ryzen. You may get 10% more speed on memory intense tasks, but not on everything.

 

[vincjij]

Not sure. At the memory size I am buying, I didn't want to spend 50% more for the 3200. Mine is 2666. I have some 2933 coming, and it was a little more, but it will be mixed with the 2666, so it will probably run at that speed.

NOW, mind you this is a guess based on Ryzen. You may get 10% more speed on memory intense tasks, but not on everything.

Got it. Thanks. I currently have a machine with 64GB ECC UDIMM @2400MHz. Its paired with a 3900x on a consumer board. I think I can do an evaluation by overclocking the memory slightly may be upto 2666 or 2933 and see the results. Will be good to know I guess.

 

[StefanR5R]

How much of a difference do you think a 3200MHz RAM (CL22) make compared to a 2400MHz CL17 one? The primary area of usage is experimenting with SIEM /DataProcessing/BigData Products which does a lot of in-memory processing. Will it make a noticeable difference?

I think you'd have to look for an opportunity to test this. Are the memory accesses dependent on bandwidth, or more on latency?

In #92, you mentioned EPYC 7451. That's a 24-core Naples at 180 W TDP. It supports 8 channels of DDR4-2666, no higher memory clock. Did you actually mean 7452, that is, 32-core Rome at 155 W TDP? This one has DDR4-3200 support. But 8 channels of DDR4-2400 should still provide plenty of bandwidth for many use cases when bundled with a 155 W Zen2 processor.

 

[Topweasel]

I think you'd have to look for an opportunity to test this. Are the memory accesses dependent on bandwidth, or more on latency?

In #92, you mentioned EPYC 7451. That's a 24-core Naples at 180 W TDP. It supports 8 channels of DDR4-2666, no higher memory clock. Did you actually mean 7452, that is, 32-core Rome at 155 W TDP? This one has DDR4-3200 support. But 8 channels of DDR4-2400 should still provide plenty of bandwidth for many use cases when bundled with a 155 W Zen2 processor.

He was specific in First gen Epyc in his follow up about the memory and board choice.

 

[StefanR5R]

Then the answer is that 3200 RAM won't make a difference*, because it would run at 2666.

*) except in the wallet