Discussion EPYC builders thread

[Markfw]

So now that I have 2 boxes, I thought I would start a builders thread, and also pass along what I have learned.

First, They are are cheap, and they are a different animal to build (see below)

OK, so first, I got these 2 7551 ES chips for $300 each, I thought, well that is cheap for a 32 core CPU, and with 8 channel memory they won't be handicapped like my 2990wx's.

Now to the bad... First they were only compatible with ONE motherboard that was available to buy, and with a BIOS that I had to fight with Gigabyte to provide me with ! (not available online)
So I tried several other motherboards and different BIOS's since I could not even find this one online. FINALLY I found the board at $470. I updated the BIOS (odd way, boot into motherboards default stream environment (or something like that) and do a DOS bios update. Well, it worked. Then while trying to install linux mint 19.2, no internet ???? Oh, I did not read the fine print. No ethernet that goes outside the local lan. Only SFP+ (whatever that is), so after some reasearch, I find the adapter for $13 more.

OK fast forward to when I now have an install and updated with ethernet. This 32 core chip only runs turbo all core at 1.6 ghz !!! So I spent almost $500 on the motherboard, and for 128 gig of ECC registered (the cheapest I could find@64 gig on 8 sticks or more) at another $530 and a $300 CPU I have $1330 invested in a 1600 mhz 32 core box... Picture:

And before I got this working, one of the motherboards I tried (dual socket) that would not work with these ES chips, I found 2 7601 used retail chips for $1750 and a $650 motherboard. Now this one needed 16 sticks so that was $1100. So now I am in this one $3500 with no SSD, case or PSU. Well, I get it running, and it only boosts all core to 2.6 ghz. But at least thats a far cry to 1600 mhz.

Here:

Bottom line ? The new 3960x and 3970x are not overpriced. At the same speed they are almost twice as fast. And don't venture into EPYC chips unless you have wads of cash to blow and learn. I bought another motherboard for my 2nd 7551 ES chip, so now I have to build that box too !

But I will soon have 3 server grade ECC memory chip boxes with 128 cores and 256 threads for almost $6000. I can't even get video cards to fit in these, as the memory and heatsinks are in the way ! I have to use the on-board video.

 

[balnazzar]

Hi folks!

I got to build an Epyc Rome workstation (target: deep learning research), and I have a specific requirement: the board must allow sleep (suspend-to-ram) or at least hibernation (suspend-to-disk).

I already had an EPYCD8-2T, and I had to return it since there was no way to put it into sleep.

Maybe you can report if your own boards do allow sleep/hibernation?

Thanks in advance!

 

[Markfw]

My 5 EPYC systems are always@100% use, so no idea. But 3 of my 5 motherboards are the EPYCD8-2T

 

[StefanR5R]

Exactly like Markfw, I have no use for sleep or hibernation.
I set up my swap partitions too small for hibernation, hence can't test it unless I add another disk.
I now tried sleep (initiated it from the respective button on the login screen) and it went to sleep but I did not find a way to wake it back up. I had to power-cycle it in order to recover.

(This is dual EPYC 7452, Supermicro H11DSi, openSUSE Leap 15.2 with kernel 5.3.18.)

 

[StefanR5R]

For anyone with Supermicro boards with BMC (AMD boards as well as Intel boards), here is useful information on fan control:
https://forums.servethehome.com/index.php?resources/supermicro-x9-x10-x11-fan-speed-control.20/

H11DSi features Supermicro's infamous problem with slow-spinning fans: The BIOS misdetects fan failure all the time and periodically spins the fans up, then down again.
My workaround:
Use a triple fan adapter to plug the small and faster spinning 40 mm fan and the two slow spinning 140 mm fans into a single fan port, and show only the RPM signal of the 40 mm fan to the board.

I am still using the workaround of disconnecting the RPM sensor wire of fans which have a too low minimum speed. It's a bother.

The board firmware has got these fan speed thresholds:

• "Low nonrecoverable": less than or equal to 300 RPM — If at least one fan runs at this or a lower speed, the BMC firmware logs a respective event, and the board ramps all fan headers up to 100 % PWM duty cycle.
• "Low critical": less than or equal to 500 RPM — This just causes a respective event being logged.
• "High critical": 25400 RPM; I don't have fans which would go that high.
• "High nonrecoverable": 25500 RM; again I don't know what will happen at that threshold. Best to approach such systems only wearing ear protection and, importantly, certified eye protection.

Fan speeds are represented in the firmware in steps of 100 (i.e. obviously by an internal 8 bit value between 0…255).

I now tried the Noctua NA-FC1 fan controller. This one has got a "no stop" mode in which it controls the fan such that at least 300 RPM are maintained. Unfortunately, the Supermicro firmware still asserts "Low nonrecoverable" state when a fan is in this mode, and applies the mentioned 100 % PWM duty cycle panic.

 

[zvonimir]

Hi, I am building a 72 TB storage system using Supermicro's H11DSi. How much care I need to take when matching CPUs for 2P configuration? Is it sufficient to just buy any two EPYC 7601 and things will just "work", or some additional matching is needed?

THANKS!

 

[StefanR5R]

Hi, I am building a 72 TB storage system using Supermicro's H11DSi. How much care I need to take when matching CPUs for 2P configuration? Is it sufficient to just buy any two EPYC 7601 and things will just "work", or some additional matching is needed?

I am not aware of different revisions within the 7001 series, which might prevent two specimen of the same SKU to work together in a 2P board. In other words, you should be fine with CPUs bought from e.g. different sellers. But keep the following in mind:

• Obviously the P models (7551P, 7401P, 7351P) are for single-socket computers only.

• If you buy used, then CPUs which were taken from e.g. Dell servers won't work anymore in a motherboard of a different vendor.

• Mainboards which already received a Milan-ready BIOS won't have support for Naples. Even a Rome-ready BIOS might not contain Naples support. I am not sure about the state of Naples support in Supermicro's BIOSes in particular.

 

[zvonimir]

Super thanks for nice answer. I received yesterday another CPU that does not seem to work.

I have ordered an new, unboxed "HPE AMD EPYC 7302". Do you know if this is also somehow locked to HPE systems only? It seems from the article you quoted that is does not...very frustrating! I tried to save some $ initially, and did not expect that this locking would burn me so bad...

 

[Markfw]

I got burned, sort of. They did not specify that it was "dell only" cpu, so I got a full refund. BUT, if I read this correctly (quote from the article) if its NEW, its not locked yet, only when it first powers up, its then locked to that hardware vendor.

"HPE does not use the same security technique that Dell is using for a BIOS hardware root of trust. HPE does not burn, fuse, or permanently store our public key into AMD processors which ship with our products. HPE uses a unique approach to authenticate our BIOS and BMC firmware: HPE fuses our hardware – or silicon – root of trust into our own BMC silicon to ensure only authenticated firmware is executed. Thus, while we implement a hardware root of trust for our BIOS and BMC firmware, the processors that ship with our servers are not locked to our platforms. (Source: HPE)"

 

[eek2121]

I am not aware of different revisions within the 7001 series, which might prevent two specimen of the same SKU to work together in a 2P board. In other words, you should be fine with CPUs bought from e.g. different sellers. But keep the following in mind:

• Obviously the P models (7551P, 7401P, 7351P) are for single-socket computers only.

• If you buy used, then CPUs which were taken from e.g. Dell servers won't work anymore in a motherboard of a different vendor.

• Mainboards which already received a Milan-ready BIOS won't have support for Naples. Even a Rome-ready BIOS might not contain Naples support. I am not sure about the state of Naples support in Supermicro's BIOSes in particular.

So it sounds like the CPUs are tied to that specific board. That is really dumb. I wonder what happens if the board dies?

 

[zvonimir]

Are you guys saying that new "AMD HPE" will bind permanently to the SuperMicro board, or you are saying it will not work at all?

My interpretation is that only HPE bios would do something, and since my SuperMicro is generic, it should not do any locking? What do you think?

 

[Markfw]

Are you guys saying that new "AMD HPE" will bind permanently to the SuperMicro board, or you are saying it will not work at all?

My interpretation is that only HPE bios would do something, and since my SuperMicro is generic, it should not do any locking? What do you think?

I am not sure, after reading that quote several times. But your odds of it working are very good I think. NEW is the key. But you could return it if not.

 

[zvonimir]

Thanks! I am so eager to get this system started!

 

[lightmanek]

Are you guys saying that new "AMD HPE" will bind permanently to the SuperMicro board, or you are saying it will not work at all?

My interpretation is that only HPE bios would do something, and since my SuperMicro is generic, it should not do any locking? What do you think?

You're right, Supermicro will not lock your CPU. This locking is available for OEM vendors and they can choose to use it (Dell) or not (majority of others). Any board you purchase on an open market like Supermicro or Tyan will not lock any EPYC CPU, but buying 2nd hand processor carries the risk of purchasing a locked unit if it was recycled from a server where locking is implemented. As far as I understand that security mechanism, it fuses CPU to a specific vendor and it should work on another board / server from that vendor. It should not be locked to single board or even board model.

 

[zvonimir]

Thanks! Yes, I think I am interpreting it that way, as well.

By the way, most datacenter grade hard disk drives and solid state disk drives have locking mechanisms, either as a part of ATA security, or via various TCG security interface mechanisms. However, when these are to be disposed, vendors usually invoke Security erase mechanisms which shred the original encryption key(s) altogether, AND also
bring the drive into original unlocked state. However, CPUs have no useful data on them, to incentivize the vendor to bring it back to original state.

If this thing with DELL locking is really true - i.e. that some fuses are permanently set etc, it is a real shame in terms of creating electronic waste. One thing is true for sure: used Dell AMD EPYC parts on e-bay are cheaper!

 

[lightmanek]

Thanks! Yes, I think I am interpreting it that way, as well.

By the way, most datacenter grade hard disk drives and solid state disk drives have locking mechanisms, either as a part of ATA security, or via various TCG security interface mechanisms. However, when these are to be disposed, vendors usually invoke Security erase mechanisms which shred the original encryption key(s) altogether, AND also
bring the drive into original unlocked state. However, CPUs have no useful data on them, to incentivize the vendor to bring it back to original state.

If this thing with DELL locking is really true - i.e. that some fuses are permanently set etc, it is a real shame in terms of creating electronic waste. One thing is true for sure: used Dell AMD EPYC parts on e-bay are cheaper!

It is true that Dell lock EPYC CPU's

Here is a great video about that topic, pros and cons of this approach and reason it even exists:

 

[zvonimir]

Thanks, this is indeed a good video about this! I guess I should have studied this a little bit more. In my recent experience with e-bay used parts, I have discovered that there is a lot of parts changing hands, and not all of them are tested for being Dell parts. Even worse, from what I was able to see, there
is no mechanism to differentiate between Dell CPU and non-Dell CPU
that is dead. My BMC in H11Dsi does not see anything at all with Dell CPU...