What controls Turbo Core in Xeons?

[ziollos]

Hi guys,

I'm wondering if there's anything else I could do to increase performance of my pair of E5-2697 v3. Motherboard: Z10PE-D16 WS.
Microcode in BIOS removed. Undervolt applied. Didn't change anything else in bios. My cb15 score is 3926. Half year ago when was fiddling around a lot with this setup I managed to achieve 4495, but unfortunately I forgot how I did it.
Could anyone share his thoughts?

 

[protator]

Hey guys,

anyone here who could re-upload a mc-less bios for the Z10PE-D8 WS ?
foureight84 's original upload has been deleted.

Also is anyone else getting poor single-core performance with that board?
My 2690s should be around 140cbp in CB R15 ... what I get is 40-45. Even the 2693 ES @ 3.2 in my old rig manages 133cb.
Strangely MT seems fine, 3700cbp with stock bios settings on a fresh W10 install.
No idea what to make of this.

Any help is much appreciated.

 

[Toysrme257th]

I've got an Asus Z10PE-D16 WS and 32gb of Crucial [redacted]

Ran great!

[redacted]

We do not allow the selling of items outside of the
FS/FT forum, and you need 25+ posts in order to
post in there.

AT Mod Usandthem

 

[FST]

Hey guys . Is it possible to set the turbo boost above the maximum turbo boost value?

 

[MOF]

Hey guys . Is it possible to set the turbo boost above the maximum turbo boost value?

It can be set any number up to 80 but never goes beyond to default maximum turbo boost value.

 

[protator]

Still hoping for a re-up of the Z10PE-D8 Bios.

I've "solved" the performance issues at least.
It seems most of the countless power management and performance setting affect the cpu's idle clocks ...
raise them for better responsiveness ... which conflicts with the default TDP profile.
More continuous juice to the whole package -> less or no headroom for single core boost.
It actually makes sense and is probably nothing new to ppl more experienced with server hardware than me.
But even with that knowlage in tow I just wasn't able to find a combination of settings that fully unleash the cpus. Well, it's an Asus board so the manual is useless for anything more advanced than setting the UI language and I assume there's a setting that does something different than what the name implies.
So I set everything back to "auto" and the base profile to "energy efficient" ... and who would've guessed:
continuous all-core-boost when needed and full single-core clocks.
In good old Asus fashion switching to manual and leaving everything on default does not result in the same behavior as the default preset so there's no way for me to figure out what's wrong with my manual settings.
Oh well, in the end the only difference is that I can't change cpu states like this - I would've liked to limit the package to C0/C1 - but a cool and stable system with 3770/149cb in R15 isn't bad at all.

 

[protator]

Btw is there a chance that removing the ucode also helps with memory OC?

My Samsung RDimms refuse to run at anything faster than 2133 (well, 2240 @105BClk).
The few posts I was able to find on the net all stated that simply deactivating "POR" and setting a frequency manually should be all that's neccessary to increase speed by at least one step.
But nada.
So it's probably the IMC - which only supports 2133 - over ruling the board.
Not being able to raise clocks nor tighten timings is really frustrating.

 

[himtan]

Hello. Was anyone able to successfully do this mod with an ES (306f1) cpu ?

 

[protator]

Hello. Was anyone able to successfully do this mod to an ES (306f1) cpu ?

It depends on the individual ES.
My E5 2693 V3 ES for example does not boost single cores beyond the all-core boost frequency - like the majority of ES chips - and I don't need a mod to lock it at that frequency.
"Multicore Enhancement" enabled, Bclk set to 105 -> all cores locked at 3250MHz, but that's as fast as she'll ever be.

If you have one of the rarer ES chips that have higher single-core multipliers enabled, or a QS chip, then there's no reason why it shouldn't work.
All E5 V3 Xeons - ES, QS, retail - have this bug.

 

[himtan]

It depends on the individual ES.
My E5 2693 V3 ES for example does not boost single cores beyond the all-core boost frequency - like the majority of ES chips - and I don't need a mod to lock it at that frequency.
"Multicore Enhancement" enabled, Bclk set to 105 -> all cores locked at 3250MHz, but that's as fast as she'll ever be.

If you have one of the rarer ES chips that have higher single-core multipliers enabled, or a QS chip, then there's no reason why it shouldn't work.
All E5 V3 Xeons - ES, QS, retail - have this bug.

Haha what a coincidence. I have the exact same CPU as you. I have also bulk set to 105 and asus multicore enhancement enabled. Turbo speed is 34x so theoreticaly it should do 105x34 = 3570Mhz to all cores with this mod from 3250Mhz that is hitting now.

 

[ziollos]

Hi guys,

I'm wondering if there's anything else I could do to increase performance of my pair of E5-2697 v3. Motherboard: Z10PE-D16 WS.
Microcode in BIOS removed. Undervolt applied. Didn't change anything else in bios. My cb15 score is 3926. Half year ago when was fiddling around a lot with this setup I managed to achieve 4495, but unfortunately I forgot how I did it.
Could anyone share his thoughts?

Could someone elaborate on this? I know that almost 4k in CB15 is very reasonable, but knowing that it is still possible to bump it up by 10% is very enticing

 

[serverpanda]

(please delete my post)

 

[balnazzar]

Hi guys,

I see that this thread is still active, so I'll dare and post.

Since the thread is so large, I find it quite difficult to use it as a source of information. Particularly, while I found many post to be useful, it is hard to understand the current state of affairs.
I got a X99 Taichi, and I'm in the process of selecting a xeon (I'll probably go for a 2697v3).

I understand I need a modded bios (which I found: the x99tc_1_04), but I still don't understand what kind of EFI I need (and more importantly, why).

It would be great if any of you could put a sticky post at the beginning, just with a brief summary including the essential steps.

Thanks a lot!!
Andrea

 

[serverpanda]

I understand I need a modded bios (which I found: the x99tc_1_04), but I still don't understand what kind of EFI I need (and more importantly, why).

It is actually quite simple. To apply the code to enable all core turbo you have to make sure that no microcode has been loaded. To achieve this, you have to modify the Bios and remove or mask the microcode update in the Bios. Then you want to load the code to enable the turbos. See for example here, there are also some releases pre-built. One way of loading the code is to add an EFI file to load from your UEFI Bootmanager before booting the Operating system. Another way would be to (somebody correct me, if I am saying it wrong) to take the EFI, package it into an FFS (basically the EFI with a GUID and some headers added) and add it to the Bios as an additional driver. The last thing to do, is to make sure that your Operating System only applies a microcode update that does not break the functionality of the turbo-hack. People therefore use microcode revisions prior to 0x39. On Linux you use the iucode-tool to restrict the microcode to older versions, on Windows people use some kind of VMware driver.

 

[balnazzar]

First and foremost, thanks for your reply, @serverpanda !

It is actually quite simple. To apply the code to enable all core turbo you have to make sure that no microcode has been loaded. To achieve this, you have to modify the Bios and remove or mask the microcode update in the Bios.

Ok. And, if I'm not making mistakes, this is done by the modded bios.

Then you want to load the code to enable the turbos. See for example here, there are also some releases pre-built. One way of loading the code is to add an EFI file to load from your UEFI Bootmanager before booting the Operating system.

Ok. And this would require to install all your OSes in uefi mode (not legacy). Am I correct?

Another way would be to (somebody correct me, if I am saying it wrong) to take the EFI, package it into an FFS (basically the EFI with a GUID and some headers added) and add it to the Bios as an additional driver.

That would free us from the necessity of an UEFI-only install for the OS?
Furthermore, let me ask: is there around any guide about how to do that?

The last thing to do, is to make sure that your Operating System only applies a microcode update that does not break the functionality of the turbo-hack. People therefore use microcode revisions prior to 0x39. On Linux you use the iucode-tool to restrict the microcode to older versions, on Windows people use some kind of VMware driver.

I think I understand now, thanks. Again, let me ask for a guide about how to do it for both windows and linux without making damages. I sought a method to unlock the turbo once and for all (e.g. without the necessity of tinkering with the OSes) but it seems that's not an option.

Last but not least, by reading a vast amount of previous posts, it seems that the "responsiveness" of the various xeon e5-26xx v3 varies a lot. The ones more suited to go at full "1-core" turbo speed with all cores seem to be the 2699 and the 2697, among those with more than 10 cores. I was oriented towards a 2698 (16 cores, 2.3G base freq but 13X boost for 1-core, hence 36X multi, good price/core count/multi ratio, at least on paper), but it seems somewhat not advised by other users.. What do you think?

Thanks!!

 

[serverpanda]

Ok. And, if I'm not making mistakes, this is done by the modded bios.
That would free us from the necessity of an UEFI-only install for the OS?
Furthermore, let me ask: is there around any guide about how to do that?

Yes, I think you are right, but I am just a simple user. Some of those consumer-grade mainboards also allow modifying the turbo speeds directly in the Bios, but I am not familiar with this.

I would suggest you use the search function at the top of this page and limit the search to this thread. It is easier then trying to go through it.

As for the Bios modifications and driver injection just google for UBU, UEFItool, MMtool... But it depends on the mainboard.

Last but not least, by reading a vast amount of previous posts, it seems that the "responsiveness" of the various xeon e5-26xx v3 varies a lot. The ones more suited to go at full "1-core" turbo speed with all cores seem to be the 2699 and the 2697, among those with more than 10 cores. I was oriented towards a 2698 (16 cores, 2.3G base freq but 13X boost for 1-core, hence 36X multi, good price/core count/multi ratio, at least on paper), but it seems somewhat not advised by other users.. What do you think?

I would compare the relevant benchmarks for your use case with the recent processor generations. Unless you need to use a lot of memory or dual-CPU, an AMD might be a more modern choice.

 

[topmysteries5]

First and foremost, thanks for your reply, @serverpanda !



Ok. And, if I'm not making mistakes, this is done by the modded bios.



Ok. And this would require to install all your OSes in uefi mode (not legacy). Am I correct?



That would free us from the necessity of an UEFI-only install for the OS?
Furthermore, let me ask: is there around any guide about how to do that?



I think I understand now, thanks. Again, let me ask for a guide about how to do it for both windows and linux without making damages. I sought a method to unlock the turbo once and for all (e.g. without the necessity of tinkering with the OSes) but it seems that's not an option.

Last but not least, by reading a vast amount of previous posts, it seems that the "responsiveness" of the various xeon e5-26xx v3 varies a lot. The ones more suited to go at full "1-core" turbo speed with all cores seem to be the 2699 and the 2697, among those with more than 10 cores. I was oriented towards a 2698 (16 cores, 2.3G base freq but 13X boost for 1-core, hence 36X multi, good price/core count/multi ratio, at least on paper), but it seems somewhat not advised by other users.. What do you think?

Thanks!!

As far as I know 2697v3 runs all 14 cores @3.3-3.4ghz under heavy load with this mod. You're looking at roughly 2300CB score in cinebench. Also do note that you'll need a good liquid cooler to cool it, since TDP is almost doubled, cpu will consume around 230-240watts on full load. Cheapest CPU which can work with this is E5 2678v3, costs ~160 usd on aliexpress. It can run all 12 cores @3-3.1ghz under heavy load. Now is the time to get newer ryzen than investing in this. Overclocked ryzen 3700x performs similar in multithread and has much higher single thread performance. Xeon platform is best if you need lots of RAM with ECC, want to use multiple GPUs at full PCIe bandwidth, multiple nvme in raid without sharing pcie bandwidth from GPU.

You can check my last post in which i shared benchmark scores from my Dual 2686v3 rig.

 

[mek5]

Hi,
you can download newest moded bios for ASUS Z10PE-D8 WS here:

Z10PE-D8-WS-ASUS-4001.CAP and 2 more files

3 files sent via WeTransfer, the simplest way to send your files around the world

we.tl

Contents:
Z10PE-D8-WS-ASUS-4001.CAP // original BIOS file from ASUS
Z10PE-D8-WS-ASUS-4001_MOD.ROM // moded BIOS w/o microcodes
v3x2_60-50-50_microcode_39_vcc1.8.efi // efi driver that works stable for me (24/7 for few months - used with BIOS 3407 before)

Unfortunately you need to use AFUWIN64_v3.05.04 to flash ROM bios (older version that allows to use /GAN command). The trick is to flash with proper BIOS first:

afuwinx64.exe <NAME OF THE ORIGINAL BIOS>.CAP

and then flash with moded (without restarting):

afuwinx64.exe <NAME OF THE MODDED BIOS>.ROM /GAN

After restart you will have new BIOS without microcode. Apply any efi driver that suits your platform.

PS. if anyone know, how to change moded ROM to CAP let me know (to use with ezflash/flashback or ASUS Update).

CINEBENCH R15 score for DUAL E5-2686v3 and 103.5BCLK is 4860 points. With 105.0BCLK it's over 5000. Turbo for all cores is 3100 but with other efi driver I can achieve 3200-3300.

 

[mek5]

Hey guys,

anyone here who could re-upload a mc-less bios for the Z10PE-D8 WS ?

see my post above

 

[balnazzar]

Thank you @serverpanda and @topmysteries5 for the information and advice you gave me.

I know that Ryzen 3000 are very capable cpus with an almost unbeatable price/performance ratio. They are not for me, though. Indeed, I'm building my next deep learning platform: I need more than 16 lanes (the bare minimum are 3 slots at 8x each for the GPUs, the taichi is good since its 16x/16x/8x layout), and 128gb of ram, preferably ECC.
But most of all, I need support for Intel MKL, which the amd cpus do lack. The best option for me would be a Xeon Gold/Platinum, for which I found many QS samples at good prices on ebay, with a core count between 16 and 22. However, I cannot manage to find a single-socket lga3647 board with enough 16x slots...

In my experience @topmysteries5, the Xeons E5 do consume way less power than their nominal TDP. My E5-2680v2 (10c/20t, 115W) tops at ~90W under prime95 at 3.1ghz (all cores). If I disable turbo, it drops to ~60/65W @2.8Ghz, measured with coretemp. I do expect however, that the actual power draw won't scale linearly with frequency. I do expect the -97v3 to scratch 200W when modded, which maybe will require liquid, or at least an air cooler so good it would cost almost as a liquid.
One option to stay cool would be the 2660v3. It has 10 cores with a base freq of 2.6 ghz. Max turbo 2.9Ghz on all cores, that would become 3.3 Ghz with the mod. Ten haswell cores at 3.3 won't be bad, and since the cpu has a TDP of 105W, I don't expect it to draw more than ~120W as modded. In any case, I'll be fine with the cooler I always use with 2011 processors, the CM Hyper 412R (20$).

The next option would be the 2690v3 (12C, 2.6,3.1,3.6 respectively base, turbo on al cores, turbo modded). It's a 135W cpu, I dunno if I'll be ok with the air cooler.

As a footnote, I find the performances of Ryzen 3000 to be a bit too good to be true... Look:

PassMark - AMD Ryzen 7 3700X - Price performance comparison

PassMark - Intel Xeon E5-2697 v3 @ 2.60GHz - Price performance comparison

PassMark - Intel Xeon W-2155 @ 3.30GHz - Price performance comparison

The 2697v3 has almost twice the cores of the 3700x, and even the mighty W-2155 is beaten, no matter being latest intel generation and with 10 cores..

Don't know, it stinks a bit..

 

[topmysteries5]

Thank you @serverpanda and @topmysteries5 for the information and advice you gave me.

I know that Ryzen 3000 are very capable cpus with an almost unbeatable price/performance ratio. They are not for me, though. Indeed, I'm building my next deep learning platform: I need more than 16 lanes (the bare minimum are 3 slots at 8x each for the GPUs, the taichi is good since its 16x/16x/8x layout), and 128gb of ram, preferably ECC.
But most of all, I need support for Intel MKL, which the amd cpus do lack. The best option for me would be a Xeon Gold/Platinum, for which I found many QS samples at good prices on ebay, with a core count between 16 and 22. However, I cannot manage to find a single-socket lga3647 board with enough 16x slots...

In my experience @topmysteries5, the Xeons E5 do consume way less power than their nominal TDP. My E5-2680v2 (10c/20t, 115W) tops at ~90W under prime95 at 3.1ghz (all cores). If I disable turbo, it drops to ~60/65W @2.8Ghz, measured with coretemp. I do expect however, that the actual power draw won't scale linearly with frequency. I do expect the -97v3 to scratch 200W when modded, which maybe will require liquid, or at least an air cooler so good it would cost almost as a liquid.
One option to stay cool would be the 2660v3. It has 10 cores with a base freq of 2.6 ghz. Max turbo 2.9Ghz on all cores, that would become 3.3 Ghz with the mod. Ten haswell cores at 3.3 won't be bad, and since the cpu has a TDP of 105W, I don't expect it to draw more than ~120W as modded. In any case, I'll be fine with the cooler I always use with 2011 processors, the CM Hyper 412R (20$).

The next option would be the 2690v3 (12C, 2.6,3.1,3.6 respectively base, turbo on al cores, turbo modded). It's a 135W cpu, I dunno if I'll be ok with the air cooler.

As a footnote, I find the performances of Ryzen 3000 to be a bit too good to be true... Look:

PassMark - AMD Ryzen 7 3700X - Price performance comparison

PassMark - Intel Xeon E5-2697 v3 @ 2.60GHz - Price performance comparison

PassMark - Intel Xeon W-2155 @ 3.30GHz - Price performance comparison

The 2697v3 has almost twice the cores of the 3700x, and even the mighty W-2155 is beaten, no matter being latest intel generation and with 10 cores..

Don't know, it stinks a bit..

It seems you have been misinformed about power consumption. Check my post #2804, dual 2686 consumes ~600-650watts of power after this mod, before it used to consume maximum of 350watts. If you want so many PCIe lanes, then go with Asus Z10PeD16 ws or D8 WS. Dual 2696v3 can reach 4ghz with oc. With dual CPUs you'll have 80 pcie lanes. Go with 8 memory sticks for 8 channel, dual 2696v3 score ~ 5600cb in cinebench r15. Plz don't use core temp to measure cpu power, use corsair link or wattmeter on wall for accurate measurement. 250-300w each cpu in a closed case is way too much for even best air coolers like noctua NHd-15, unfortunately i cant fit 2 of them on my asus mobo. Liquid is only option here. You'll have to expect 1.6-1.8x tdp after this mod. Plz do check E5 2678v3 (QS version of 2680v3), they sell for very cheap and can run 3-3.1ghz on all 12 cores. I'm running 2x 2686, max speed under load is x30*105mhz (3.15ghz on all cores). If i enable only 10 cores on each cpu, i get fixed 3.66ghz on 20 cores 40 threads, which is way better for softwares which don't use all those cores. Similarly 2696v3 can lock to 4ghz (105mhz bclk), when only 10-11 cores are enabled.

 

[balnazzar]

Thanks @topmysteries5. If you measured accurately such values, I could even have problems running a single hefty processor. For example, supposing that the modded 2697 draws 220-240W as you mentioned, I'm at risk of system shutdown. I got a CM V1200 psu (1200W), but I got three 1080ti, each one of which peaks at 300W, and in deep learning, they are used concurrently.

I think I'll go for the 78v3 you mentioned, for 12C@3ghz, or the forementioned 60v3, which should be able to do 10C@3.3 ghz at an evel lower power draw (I think the 78v3 is rated 120W).

Again, thanks!

 

[Micrornd]

balnazzar,

Have you looked at these?

For 2066 W2000s

MW51-HP0 (rev. 1.0) | Server Motherboard - GIGABYTE Global

Intel® Xeon® W-2200 and W-2100 Processor FamilyIntel® C422 Express ChipsetQuad Channel RDIMM/LRDIMM DDR4, 8 x DIMMsIntel® VROC readyASMedia 2142 USB 3.1 Gen...

www.gigabyte.com

For 3647s Gold, Platinum, W3000s

C621-SU8 (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

Lasting Quality from GIGABYTE.GIGABYTE Ultra Durable™ motherboards bring together a unique blend of features and technologies that offer users the absolute ...

www.gigabyte.com

 

[balnazzar]

balnazzar,

Have you looked at these?

For 2066 W2000s

MW51-HP0 (rev. 1.0) | Server Motherboard - GIGABYTE Global

Intel® Xeon® W-2200 and W-2100 Processor FamilyIntel® C422 Express ChipsetQuad Channel RDIMM/LRDIMM DDR4, 8 x DIMMsIntel® VROC readyASMedia 2142 USB 3.1 Gen...

www.gigabyte.com

For 3647s Gold, Platinum, W3000s

C621-SU8 (rev. 1.0) Key Features | Motherboard - GIGABYTE Global

Lasting Quality from GIGABYTE.GIGABYTE Ultra Durable™ motherboards bring together a unique blend of features and technologies that offer users the absolute ...

www.gigabyte.com

Thanks for your suggestions.

I have had VERY bad experiences with Gigabyte stuff. In my humble opinion, the best are the Asrock Rack (some models are very hard to find though), followed by Asrock consumer, followed by Asus, followed by MSI.

The specs of the Gigabyte s3647 are great, however. Exactly what I'd need. The 2066 is interesting, but the W2000 are really hard to find at interesting prices (>1000usd for an used 8 cores, come on...), whereas the Scalable can be found in QS versions at good prices.

If Asrock releases something single-3647 aimed at workstations, given the recent W3000 release, it should be the basis for my next rig!

I have decided, for now, to go and purchase the 2660v3, which is crap cheap, and in the end 10 haswell cores at 3.3Ghz should be sufficient, at least for now.

 

[Micrornd]

I have had VERY bad experiences with Gigabyte stuff. In my humble opinion, the best are the Asrock Rack (some models are very hard to find though), followed by Asrock consumer, followed by Asus, followed by MSI.

Please bear in mind these are not consumer or prosumer boards, they are from the business division of Gigabyte.
I have used only their dualies since SM denied our claim for burst capacitors 1 month out of warranty (way back when, and when everyone else was fixing them).
Their business class workstation and server boards are rock solid, but do not have OC abilities (maybe a step or 2 in memory, but no CPU OC)
These boards can be hard to find, but they will sell direct to a business or "business".
They also have some really nice and competitive pre-built servers, if that interests you.
And if you need to call with a problem, when you tell them the board #, you are transferred to the business division.
That is a whole world away from the consumer, prosumer help line, trust me.
And they will cross-ship on warranty if you need in now (as any business division should)

All I can tell you is that if you want something solid and dependable to work with, their business class is it.
(And yes, their claims of multiple gpus running on their boards are honest, I ran quad 7970s for a while on a PESH3, and 1070 TI sli on this MW30)

If you are looking for something to benchmark and OC, these ain't it

BTW, I do have a nice Gigabyte C621-WD12 board sitting here waiting for prices on 3647s to drop.
Now that's a board

Oh, and my MW30 runs dual 2696v3 @ 3.4ghz (all 36 real cores) with the bios mod and Windows 10 loading the microcode.