What controls Turbo Core in Xeons?

[ajschot]

https://software.intel.com/en-us/articles/intel-power-gadget-20

This is really all there is for proper turbo frequency monitoring. For older than Sandy Bridge you need to go back to CoolBook with its frequency monitor in the menu bar.

Thanks i already using that app but you can only see one core i can not see all 14 cores individually.

 

[ajschot]

After few days of testing, here's what I've discovered:

The most essential thing is that the CPU is initialized WITHOUT a microcode. Allegedly it is possible to initialize the CPU with an extremely old microcode version, but so far I haven't been able to find such version (hence allegedly). Microcode version 0x1F (06/03/2014) is already too "new" to prevent this exploit from working. Since each and every motherboard bios is supplied with a microcode present (for obvious reasons), initializing the CPU without a microcode mandates that the microcode is completely removed from the bios binary. This naturally involves modifying the bios and updating it, which in some cases can be little tricky.

After testing all of the different microcodes I could find, I've found out that there are rather large differences between them. The most important thing is, that it appears that Intel has no direct or indirect means to completely prevent this exploit from working. Technically they can reduce the "yield" (clocks) in certain workloads, but not prevent it completely as it is too late when the CPU has already been initialized. Newer microcode builds generally contain workarounds for errata and because of that it is generally recommended to use the newest build available. When using this exploit you'll need to decide if you want to have the highest possible performance in all workloads, possibly at the expence of reliability or alternatively slightly lower performance at the best known reliability (i.e with the most recent microcode update).

Haswell was the first "wide" core from Intel (256-bit FP). In order to preserve power, the Power Management Unit (PMU) power gates the upper 128-bit of the FP when 256-bit instructions are not executed. In somewhere between August and September of 2014 Intel changed the behavior of the Turbo on Haswell. Previously the Turbo behavior was identical regardless if the upper 128-bit of the FP was executing or not (i.e same clocks for 128-bit and 256-bit workloads). In the microcode released in September 2014 the Turbo behavior was changed significantly, from static to workload dependant. In this microcode and all the newer ones the Turbo clocks are exactly the same for 128-bit workloads as before, but significantly lower for 256-bit workloads. On my CPU the difference is 400MHz.

The newest microcode version for the Haswell-E/EP/EX/EN production stepping (CPUID 0x306F2) is version 0x39 (10/07/2016). This microcode can be used for this exploit, however it will result in lower yield (clocks) than the earlier ones. This microcode is highly recommended if you are satisfied with a more modest boost, or require maximum reliability (professional use). This microcode also has an additional advantage on systems, which lack both the "Power Limit" or "CPU telemetry feature" (SVID) options in the bios. Version 0x39 microcode is one of the few versions, which doesn't feature the bug I call as the "LFM bug". The best way to describe the "LFM bug" is that when you use this exploit, load a newer microcode in flight and then try adjusting any of the CPU parameters (frequency, voltage, power limits, etc), the CPU will lock to the LFM state (typically 800MHz).

I personally ended up using microcode version 0x27 (08/08/2014), and this is the version which offers the best performance. This versions still features the static Turbo behavior (same for 128/256-bit workloads) and has some of the most critical Haswell-Ex erratas (such as TSX) already fixed.

Additionally there appears to be some Turbo rules, which appear to be core configuration dependant and completely fixed.

These apply on my Haswell-E HCC, but they might be different on other variants:

- >= 10 cores == Maximum Turbo Ratio available
- >= 12 cores == Maximum Turbo Ratio - 100MHz
- >= 14 cores == Maximum Turbo Ratio - 200MHz
- >= 16 cores == Maximum Turbo Ratio - 400MHz
- >= 18 cores == Maximum Turbo Ratio - 500MHz

This means that when 0x27 microcode is used, I can run my 2699 at 3.6GHz (1-10 cores), 3.5GHz (with 12 cores), 3.4GHz (with 14 cores), 3.2GHz (with 16 cores), 3.1GHz (with 18 cores), regardless of the workload.

Since the microcode can be updated in flight, controlling the microcode version in Windows might be slightly harder.
For Windows 7 - 8.1 (including their server variants) update KB3064209 must be uninstalled, in case it is found in the system. This is a microcode update, which contains microcode version 0x2E for Haswell-Ex.
Windows 10 meanwhile is distributed with microcode version 0x36. To remove it, file named "mcupdate_GenuineIntel.dll" found in System32 folder must be renamed so that the system no longer finds it. Note that I haven't tested this procedure personally, since I'm still using Windows 7.
For Linux using a specific microcode version should be quite well documented else where.

The microcode in Windows can be updated with a driver released by VMWare: https://labs.vmware.com/flings/vmware-cpu-microcode-update-driver
Here are version 0x27 & 0x39 microcodes for Haswell-Ex (0x306F2) in VMWare driver / Linux compatible format: https://1drv.ms/u/s!Ag6oE4SOsCmDhFnET3uw9wHeV4EA
Rename the desired version to microcode.dat, and proceed as instructed by VMWare.

Personally I gained around 28% of performance with this exploit.

Is there a way to get rid of your microcode? I loaded 0x39 but now using the new V3x2_cup27.EFI so i want to remove the microcode in windows but how?

 

[cpierr03]

Hello, guys! Here is simplified method to modify Bios on motherboards where
UBU doesn't work, such as MSI and Gigabyte. This method have already proved
itself on several motherboards so I think it will work on most other
boards. The main idea belongs to Dufus and have been creatively reworked by me). I hope he does not mind.
The whole point of method is just to change the name of microcode 306F2 to 506F2 in bios
instead of cutting it. You can do it like this:
-Download HxD hexeditor and open you bios in it.
-press Ctrl-F , choose hex-values
-find all strings 03 F2 06 03 00 , and change to 03 F2 06 05 00,
-again Ctrl-F, find all strings 00 F2 06 03 00, and change to 00 F2 06 01 00
-save new bios, done
-check new bios in UBU, we have to see 506F2 instead of 306F2 in microcode section

Any one have any suggestions for "Error in replacing file" in UBU? I tried the hex editor change, but it doesn't seem to work.

Board is an ASUS Z10PA-U8 using this BIOS

 

[ajschot]

did anyone ever tried with other cpu's than the E5-2683v3? Like the E5-2686v3, 2696v3 or 2699v3?

 

[ajschot]

on the OC ANANDTECH pack you have 06 VMWARE MICROCODE UPDATER, cpumcupdate2.1
in this last folder you have 2 options, install or uninstall

Thanks i found it .... sorry it was a very stupid question

 

[cekim]

did anyone ever tried with other cpu's than the E5-2683v3? Like the E5-2686v3, 2696v3 or 2699v3?

yes

 

[Methanoid]

Would be nice if someone could write a guide post with links to s/w

As far as I can tell the process is

1) Take latest (?) BIOS for your mobo - flush CPU microcode (some s/w needed to do this)
2) Flash BIOS to mobo (ensure you have a dual BIOS board!) using some diff BIOS updater s/w (link?)
3) Some EFI shell thing
4) Boot Windows (but what about Linux, and does it have to be an EFI booting OS - I want to boot unRAID)

Would help a lot if those who HAVE done this could help refine some proper instructions

 

[cpierr03]

So let me share some of my experience going through this modification:

Fist, I had to modify my ASUS .cap BIOS to get UBU to read it without the "Error in replacing file" error. This was still necessary even after dumping my BIOS, so it seems the BIOS is still protected that way as well.

First, I had to take my BIOS and open it in UEFI Tool (included with UBU) and search for GUID 1BA0062E-C779-4582-8566-336AE8F78F09. I extracted "as is", and opened it in a hex editor.
I had to modify the 0x10 value from 0C to 08, AND (other guides aren't clear about this) 0x10 from 86 to 8A. This allowed me to strip the microcode as noted in earlier guides on this thread.

However, my board, Z10PA-U8, does NOT support BIOS Flashback, only ASUS EZ-Flash. Problem is, I couldn't flash the BIOS in EZ-Flash, due to a signature verification error. I got similar errors trying to flash it in a couple Windows and DOS tools as well.

The only way I was able to flash the modded BIOS was using an old version of AFUWINx64 that supports the /GAN switch to bypass the image verification. This is a dangerous procedure, but it worked once I found the old version. The only other way to have done this would have been to use an SPI flasher, which I do not own.

Loading the V3 driver was easy, I used V3x2_cup39.efi to avoid having to remove/replace microcode within Windows (thanks Welsper!!!!). I tried a few of the efi files Welsper provided, but this one gave me the best performance. I was able to get the idle power consumption down by enabling C3/C6, which seem to work just fine on my board with these modifications.

I have a QS version of the 2683 V3, S-Spec QGN5. For what it's worth, I was able to boot all of the efi drivers provided, and I was able to boot without any microcode whatsoever with this CPU.

Hope some of this information helps anyone out there.

 

[Eltama]

Can you share the modded bios? I have your same same board and cpu.

Thank you in advance for any help you can provide

 

[Mr.Qaz]

Hi all,
joined the forum to ask for some help removing the microcode from my Gigabyte x99 SLI motherboard.
My CPU is a e5-2686 v3 ES with the cpuid CPUID Revision 000306F2h
I'm not sure which program I should be using to remove the microcode from my motherboard, as UBU isn't working.
Any help would be appreciated!

 

[b9creations]

Hello everyone, awesome thread!
So would purchasing one of these taichi boards be the easiest way to set up a 2683 V3? Or is there a recommended dual CPU board I can get modded bios for?

 

[TLN]

Hello everyone, awesome thread!
So would purchasing one of these taichi boards be the easiest way to set up a 2683 V3? Or is there a recommended dual CPU board I can get modded bios for?

ASUS and Asrock Dual CPU been reported as good. I'm running ASUS.
Taichi is probably the most popular option, but if you find something else cheaper - why not? You can mod bios for almost any board. Except may be Gigabyte.

 

[b9creations]

Hi TLN, thanks for the reply. Probably not dell mb either? : ) Just bought precision workstation thinking there wasn't much to clock with the xeons. : ) Which ASUS board are you running?

 

[TLN]

Hi TLN, thanks for the reply. Probably not dell mb either? : ) Just bought precision workstation thinking there wasn't much to clock with the xeons. : ) Which ASUS board are you running?

I know QS CPU's are not working on Dell. But you can try modifying it. Who knows.

I'm running Z10PE D16 WS, but here're people with all kinds(WS, non-WS, 8 and 16 DDR) and all reported it's working.

 

[kevin335200]

>Try this one. https://ufile.io/5f5c31 (other hacks run per package as before)

Now this loads 0x39 ucode for all cores, thanks.

Hi randir, Could you show me the cinibench score of 2x E5 2683 V3 after the exploit?
Thanks a lot! I want to build a dual-xeon workstation too

 

[randir]

Hi randir, Could you show me the cinibench score of 2x E5 2683 V3 after the exploit?
Thanks a lot! I want to build a dual-xeon workstation too

Cinebench R15:

No uCode ~3650
39 uCode ~3860

This is achieved with no BCLK overclocking and far from optimal memory configuration (2ch one cpu and 3ch another). So for a baseline I've made a run without any overclocking and no ucode loaded (I was too lazy to prepare a separate image) and got ~2950 points for that. So you can approximate what you can get out of your system with both CPUs at 4ch from other's scores.

 

[Oneto]

Hi

It's my overboost.
CPU: 2683v3 QS
MB: ASRock x99 Gaming i7

CPU-Z: over 24000
Cinebench R15: over 2300
Corona 1.3 Render: 1.24

Link for my video proof:

Can you share your modified BIOS for the ASRock x99 Gaming i7?

 

[ZGR]

Thanks i already using that app but you can only see one core i can not see all 14 cores individually.

Want to help write the software?

 

[Oneto]

I was able to find UBU and modify my BIOS
Everything was pretty easy following the steps on page 3

Asrock X99 Professional Gaming
E5-2695 V3 ES/QS 306F2


Before Changes:

After Changes:

 

[rottwag]

Hi everybody!!!

This is really unbelievable!! Amazing news for everybody... I am one of the "stupid" people that have a Xeon E5 2618L V3 (none ES/QS... its retail). Got it for 150$ - so thats fair I think. It is basically identical to the 5960X but with 75 TDP, locked and lower freq.

Turbo bins are:
36 / 36 / 30 / 29 / 28 / 27 / 26 / 25

Running currently rock stable @105.9 BLCK on a Asus X99 A (not A-II).... not bad so far... so but using all cores resulting in 2650 Mhz currently.... not too much. I have a setting for MCE (currently disabled as not working) and I know that I do not have the latest BIOS version.

So am I right that I need to do the following:
- Deleting the microcode in my BIOS (as I have an ASUS Board it must be combined with an BIOS upgrade ?)
- Changing the BIOS settings to MCE = enabled, TDP override = 240 Watt (currently 150)
- Renaming this .dll in my Windows 10 ?

Is it correct? Anything else to do?

Thanks + best regards
Andy

 

[knowndragon]

Okay guys, This thread has been pretty helpful. It has cause me to sign up thanks to the OP.
Specs:
X99 Sli. Gigabyte mobo
e5 1320 v3
R9 fury x

I have attempted to follow the instructions. I downloaded ubu v1.65 and seem to have run on a stump. ( I am completely new to bios modding.) I would like to achieve this because sad part is when under workload my cpu shuts down to one core at max overclock I.E. 3804mhz. I believe if I could get this to work I could work all cores instead of some shutting down. I really don't understand what I am supposed to change except tricking the motherboard into no microcode. I have googled a lot and then I signed up. Does anyone think that they are up to helping?

I download the bios I want. x99sli.f1 use ubu tool and press 7 and then 1. I then examine mcode and it state 29 and 13 I believe. I will post screen shots after work. I try the 0/skip just to have the program closed out. Tried to flash to other microcode and it runs into error replacing file. I close mmtool window down. UBU prompts wish to terminate job. I press N and it finishes but when I open file again nothing has changed. I can post bios and screen shots if anyone is willing to help.

 

[RamRock]

Welcome all I usually don't do that (register to forum asking for help, I usually give help)

I know QS CPU's are not working on Dell. But you can try modifying it. Who knows.

I'm running Z10PE D16 WS, but here're people with all kinds(WS, non-WS, 8 and 16 DDR) and all reported it's working.

I have same problems Z10PE D16 WS -> can't delete microcode. I tried 3_x.EFI, tried put 39ucode but...
everything stuck at 2800MHz (2696V3)... as I understood from first posts I could run it at 3,6GHz (for 6cores)....
... everything what I have done I verify with Cinebench and hash test (running on 6 cores)... it's always 2,8GHz.
Results are almost the same.

Can someone help?

EDIT:
Link to bios
https://1drv.ms/u/s!AuYW-NmhIHupljydbx5p8eitsb1g

 

[TLN]

I have same problems Z10PE D16 WS -> can't delete microcode. I tried 3_x.EFI, tried put 39ucode but...
everything stuck at 2800MHz (2696V3)... as I understood from first posts I could run it at 3,6GHz (for 6cores)....
... everything what I have done I verify with Cinebench and hash test (running on 6 cores)... it's always 2,8GHz.

If you check the CPU in BIOS - what does it shows there? Microcode version - 0, you've successfulyl removed microcode. Until you do that - you may not even try loading your own.

 

[RamRock]

If you check the CPU in BIOS - what does it shows there? Microcode version - 0, you've successfulyl removed microcode. Until you do that - you may not even try loading your own.

39
I read somewhere here that 39 microcode also got the bug.
Can you share no microcode Z10D16WS bios?

 

[knowndragon]

So I managed to remove all mcode for GIGABYTE x99 sli. If you do this do so at your own risk. It was a rather barbaric way of doing so. Windows 10, simply download ubutools. Once this is done, download bios. I used f1 bios.

Go to UBU folder. Then open modules. Then open mcode. Then open 2011-v3 and expand the file to where you can see date. Run UBU as admin on the desired bios. Press any key to continue. Then press 7 then select 9 (view cpu microcode) see the date of the first one.

Now go back to the modules/mcode/2011v3 and find the one that corresponds with your mcode. I used HXD hex editor it was easier for me. Opened the mcode that corresponded with my date. Copied the first line of code. Then use Uefitool opened the bios I wanted to use. Search for body and text and past the copied hex and find. In message windows should find double click there. It will take you to main window code. Right click on the highlighted portion and extract body. I saved it in main folder as mcode. I then opened this up and was lost so I deleted everything. Saved the file and then went back to the main window in uefitool where you extracted the body. Then right click and choose replace body save the file in main folder. Just to make sure all went well run ubu again hit any key to continue. Then press 7 then select 9. MMtool should pop up and it will have two things both saying (NO CPU MICROCODE PATCH PRESENT IN VOLUME 1 AND THE SAME FOR 2.) Close this window and go back to main ubu window and exit don't X out but hit 0 and then 0 again. It will prompt to save as 1 or other. Select bios.bin or 2nd choice. Rename file and flash on default setting using qflash.

Booted up and rebooted adjust bios files and Windows 10 now loaded microcode. I will touch back later if the efi shell works.