What controls Turbo Core in Xeons?

[thupig]

Rename mcupdate_GenuineIntel.dll in C:\Windows\System32 to prevent loading at startup for no ucode in Windows, otherwise ucode 36 will be loaded.

Do you know what should be modified under ubuntu? Not using windows currently.

 

[CANONKONG]

I (think I) have successfully added SMP to my v3.asm

I have uploaded it to https://peine-braun.net/public_files/XEON_V3_BIOS_MODS/EFI-Drivers/v3x2/
one is -20mV and one -50mV on both core and cache.

IT is untested YET.
Unfortunatly I have no Xeon v3 around to test it out and my friend with the dual CPU board is not around for a few days.

If you would kindly test v3x2_payne*.efi and let me know how it goes.
I have also prepared *.ffs files, but please test .efi first before bricking your boards.

Again: completely untested.

Can you help me make a V3-75-50.efi, or more efi files . I think 5mv decline will better than 10mv.

 

[pututu]

Stepping 1 v3 Xeons (ES) can use V3-1.FFS. I have test v4 Xeons，it worked with change the MSR.
http://pan.baidu.com/s/1skQZ2c1
http://pan.baidu.com/s/1qYSDCF6

So, E5 v4 chips can also run all cores at max turbo? Wow, this is awesome! I saw your 22 cores v4 potentially running all cores at 37x? What about power?

 

[pututu]

That probably wont work... for me all V3x2 cannot be included for some reason... BIOS wont post.

Not Sure why... thats why I have been asking for the source a few times. Maybe its issues with the C compiler, or it is related to SMP functions. I am trying to find out.

I have uploaded some more undervolting core/cache voltage driver combos, also including the .ffs already:

https://peine-braun.net/public_files/XEON_V3_BIOS_MODS/EFI-Drivers/

Enjoy.

Use UEFITOOL to add .ffs files directly to BIOS after finding out your silicon luck with .efi

I tried the v3_payne_90_50.efi and it appears to be quite stable for the 2683v3. Previously with v3.efi, the CPU votlage averages about 0.93V. Now it is about 0.86V.

See screenshot here. Not sure if I should try to push the Vcore voltage lower. Thanks for the modded efi file.

 

[C_Payne]

Working...
Error in Driver shows: "Failure - EFI_MP_SERVICES_PROTOCOL Error"
Boot into Windows was sluggish... like multiple minutes of the spinning twirly before finally coming to rest at logon screen

Observables:
-- Cores set to maximum multi by MSR
-- Uncore stays at 3Ghz even with no ucode loaded!!! Best of both worlds: highest multi's for AVX/non-AVX workloads (no ucode) and max Uncore frequency! Perhaps this is because the earlier v3x2_vcc files also applied a -50V bias to SA and this driver does not
-- Cores are using more energy... idle temps up about 5C it seems. Needs longer period of observation and characterization to confirm any real diff... PowerCut looks to be in operations and SVID telemetry is non-nonsensical
-- Performance limit reasons appear unchanged (EDP/Max Turbo)
-- Highest performance/scoring driver I have used with my SMP Xeon system

On another note, my highest performing CPU switched from CPU#1 to CPU#0. This is good in that it indicates that uneven multi limits on SMP systems is not necessarily a result of individual CPU performance but rather a feature of working with this hack (thus far).

Can you build two more? One at -70mV and a second at -100mV?

I will gladly build more versions and also look into TDP limits, SA voltage, SVID telemtry in the future.

I have to debug "Failure - EFI_MP_SERVICES_PROTOCOL Error" beforehand.
(Maybe its related to Windows starting slowly, but i think it could also be the missing uCode)

 

[CANONKONG]

Work very well.
http://pan.baidu.com/s/1bp3vIBd

 

[junkim]

Can someone please share step by step how to insert v3.efi in the bios I tried the webpage provided by kjboughton but it’s not helpful for me .

 

[Podshivalovs]

Why my 2699V3 prod version multiplyer is only x36 not x38 ?

 

[CANONKONG]

Can someone please share step by step how to insert v3.efi in the bios I tried the webpage provided by kjboughton but it’s not helpful for me . CANONKONG does not want to share how he did. He asking for help all the time and he does not give any help.

Adding a driver to UEFI firmware:
https://github.com/pbatard/efifs/wiki/Adding-a-driver-to-a-UEFI-firmware
I use this tool. Because our methods are the same，so i don't repeat again.
---------------------------------------------------------------------------
http://pan.baidu.com/s/1c2w8e9U ---And you can use this tool .
run the Star.bat,than input xfmod xx.efi and press Enter ,xx means the file name.

 

[junkim]

Adding a driver to UEFI firmware:
https://github.com/pbatard/efifs/wiki/Adding-a-driver-to-a-UEFI-firmware
I use this tool. Because our methods are the same，so i don't repeat again.

This link is not helpful it doesn’t say where to add v3.efi. Since I don’t have any knowledge in programing I was hoping for more details step by step guide.

 

[CANONKONG]

This link is not helpful it doesn’t say where to add v3.efi. Since I don’t have any knowledge in programing I was hoping for more details step by step guide. it seems you are kind of selfish don’t like to help

http://pan.baidu.com/s/1c2w8e9U ---And you can use this tool .
run the Star.bat, input xfmod xx.efi and press Enter ,xx means the file name.
if you want V3.EFI to V3.FFS,run the Star.bat, than input xfmod V3.efi and press Enter,The ffs file will appear.

 

[junkim]

http://pan.baidu.com/s/1c2w8e9U ---And you can use this tool .
run the Star.bat, input xfmod xx.efi and press Enter ,xx means the file name.
if you want V3.EFI to V3.FFS,run the Star.bat, than input xfmod V3.efi and press Enter,The ffs file will appear.

Thanks for the tool but the ffs files already shard by C_Payne. What I am looking for is exactly where to add the ffs in the bios

 

[CANONKONG]

Thanks for the tool but the ffs files already shard by C_Payne. What I am looking for is exactly where to add the ffs in the bios

use mmtool to open the bios,and insert the ffs file to the DxeCsm or DxeCore（Different brands of motherboards do not have the same name ), than save the bios,and use UBU to recalibration.
https://www.chiphell.com/thread-1717625-1-1.html You can use google translate，this article summed up all the methods.

 

[Zladimir]

why would I care about power? serious question. the cpu is sitting at 32 c.

'Cause it makes no difference if you enforce uncore update. By my observation uncore neither makes a difference in maximum frequency nore in average. In fact it shouldnt even matter in overall performance in an Intel cpu. The C-state jumping happens in picoseconds, but is an important(!) Intel feature to reduce heat and power consumption.

@Welsper @randir
I would love if you would reupload your sources, so i can try to find out why i cant include them as ffs modules and/or add changes for dual CPU boards to my modified v3.asm
Also i would like to see how you append uCode.

As you and kjboughton asked, here is Welsper's source file:
http://www20.zippyshare.com/v/MzeqpnFO/file.html

(Thank you Welsper for your work!)

@randir never published his source, but I am highly interested in them, as he found the way to break telemtry calculation of power consumption on cpu power package.

Would be helpful if someone could add definitions and/or explanations to the following EFI driver option files:

v3.efi : First efi by @Dufus, single cpu only, no mc included

v3x2_50_vcc.efi : by @randir, most recent efi without mc and a bit higher voltage for core, cache and sa (50 for each) and faked TDP readings (no AVX2 throttle)
v3x2_50_39_vcc.efi : same but with mc39

v3x2_cp.efi : first double cpu efi (also single-cpu of course) by @Welsper, based on @Dufus idea; includes small core(20?) offset, but no mc
v3x2_cp27.efi : above and mc27
v3x2_cp39.efi : above with mc39

v3x2_cup.efi : see v3x2_cp, but with uncore feature (higher idle frequency), but no mc
v3x2_cup27.efi : above and mc27
v3x2_cup39.efi : above and mc39

Spoiler: Here

picture with power

Thanks for the picture. But it isn't the important power draw. There is only one power draw which is considered for TDP throttle. Its called "CPU Package Power" under category "CPU[#0/1] Intel E5-xxxx v3" of HWinfo. Would you be so kind to make another picture showing those reading in comparison to the other one under load ? Thanks!

Interesting, IIRC that's the MSR for changing bclk straps, i.e. running a strap of 1.67 would push a 30x multi to 50x and allow RAM with a limit of 2133MT/s up to 3500MT/s while keeping everything else happy at normal clocks.

Yes, exactly. I wouldn't go for 1.67 of course. PCIe ratio of 5:4 (1.25) would be already awesome. Especially as it will increase our TDP and EDP calculation in fact (this is written in the thermal guide and the electrical datasheet of e5-v3 made by Intel).
Im very eager about experimenting here, but I did no made much progress so far. I think after loading the efi he even sets the ratio, but something bothers him, that he freezes. I tried both with and without delayed PLL adjustment.
Please have a look into the 64-32 architecture software paper by Intel.

I think we're looking at a board limit that may be further programmable using MSR in EFI driver.

Here we can see the result of using P95 to fully-load CPU0 using FMA3 code.

IA: Electrical Design Point/Other (ICCmax, PL4, SVID, DDR RAPL) is asserted.
Assuming this is an exhaustive list of enumerated reasons for assertion (hmm), we have the following possible causes:

1. ICCmax: Likely culprit by process of elimination...
2. PL4: Unlikely as this power limit does not appear to be set as far as ThrottleStop is reporting. Option according to Intel specs.
3. SVID: Unlikely as this has to do with CPU<->VRM interface and we're not stressing that relationship at all with VCCIO at 1.7V well within typical range.
4. DDR RAPL: RAM Average Power Limit with RAM at stock 2133 MT/s.

Yes and no.
Actually after setting the max multi ratio introduced by @Dufus, there are only two throttles left we have to manage (yet):

1) TDP throttle: This throttle is solely based on nominal TDP and TDP limit 1 designed in every CPU by Intel. Those limit is responsible for AVX2 throttle. Theoretically it is writable, but neither with me or on any picture I saw proofed that a modification has an influence (he throttles on original values). Also changing TDP-0 is impossible as it is absolutely hardwired and R/O (see 64-32 software architecture guide by Intel).
However e5-v3 TDP is compared solely on the value "CPU Package Power" under "CPU [#0/1]" in HWinfo as mentioned above. I did not thought that it is possible, but @randir managed to somehow fake those value.
This value is a pure calculation! No RAPL value (core, cache, system agent and ddr) modifies this calculation. However designed by Intel VCCin gives an reverted offset to the "result" of calculation (see further downwards).
In short: Having @randir sources or understanding how he managed we successfully fight TDP throttle.

2) Electrical Design Point: This limit is the reason that we don't reach max multi on all cores.
These value is pure calculation and consists of (don't let me lie) 9 factors. It is described in detail in the electrical design sheet of e3-v5 (google). Also this value is calculated twice each for core and ring. However the last doesn't make us problems, but we throttle on core EDP (my observation).
I can say with sureness (observation goes coincide with electrical design sheet) both RAPL sytem agent and DDR have no influence on the calculation. Also VCCin and reading of CPU Package Power have absolute no influence on the calculation. However RAPL core and cache voltage offsets have! Clearly the core offsets influence is much greater than the caches one.
Yet none of us found a way except stretching the voltage limits to increase it. My best oberservation on my own is with an efi having core offset -100mV and cache offset of zero. (less cache offset seems to give a more stable higher offset for core on me)
Altough I hoped maybe to fake the reading somehow, too, like @randir managed. If not my alternative thought was to set cpu msr of PCIe-ratio as mentioned above, as it increases EDP (I read it in one of the many papers of Intel)

It looks to me as though lowering IA core Vcc is the correct approach. Of course at some point you run out of margin as you're eating into it from both ends (increased freq, decreased Vcc) and risk instability. There will exist a finite limit of do not cross.

Read above, lowering VCCin won't work, as it increases even the TDP readings (AVX2 throttle). However I by myself highly recommend to "fix" or throttle the VCCin, as by an high multi ratio the cpu vccin draws up to 1.9V and higher. This is risky as Intel clearly states in their electrical design sheet that an voltage draw of higher than 1.85 for a longer period will permanently damage your CPU.
@randir considered this in his efi version and fixed the VCCin at about ~1.65V. I by myself keep it at 1.60V and have no issues (except an higher CPU Package Power reading, which we can overthrow using randirs efi).
So in your own interest, please have an eye on your VCCin voltage, guys!

So, where do we go from here? Shall we start digging into MSRs to be programmed alongside the multi changes in the EFI driver?

Yes, MSR testing is needed I believe. Finding out if its possible to somehow fake EDP readings. Studying more deeply Intel's papers. Trying out to somehow reset PCIe ratio. I strongly believe we can get much more out of our CPUs.

And with HSW the package power estimation is made presuming 1.8V VCCIN and reported VRM current so to show lower power one actually wants to increase VCCIN as the current will be less for the same power. i.e. 100W at 1.667V would be 60A while at 2V would be 50A, so 60x1.8 (108W estimated) vs 50x1.8 (80W estimated) for the same power use of 100W. BTW those minifit connectors should be rated for 13A per pin although usually derate to 8.5A which should give 400W at 12V

Yes, it's made on purpose so by Intel. It has something to do with momentum heat production and voltage jumps if I recall it correctly from the sheets. However Intel describes this offset even with an voltage droop curve in one of their papers.
I believe their only mistake they made was, that this calculation wasn't negated in the TDP-readings, which they corrected later in one of their mc-updates (described in Haswell's error table of Intel).

I wasn't around when @randir was posting but the powercut idea was one of mine.
http://forum.notebookreview.com/threads/the-throttlestop-guide.531329/page-420#post-10204115
It's a software solution that simply disables SVID comm's which messes up power reporting. Alternatively there's IOUT slope on MSR 0x1AA that can be adjusted, however this needs to be done before BIOS post.
@C_Payne well done.

I did not know that you mentioned it before in another forum. However, did you tried it ? Does it work ? How it messes it up ? Do you have a clue if it's the same solution as randir's one ?
I definately will have a look into it. Might not need randir's efi source than. Thanks for that, too.

I wrote the first one (EFI driver) just as a proof of concept, it was never meant to be the solution. A BIOS mod was and still is the recommended choice IMHO. What the first EFI driver did was fully outlined in an earlier post. There wasn't any power modifications in that one, perhaps someone else added them later on.

A clear thanks for it! Without you, we wouldn't have our unlocked CPUs, now!

Why my 2699V3 prod version multiplyer is only x36 not x38 ?

E5-2699-v3 max multi is x36
E5-2696-v3 max multi is x38
E5-2686-v3 max multi is x35

All three are 18core cpus. See Wiki for details. It is as designed by Intel and fully correct.
There is no known way to increase it.

 

[junkim]

use mmtool to open the bios,and insert the ffs file to the DxeCsm or DxeCore（Different brands of motherboards do not have the same name ), than save the bios,and use UBU to recalibration.
https://www.chiphell.com/thread-1717625-1-1.html You can use google translate，this article summed up all the methods.

thanks. i will try to do it

 

[Cata40]

http://pan.baidu.com/s/1c2w8e9U ---And you can use this tool .
run the Star.bat, input xfmod xx.efi and press Enter ,xx means the file name.
if you want V3.EFI to V3.FFS,run the Star.bat, than input xfmod V3.efi and press Enter,The ffs file will appear.

If you insert into these bios directly, ffs file, do you need to copy efi and set bcfg driver?

 

[CANONKONG]

If you insert into these bios directly, ffs file, do you need to copy efi and set bcfg driver?

Use ffs into the bios,you don't need any efi copy or set bcfg driver.Also the PC can sleep and wake up Perfect instead the old method.

 

[Cata40]

Use ffs into the bios,you don't need any efi copy or set bcfg driver.Also the PC can sleep and wake up Perfect instead the old method.

Thanks very much, and super tool efi to ffs

 

[Zladimir]

@Dufus, I experimented with MSR 0x1AA now. Did not made any difference whatever bit I set.
However I locked SVID like you said and it worked. Now I've constant faked cpu package power draw. Thanks again!

 

[kjboughton]

@Zladimir
If I'm not mistaken, the work done by @randir to modify the V3.efi for SMP, resulting in V3x2_50_vcc.efi and V3x2_50_vcc, etc. are a reduction in Vccin of 50mV not an increase. The vcc portion refers to the PowerCut feature to break accurate CPU Package Power reporting. BTW, this is the same functionality in ThrottleStop in the FIVR section. I do believe Dufus corroborated with the main developer to implement this feature. Would be great if we could get this code in the public domain.

Please have a look into the 64-32 architecture software paper by Intel

There are so many volumes... a link would be nice

 

[C_Payne]

Working...
Error in Driver shows: "Failure - EFI_MP_SERVICES_PROTOCOL Error"
Boot into Windows was sluggish... like multiple minutes of the spinning twirly before finally coming to rest at logon screen

I have updated my v3x2 drivers and found the reason for the EFI_MP_SERVICES_PROTOCOL Error.
That be fixed now. Maybe it also fixes the boot delay?!?
Also I have added some more core/cache voltage combinations.

Drivers can be found here:

https://peine-braun.net/public_files/XEON_V3_BIOS_MODS/EFI-Drivers/v3x2/

I would love to know if the driver can be integrated into BIOS.
Btw: I use UEFITOOL to integrate the .ffs files.

 

[kjboughton]

I have updated my v3x2 drivers and found the reason for the EFI_MP_SERVICES_PROTOCOL Error.
That be fixed now. Maybe it also fixes the boot delay?!?

Success! Booted into Windows 10 (really, Server 2016) without delay! I bet it was caused by an orphaned AP thread. Message showed success for both CPUs.
Using -70mV core, -20mV cache to start...

UPDATE: Saw the highest all-core multi ever seen on this dual 2696v3 syste: 35x
CPU0: 33x @ loaded VID ~ 0.98V
CPU1: 35x @ loaded VID ~ 0.94V

Is it possible to do differential Voffset reduction per Package?

 

[Zladimir]

@kjboughton
No the reduction itself of VCCin did not faked the CPU Package Power. Disabling SVID comm's in combination with locking VCCin into fixed mode did the trick.
Dufus already told so and the functionality of Throttlestop does exactly this, as I could follow the thread correctly (+600 pages).

Here are the Xeon sheet I have:
http://www70.zippyshare.com/v/oGaxPvYs/file.html

However faking the readings is done easily and was already described by Dufus on page 40. Just have a look at it how to disable SVID comm's:

EDX = Mailbox Function
Bit(s)
7:0    Command
10:8   Domain
31     Run

EDX also returns error codes 7:0? when MSR read back, 0 = Success

EAX = Data


Domains
0      Core
1      Graphics
2      Cache
3      System Agent
4      Ananlogue I/O
5      Digital I/O

Command 0x1 Read Capabilities
Bit(s)
7:0    Maximum Ratio
8      Ratio changeable
9      Static Voltage supported
10     Offset Voltage supported

Command 0x2 Read Turbo ratios (across domains)

Command 0x10 Read Voltages and ratios, 0x11 Set Voltages and ratios
Bit(s)
7:0    Ratio limit
19:8   Static Voltage   ( V / 1024)
20     0 = Dynamic Voltage, 1 = Fixed Voltage
31:21  Offset Voltage, -1024 to 1023 (-1V to +0.999V)

Command 0x12 Read SVID, 0x13 Set SVID
Bit(s)
11:0   SVID (VCCIN),    Voltage divided by 1024, 0 = Dynamic
31     1 = Lock / Disable SVID comm's

Command 0x14 Read FIVR parameters, 0x15 Set FIVR parameters
Bit(s)
0      FIVR Faults,        ignore = 1
1      FIVR Efficiency,   disable = 1


While it appears fixed voltage should be supported for HSW Xeons only offset mode appears to work for my 2683.

 

[C_Payne]

I bet it was caused by an orphaned AP thread

I switch BSP 2 or 3 times, depending on initial BSP number.
I accidentaly left the AP in disabled state when switching, hence I could not switch back with my first version, hence the error. And that left 1 AP in disabled state.

 

[kjboughton]

@C_Payne
Cache offset for *_70_50.* files should be -20mV. Throttlestop is reporting -50mV. Core is correct @ -70mV.