What controls Turbo Core in Xeons?

Page 18 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Welsper

Member
Mar 5, 2017
25
18
81
I was speaking about total system power consumption (measured with watt-o-meter in power socket). In HWInfo all the time it shows me (in idle) CPU around 6-8W, DRAM around 2W. But the funny thing is it was showing similar values when I had ~56W consumption from the 'wall' and ~36W ... so I do not trust it so much ;)

Yap, that method of measuring is better. I think I need such device too. ;)
Anyway I just checked C3/C6.
C3 can be enabled now and C6 can be partial enabled (I've 2x option C6 in BIOS setup, second one for memory/PCI-e etc. still doesn't work).
Windows didn't freeze at loading - good sign. I'm going to check stability of system.
Values in HWInfo64 become little less, not as good as yours (6-8 and 2W wow!!!) but better than before:
e9915e419ea8.png
 

vermon

Junior Member
Mar 18, 2017
10
0
1
Yap, that method of measuring is better. I think I need such device too. ;)
Anyway I just checked C3/C6.
C3 can be enabled now and C6 can be partial enabled (I've 2x option C6 in BIOS setup, second one for memory/PCI-e etc. still doesn't work).
Windows didn't freeze at loading - good sign. I'm going to check stability of system.
Values in HWInfo64 become little less, not as good as yours (6-8 and 2W wow!!!) but better than before:
e9915e419ea8.png

@Welsper - any chance for you to provide a source code for your EFI modification plus a very short description / link on how to modify/compile it?
 

Welsper

Member
Mar 5, 2017
25
18
81
@Welsper - any chance for you to provide a source code for your EFI modification plus a very short description / link on how to modify/compile it?
Source code: https://ufile.io/e548b

In Windows, you will need:
1) UDK2015: http://www.tianocore.org/udk/udk2015/
2) Any C-compiler that is supported by UDK2015. I've used VisualStudio2013.
3) In file [UDK2015]\BaseTools\Conf\target.txt change next parameters to:
ACTIVE_PLATFORM = MdeModulePkg/MdeModulePkg.dsc
TARGET = RELEASE
TARGET_ARCH = X64
TOOL_CHAIN_TAG = VS2013x86

4) open console, go to folder [UDK2015]\BaseTools
5) execute edksetup.bat
6) execute build
If it prints - Done - all is fine and UDK2015 works.

then
7) unpack my source into [UDK2015]\BaseTools\MdeModulePkg\
8) In file [UDK2015]\BaseTools\MdeModulePkg\MdeModulePkg.dsc into section [Components] add string MdeModulePkg/XR/V3x2/V3x2.inf
9) execute build in console again
10) wait for - Done - and take compiled efi driver from
[UDK2015]\BaseTools\Build\MdeModule\RELEASE_VS2013x86\X64\MdeModulePkg\XR\V3x2\V3x2\OUTPUT\V3x2.efi

Modify: It's short simple C program. It must be easy to change something.
Basic options can be swithed by #defines after all #includes . but be careful, I didn't check all possible combinations.
 
Last edited:

vermon

Junior Member
Mar 18, 2017
10
0
1
Source code: https://ufile.io/e548b

In Windows, you will need:
1) UDK2015: http://www.tianocore.org/udk/udk2015/
2) Any C-compiler that is supported by UDK2015. I've use VisualStudio2013.
3) In file [UDK2015]\BaseTools\Conf\target.txt change next parameters to:
ACTIVE_PLATFORM = MdeModulePkg/MdeModulePkg.dsc
TARGET = RELEASE
TARGET_ARCH = X64
TOOL_CHAIN_TAG = VS2013x86

4) open console, go to folder [UDK2015]\BaseTools
5) execute edksetup.bat
6) execute build
If it prints - Done - all is fine and UDK2015 works.

then
7) unpack my source into [UDK2015]\BaseTools\MdeModulePkg\
8) In file [UDK2015]\BaseTools\MdeModulePkg\MdeModulePkg.dsc into section [Components] add string MdeModulePkg/XR/V3x2/V3x2.inf
9) execute build in console again
10) wait for - Done - and take compiled efi driver from
[UDK2015]\BaseTools\Build\MdeModule\RELEASE_VS2013x86\X64\MdeModulePkg\XR\V3x2\V3x2\OUTPUT\V3x2.efi

Modify: It's short simple C program. It must be easy to change something.
Basic options can be swithed by #defines after all #includes . but be careful, I didn't check all possible combinations.

Thank you!

Do you have maybe also a list of registers and their possible values?

Also - which part is responsible for decreasing voltage offsets? (and locking it?) Because if I remember correctly they were at -20mV and locked.

EDIT:

If I understood it correctly I removed power modification

("#define DO_POWER_UPDATE 0")

I removed blocking of OC registers

//rm = AsmReadMsr64(0x194);
//rm = rm | eax14_m;
//AsmWriteMsr64(0x194,rm);

I removed vcore offset:

//#if DO_BOOST_UPDATE
// AsmWriteMsr64(0x150, 0x80000011FD600000ULL | v1);
// rm = AsmReadMsr64(0x150);
// if ((rm & dl_m) != 0)
// {
// unlock_status = 6;
// return;
// }
//#endif


And compiled/exchanged it ..... but it still froze ;)

@Welsper , @Dufus , @The Stilt , do you have perhaps any ideas why it is freezing in C6 state? or how to lower idle power usage while keeping full processing power available when needed?
 
Last edited:

Zladimir

Member
Apr 14, 2011
34
3
71
My idle without microcode and original v3.efi and activated c-states goes down to 17W, under full load up to 95W. During usual desktop and browsing my cpu wanders betwenn 30W and 50W.


Try this one: https://ufile.io/e664a1

It changes cores boost + has some power limit experiments + updates microcode to 0x39, but doesn't touch uncore.
Looks like it works. But it may have several penalty to performance somewhere in comparision with V3.efi where uncore on FULLMAX everytime.

Are there any details how the power limits got changed ? Is there any result observed after changing them ?
I dont like that mc39 is forced inside, as I have best performance results (and fully stable) with mc00.
 

Welsper

Member
Mar 5, 2017
25
18
81
And compiled/exchanged it ..... but it still froze
Yep, С6 doesn't work properly yet. I've cought couple of freezes too.
But C3 state works well.

About MSR register list - I'm newbie here. That question to @Dufus and @The Stilt.
I just have disassembled their V3.efi to find how to change boost multipliers and have searched some info in google to expand it.

PS: "#define DO_POWER_UPDATE" - that was trying to increase TDP limits. Unsuccessful. Values've changed (in HWInfo) but nothing more.
 

Zladimir

Member
Apr 14, 2011
34
3
71
@Welsper , @Dufus , @The Stilt , do you have perhaps any ideas why it is freezing in C6 state? or how to lower idle power usage while keeping full processing power available when needed?

It is an bug in Haswell itself.
Look into the intel paper under HSE20 and HSE67

http://www.intel.com/content/dam/ww...cification-updates/xeon-e5-v3-spec-update.pdf

Like the other bugs it didnt got fixed, but a bios workaround was found (microcode)

Also take a look at HSE24 and HSE48. This should explain the strange and partial different observed power draws.
 

vermon

Junior Member
Mar 18, 2017
10
0
1
It is an bug in Haswell itself.
Look into the intel paper under HSE20 and HSE67

http://www.intel.com/content/dam/ww...cification-updates/xeon-e5-v3-spec-update.pdf

Like the other bugs it didnt got fixed, but a bios workaround was found (microcode)

Also take a look at HSE24 and HSE48. This should explain the strange and partial different observed power draws.

It may be a bug ... but it's working in stock bios (no freezes with everything enabled), but does not work after the mod (there are freezes). Despite the fact that microcode is added on UEFI level (Welsper's code).
 
Last edited:

randir

Junior Member
Mar 19, 2017
15
13
41
Hi @Welsper, thanks for your dual-xeon version, but I highly suspect some race conditions/other bugs in it. I'm able to hit stable 29x on a single socket with the original V3, but results vary from run to run with your V3_2 (from 25x mult on all cores to global freeze upon microcode update). So, I've tried latest V3_2_39, and got the following after boot:

[ 2.331167] microcode: CPU0 sig=0x306f2, pf=0x1, revision=0x39
[ 2.331177] microcode: CPU1 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331187] microcode: CPU2 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331196] microcode: CPU3 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331206] microcode: CPU4 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331215] microcode: CPU5 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331225] microcode: CPU6 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331235] microcode: CPU7 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331245] microcode: CPU8 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331254] microcode: CPU9 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331264] microcode: CPU10 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331274] microcode: CPU11 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331284] microcode: CPU12 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331295] microcode: CPU13 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331306] microcode: CPU14 sig=0x306f2, pf=0x1, revision=0x39
[ 2.331316] microcode: CPU15 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331323] microcode: CPU16 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331331] microcode: CPU17 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331341] microcode: CPU18 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331350] microcode: CPU19 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331359] microcode: CPU20 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331368] microcode: CPU21 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331378] microcode: CPU22 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331386] microcode: CPU23 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331396] microcode: CPU24 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331405] microcode: CPU25 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331414] microcode: CPU26 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331424] microcode: CPU27 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331433] microcode: CPU28 sig=0x306f2, pf=0x1, revision=0x39
[ 2.331441] microcode: CPU29 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331450] microcode: CPU30 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331459] microcode: CPU31 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331467] microcode: CPU32 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331476] microcode: CPU33 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331485] microcode: CPU34 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331494] microcode: CPU35 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331504] microcode: CPU36 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331512] microcode: CPU37 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331521] microcode: CPU38 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331530] microcode: CPU39 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331539] microcode: CPU40 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331548] microcode: CPU41 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331557] microcode: CPU42 sig=0x306f2, pf=0x1, revision=0x39
[ 2.331564] microcode: CPU43 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331573] microcode: CPU44 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331582] microcode: CPU45 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331591] microcode: CPU46 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331599] microcode: CPU47 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331607] microcode: CPU48 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331615] microcode: CPU49 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331623] microcode: CPU50 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331631] microcode: CPU51 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331639] microcode: CPU52 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331647] microcode: CPU53 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331656] microcode: CPU54 sig=0x306f2, pf=0x1, revision=0x0
[ 2.331664] microcode: CPU55 sig=0x306f2, pf=0x1, revision=0x0

Only 4 cores got theirs microcode update, and, I suspect, all other hacks too. FYI, this is a dual e5-2683 with 2x14x2 topology.
 

Welsper

Member
Mar 5, 2017
25
18
81
Only 4 cores got theirs microcode update, and, I suspect, all other hacks too. FYI, this is a dual e5-2683 with 2x14x2 topology.
Hmm... other hacks work per package. I've checked MSR registers on several cores with RU.efi
(Cores which didn't execute boost hack themselves but they have neighbors in package which did it).

With microcode...
I thought per package was right, but actually I have got a similar mess too.
4 core with 39, several with 0, and several with 36 (I think these from windows mcupdate...dll).
Why 4 of them have 39... the first on each core_cluster. But mcupdate was executed only per package (on 2 cores, so V3x2 writes 2 string on loading).

Ok. Executing McUpdate on each core isn't a problem.
Try this one. https://ufile.io/5f5c31 (other hacks run per package as before)

PS: source: https://ufile.io/2009f1
 
Last edited:

IFrost

Junior Member
Mar 19, 2017
5
0
1
Hello!

Also a new user here, so far I've made it work but when I want to enabler the V3.EFI driver automatically everytime I type in "cp fs0:\V3.EFI fs1:\EFI\Boot" it says that the specified path doesn't exist? Although fs1 is where win10 is installed... anyone got a solution for this? As I don't want to activate the driver manually everytime.. thanks! Awsome thread!
 

Mahlon17

Junior Member
Mar 19, 2017
1
0
1
Hi All, I've been lurking for awhile now reading up, and have started trying to get this v3 trick working on my rig.

I am running a E5-2683 v3 OEM CPU on an ASUS X99-M WS motherboard.

I am having issues with using UBU to remove the Haswell-E microcode, and keep encountering the "Error Replacing File". I've tried on every BIOS version available from ASUS, and have been trying to track down the fix for UBU, but everything I'm reading doesn't seem applicable to my BIOS in how to use the Hex Editor to keep this error from happening.

Does anyone here have any suggestions? I know there are plenty of people using ASUS mobos for this mod, and I'm just struggling to get past this step.

Thanks in advance!
 

cekim

Member
Mar 6, 2017
87
19
41
Regarding stability. I brought up a second chip on a second taichi. I was immediately reminded that among the steps I required to get things working was to set BCLK=100.0 (with the V3.EFI - I have not tried any of the newer ones yet).

On both systems with both 2696v3's if I don't it crashes in a few minutes once booted into linux. Again both systems, changing nothing else other than BCLK from auto to 100.0 makes them both stable under many hours of sustained 16-18 core loads. I hadn't had time to go back and try to figure out exactly why this was, but it is repeatable for me.
 

Ace123

Junior Member
Jan 23, 2008
21
2
66
Is there a version,of this v3 that uses the most power? Lol I'd like a version that is geared 100% maximum performance. C,states and power savings is not a consideration.
 

Welsper

Member
Mar 5, 2017
25
18
81
Is there a version,of this v3 that uses the most power? Lol I'd like a version that is geared 100% maximum performance. C,states and power savings is not a consideration.
Use original V3.efi (V3_1.efi) for PC with single CPU and V3x2_cup.efi (from attached link) for multi CPU PC. And remove microcode update from OS.

V3x2 with variuos configurations: https://ufile.io/b7d061
cup - with uncore max (max perfomance, but high power drain on idle)
cp - without uncore modification
none/27/39 - microcode version

I've made small test (Power profile: ballanced):
7d16ab515a70.png

R15 @ 7Zip@8Th --- max result from 8x run. Corona - single run.
cp27 & cp39 --- with C3 enabled.
7Zip@8Th - simulates the loading of not all available cores.
 

TLN

Junior Member
Jan 27, 2017
21
6
41
That's an amazing work Welsper!
I've tested this weekend with ESXi: I was able to convert and load microcode to ESXi: that was pretty easy and painless.
I haven't tried any tests yet: have to reinstall VM, since my existing Win7 doesn't recognize 56vCPU's. Will probably do that later today.
 

Dingmel

Junior Member
Mar 16, 2017
19
0
66
Use original V3.efi (V3_1.efi) for PC with single CPU and V3x2_cup.efi (from attached link) for multi CPU PC. And remove microcode update from OS.

V3x2 with variuos configurations: https://ufile.io/b7d061
cup - with uncore max (max perfomance, but high power drain on idle)
cp - without uncore modification
none/27/39 - microcode version

I've made small test (Power profile: ballanced):
7d16ab515a70.png

R15 @ 7Zip@8Th --- max result from 8x run. Corona - single run.
cp27 & cp39 --- with C3 enabled.
7Zip@8Th - simulates the loading of not all available cores.


@Welsper - Bro, do your modified EFI's work with single Xeons or only double CPU'd ones.
 

Welsper

Member
Mar 5, 2017
25
18
81
Bro, do your modified EFI's work with single Xeons or only double CPU'd ones.
V3x2 isn't blocked to double socket only.
It just does modification on each found CPU in system.
It must work well with single Xeon too.
But I've got only double socket system so I haven't checked on another one.
 

Zladimir

Member
Apr 14, 2011
34
3
71
V3x2 isn't blocked to double socket only.
It just does modification on each found CPU in system.
It must work well with single Xeon too.
But I've got only double socket system so I haven't checked on another one.

It works on single quite well.
First I used Dufus v3.efi, but as Welsper offered his source, I took over and modified by myself a bit, just for experiment.
Im by myself curious, if we could get more out of our Xeons.
btw, thanks Dufus and Welsper!

@Welsper thanks for the work comparing different mcs and benches, etc.
For me I made different experiences on performance. Having an 2686v3 on a Taichi I got no difference on/off uncore boost of v3x2 efi. Also no difference between original v3 by Dufus to the new flexible v3x2 efi. This I observe between different tests plus games and matlab. However I still observe a big gape between having no microcode and any microcode (which I could find in .dat format), but that just simply cause my allcore turbo goes to x28/29 (jumping) without any mc in comparison to maximum x27 with any mc. I still wonder why it is so.

Like Welsper I experimented with TDP. No change so far, having it on 200 and 220 (Limit 1 and 2). The time of 1 is confused sitting on value "-3670016000.00 ms", having it even not changed by register. Maybe because it isnt on default in uefi. Might change that back. Still observe by hwinfo64 that "configurable tdp level 1 (down)" is sitting on 120W, no matter what limit 1 and 2 describe. Also predefined "CPU TDP" might have influence on what I ve read on intel sheets.
http://www.intel.com/content/dam/ww...eveloper-system-programming-manual-325384.pdf

Also if I understand well there is both a thermal and a power interrupt control, both seperately for the cpu as whole and for each core forcing the cpu as whole down, which is quite interesting. I further think that hwinfo64 cant read them all out correctly.
But so far I learned that,
-number of considered TDP levels can be changed (all somehow programable)
-low power mode can be fully disabled cpu wise
-programable ratio limits by standard, which is by default deactivated (maybe can be overwritten), if set yes it will always take the defined values and if off than its branded ones (option 1 and 0)
-fully programable tdp levels (option 1 and 0), similar to upwards
-e5v3 xeon explicitly supporting bclk ratios, but must be set (thats really interesting, look page 1504 register 61E)
-that it is possible to read out "why" cpu throttled. So if we go in allcore turbo and dont reach our defined values, there might be possible to find out which setting is stopping us ! (page 1504 till 1506, register 690)

Interestingly is that hwinfo64 claims that "CPU Turbo" (its max turbo), "Current configurable TDP Level" and "CPU Power Limit" 1 and 2 would be unlocked. Might it be more programmable ?

Im curious, is it possibly to force, overwrite and change every msr register ? Or are there any hardlocked ones, who will not accept any other input ? Im not fond enough of programming msr registers (knowledge behind). Even my c is way aged already, but still enough to read Welspers source (gladly, haha).
 
Last edited:

inventoroz

Junior Member
Mar 16, 2017
3
0
1
Use original V3.efi (V3_1.efi) for PC with single CPU and V3x2_cup.efi (from attached link) for multi CPU PC. And remove microcode update from OS.

V3x2 with variuos configurations: https://ufile.io/b7d061
cup - with uncore max (max perfomance, but high power drain on idle)
cp - without uncore modification
none/27/39 - microcode version

I've made small test (Power profile: ballanced):
7d16ab515a70.png

R15 @ 7Zip@8Th --- max result from 8x run. Corona - single run.
cp27 & cp39 --- with C3 enabled.
7Zip@8Th - simulates the loading of not all available cores.
Hi, I tried this EFI, and it says loaded successfully but it also says wrong CPU.
I tried to remove the microcode from my MSI x99 bios with @fromax and @Dufus method, but I really need guidance...
Can anyone help me to get this working?
What am I doing wrong?
my cpu is 306f1
 
Last edited:

ajschot

Member
Feb 16, 2017
65
2
16
Use original V3.efi (V3_1.efi) for PC with single CPU and V3x2_cup.efi (from attached link) for multi CPU PC. And remove microcode update from OS.

V3x2 with variuos configurations: https://ufile.io/b7d061
cup - with uncore max (max perfomance, but high power drain on idle)
cp - without uncore modification
none/27/39 - microcode version

I've made small test (Power profile: ballanced):
7d16ab515a70.png

R15 @ 7Zip@8Th --- max result from 8x run. Corona - single run.
cp27 & cp39 --- with C3 enabled.
7Zip@8Th - simulates the loading of not all available cores.

If understand right.... i can load this efi instead of the V3.efi I am using now?
And if so.... it would mean i have 39 microcode on other OS than windows? So also on my hackintosh?
Maybe this solves my problem with freezing my machine.... Also somebody did the bios hack for me how do i know MSR 0xE2 is unlocked?
 

ajschot

Member
Feb 16, 2017
65
2
16
Use original V3.efi (V3_1.efi) for PC with single CPU and V3x2_cup.efi (from attached link) for multi CPU PC. And remove microcode update from OS.

V3x2 with variuos configurations: https://ufile.io/b7d061
cup - with uncore max (max perfomance, but high power drain on idle)
cp - without uncore modification
none/27/39 - microcode version

I've made small test (Power profile: ballanced):
7d16ab515a70.png

R15 @ 7Zip@8Th --- max result from 8x run. Corona - single run.
cp27 & cp39 --- with C3 enabled.
7Zip@8Th - simulates the loading of not all available cores.
Yes!!! Great got my Hackintosh working on turbo with the V3x2_cup27.EFI!
Thanks.... only I have to find a way to check if it runs on all cores... I can't check but at least my OSX is not freezing. On windows all works well (but it did also with V3.EFI)
 

ZGR

Platinum Member
Oct 26, 2012
2,052
656
136
Yes!!! Great got my Hackintosh working on turbo with the V3x2_cup27.EFI!
Thanks.... only I have to find a way to check if it runs on all cores... I can't check but at least my OSX is not freezing. On windows all works well (but it did also with V3.EFI)

https://software.intel.com/en-us/articles/intel-power-gadget-20

This is really all there is for proper turbo frequency monitoring. For older than Sandy Bridge you need to go back to CoolBook with its frequency monitor in the menu bar.