What controls Turbo Core in Xeons?

Page 45 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Dufus

Senior member
Sep 20, 2010
675
119
101
I don't like other tools like AIDA64 because they show inaccurate results.

Yes, I get very similar results on my Asrock, power drawn from the wall is about twice what the CPU estimate is. AVX2 can pull over 350W with 14 cores and a little under 3GHz but cpu estimation will show around 175W. Perhaps it's made to work like that in order that people can get some decent performance even if at the end of the month it leaves them wondering if the electricity rates have gone up. Imagine setting the TDP limit in half to realize true spec performance. For instance running a CPU with a 70W power limit to realize 140W TDP performance.


However, did anyone experienced with 0x61E ? As by design this MSR is writable. System hangs when I try to write. Not sure why. Surely Im missing here something.

Interesting, IIRC that's the MSR for changing bclk straps, i.e. running a strap of 1.67 would push a 30x multi to 50x and allow RAM with a limit of 2133MT/s up to 3500MT/s while keeping everything else happy at normal clocks.
 

CANONKONG

Member
Jul 11, 2017
98
62
46
Yes, I get very similar results on my Asrock, power drawn from the wall is about twice what the CPU estimate is. AVX2 can pull over 350W with 14 cores and a little under 3GHz but cpu estimation will show around 175W. Perhaps it's made to work like that in order that people can get some decent performance even if at the end of the month it leaves them wondering if the electricity rates have gone up. Imagine setting the TDP limit in half to realize true spec performance. For instance running a CPU with a 70W power limit to realize 140W TDP performance.




Interesting, IIRC that's the MSR for changing bclk straps, i.e. running a strap of 1.67 would push a 30x multi to 50x and allow RAM with a limit of 2133MT/s up to 3500MT/s while keeping everything else happy at normal clocks.
Can you share the v3.efi with coervoffset-70mv and cachevoffset -50mv.Because the Asrock cannot change those voltages on the bios.
 

kjboughton

Senior member
Dec 19, 2007
330
118
116
Yes, I get very similar results on my Asrock, power drawn from the wall is about twice what the CPU estimate is. AVX2 can pull over 350W with 14 cores and a little under 3GHz but cpu estimation will show around 175W. Perhaps it's made to work like that in order that people can get some decent performance even if at the end of the month it leaves them wondering if the electricity rates have gone up. Imagine setting the TDP limit in half to realize true spec performance. For instance running a CPU with a 70W power limit to realize 140W TDP performance.

I'm a little astonished by this level of discussion on the matter. I hope we're all having a good laugh.

Power from the wall does not equal CPU power. They are proportional for sure but there exist innumerable other components which also require power to operate, some discrete, others built into the motherboard, fans, power conversion losses, etc.

Power can be calculated a number of ways, the simplest of which is the product of supply voltage and current when discussing powered components. CPU Vcc is on order of 1V. To reach 200W of consumption would require on order of 200A of current. I haven't done the leg work but if you were to find the max current carrying rating for that gauge wire used in the ATX 8-pin connector I bet you would find yourself within de-rating guidelines, quite possibly outside of of absolute rating for long duration power delivery.

Haswell has a depth and breadth of internal telemetry monitoring and reporting the likes of which I have never before appreciated.
One of the new things I see when I look at the sensors in HWInfo64, I find a set of sections titled CPU [#X]: Intel Xeon E5-2696 v3: Performance Limit Reasons where X is 0 or 1. In this section you can see the intersection of each individual limit as monitored and statuses by the internal PCU microcontroller. This is the controller which provides for overall control of power, voltage (each individual core VID), power gating, current monitoring, etc. It also processes DTS values in applying policy. In here I can see for both CPUs, the reason for the performance limit applied (33x for CPU0, 34x for CPU1). The same in both cases: "IA: Electrical Design Point/Other (ICCmax,PL4,SVID,DDR RAPL)". In other words, everything's working correctly. The fact that the artificial core speed governor is out of whack has not prevented the PCU from monitoring system performance as intended and applying limits to prevent catastrophic failure. If I had my guess, it's the first (Iccmax): there exists only so much current margin before you hit the limit. Without disabling this PCU protection this is likely the limit of performance. Since these set points are fluid from design to design and the PCU microcontroller should be an evolutionary design, this suggests a MSR may exist for programming or override. It may also be hard-coded in silicon for ultimate safety purposes. We should explore this further.

I see from various monitoring programs that CPU0 (max 33x) VID at full load for each IA core is in the range of about 0.97V to 1.00V. For CPU1 (max 34x) it is about 50~70mV lower on the top-end with each IA core in the range of 0.93 to 0.95V or thereabouts. This is the only obvious difference observed as to why one set of cores is happy to hum along at 34x with the other is limiting to 33x. This appears to be a weighting factor by the PCU in determining highest allowed frequency of operation. It looks to me as though lowering IA core Vcc is the correct approach. Of course at some point you run out of margin as you're eating into it from both ends (increased freq, decreased Vcc) and risk instability. There will exist a finite limit of do not cross.

Think of it like an engine. The further you open the throttle the more power you make, which is simply the product of providing ever-increasing amount of air (Oxygen) and fuel (hydrocarbons). If you let it run free it will accelerate to speeds which result in that which we in the business call rapid disassembly. The IA cores work the same way. Fed the proper level of heat rejection and voltage/current they will accelerate to any level desired (within limit). It stands to reason every core in my 2696v3 should be cable of sustained, error-free operation at 3.8GHz. (If this were not the case then a core freq boost on said core could led to a crash and that can't be.) The PCU (Power Control Unit) is the governor that steps in an enforces package-level limits on factors such as current, power (heat), etc. The PCU is the choke that tells the IA cores to slow down. Take control of the PCU and you take control of the main processor.

OK, I just stepped in it:
http://forum.notebookreview.com/threads/tdp-and-power-limiting-haswell.766743/

So, where do we go from here? Shall we start digging into MSRs to be programmed alongside the multi changes in the EFI driver?
 
Last edited:
  • Like
Reactions: Cata40

sciff

Member
Mar 6, 2017
136
52
71
I'm a little astonished by this level of discussion on the matter. I hope we're all having a good laugh.

Power from the wall does not equal CPU power. They are proportional for sure but there exist innumerable other components which also require power to operate, some discrete, others built into the motherboard, fans, power conversion losses, etc.

Did he say "power from the wall equals CPU power"? I didn't see that. You're interpreting him wrongly.


Power can be calculated a number of ways, the simplest of which is the product of supply voltage and current when discussing powered components. CPU Vcc is on order of 1V. To reach 200W of consumption would require on order of 200A of current. I haven't done the leg work but if you were to find the max current carrying rating for that gauge wire used in the ATX 8-pin connector I bet you would find yourself within de-rating guidelines, quite possibly outside of of absolute rating for long duration power delivery.

I'm not an expert in electrical stuff, but when I see HWiNFO (AND ASUS's "DUAL Intelligent Processors" app) showing 38W in idle and 240W in AVX2 load whereas socket watt meter shows 107W in idle and 323W in AVX2, I make a firm conclusion that one unlocked E5-2696v3 can consume in excess of 200W.

You don't have to be a mathematician to make simple calculations. 240W minus 38W equals 202W difference. 323W minus 107W equals 216W difference. Sure, this 14W extra that we observe can be accounted for "innumerable other components which also require power to operate, some discrete, others built into the motherboard, fans, power conversion losses, etc" running at higher speeds.

If you were right, if one 2696v3 could not take more than 145W of power, then my socket power meter would not register 216W difference between idle and load. There are simlpy no other devices that can take 216-145=71W more power to account for such difference.

By the way, I have a pretty well cooled system and my fans don't rotate much faster when the CPU is under maximum load.
 
Last edited:

kjboughton

Senior member
Dec 19, 2007
330
118
116
I think we're looking at a board limit that may be further programmable using MSR in EFI driver.

Here we can see the result of using P95 to fully-load CPU0 using FMA3 code.

IA: Electrical Design Point/Other (ICCmax, PL4, SVID, DDR RAPL) is asserted.
Assuming this is an exhaustive list of enumerated reasons for assertion (hmm), we have the following possible causes:
  1. ICCmax: Likely culprit by process of elimination...
  2. PL4: Unlikely as this power limit does not appear to be set as far as ThrottleStop is reporting. Option according to Intel specs.
  3. SVID: Unlikely as this has to do with CPU<->VRM interface and we're not stressing that relationship at all with VCCIO at 1.7V well within typical range.
  4. DDR RAPL: RAM Average Power Limit with RAM at stock 2133 MT/s
DesignLimitAsserted.jpg


Power as reported by on-board Semi ASP1257 current/voltage monitor at time of performance limit assert:

PowerAtAssert.jpg


In this snapshot CPU0 VRM is inputting 139W to output 127W (~91% conv. eff.). Inputting (in this instance) 11.875A at 11.844V and putting out 79A at 1.652V (drooping under load).

The IA Cores are the ultimate users and so we need to calculate their supply current at their supply voltage (average of 1.0025V):

IACoreVIDAtAssert.jpg


127W / 1.0025V = 127A

A good estimate as there will be sample and reporting rounding, etc.
 
Last edited:

sciff

Member
Mar 6, 2017
136
52
71
kjboughton, my E5-2696v3 says "yes" even to 200A. However, due to core throttling (as I understand), the supply current gradually drops. Take a look at this:







 

kjboughton

Senior member
Dec 19, 2007
330
118
116
OK, lots going on there. You're starting to throttle (and limit down from 35x to 34x) because you're running out of thermal headroom. The E5-2696 v3 has a Tj max of 83C. You can see as time goes on your cores are heating to this level but not reaching or exceeding 83C (82C max shown). This is causing the reduction in frequency. As the CPU heats leakage increases and this causes it to heat some more. Which is why it's quite easy to get into a thermal runaway situation where you think you are stable but due to an incredibly slow heat-up rate you actually fail days out.

Can you show the line items in HWInfo64 that you are using to monitor CPU Power and CPU Current?
 

Dufus

Senior member
Sep 20, 2010
675
119
101
No worries FCG. Intel's power measurements have AFAIK always been described as estimations and usually work quite well if running in specification but who is doing that. Here is an old Sandybridge observation I made as some_one on XS.
http://www.xtremesystems.org/forums...80mhz-loaded&p=4994589&viewfull=1#post4994589

And with HSW the package power estimation is made presuming 1.8V VCCIN and reported VRM current so to show lower power one actually wants to increase VCCIN as the current will be less for the same power. i.e. 100W at 1.667V would be 60A while at 2V would be 50A, so 60x1.8 (108W estimated) vs 50x1.8 (80W estimated) for the same power use of 100W. BTW those minifit connectors should be rated for 13A per pin although usually derate to 8.5A which should give 400W at 12V

The ucode AVX2 multi drop can be seen just by injecting one AVX2 instruction per core every millisecond or less without pulling a relatively huge amount of power. It seems on my own CPU a VID of ~0.950V seems to be the sweet spot for it to operate at, of course cache can have an effect too. Mind you I only have the one Xeon so results are quite limited.

I wasn't around when @randir was posting but the powercut idea was one of mine.
http://forum.notebookreview.com/threads/the-throttlestop-guide.531329/page-420#post-10204115
It's a software solution that simply disables SVID comm's which messes up power reporting. Alternatively there's IOUT slope on MSR 0x1AA that can be adjusted, however this needs to be done before BIOS post.

@C_Payne well done.
 
  • Like
Reactions: sciff

kjboughton

Senior member
Dec 19, 2007
330
118
116
@Dufus, I take it then the powercut fundamental has been applied to this EFI driver? I ask because my xeon are both reporting less than a watt usages with only a small proportional change with change in load (a few milliwatts at most).

It's looking more and more like something we haven't seen and can't touch: EDP.
 

C_Payne

Junior Member
May 16, 2017
22
11
41
Thanks Payne. Just to confirm you only change the Vcore voltage offset from -20mV to -70mV and nothing else in the original V3.efi file, correct?

I had to make quite a few changes to the original code to make it compile and did some cleanup. (lots of unneccesary push/pop, lots of n00b-unfriendly formatting etc., I think it is now more readable.. at least to me (first time asm x86_64), assembler pro's might disagree)
Also i am probably using a different version of fasm (ubuntu 16.04) and I couldnt find efi3.inc.
The different size is the dos-stub that i didnt bother to remove (yet)

But other then that i did not (at least not intentionally) change any of the v3.efi behavior
... oh and i put my Name in :oops:

I am doing this for a firend of mine and dont have the hardware on hand which makes debugging hard, but he told me it was working.

Code:
format pe64 dll efiboot
entry main
include 'efi.inc'

;-------------------------------------------------------------------------------------------------------

 CoreVOffset            = -70
 CoreVStatic            = 0          
 CoreMulti              = 0          
 CoreFixed              = 0          

 if       CoreVOffset   > 999
          CoreVOffset   = 999
 else if  CoreVOffset   < -1000
          CoreVOffset   = -1000
 end if

 CoreAdjVOffset         = ((CoreVOffset shl 15)/1000) and 0ffe0h
 CoreAdjVStatic         = ((CoreVStatic shl 18)/1000) and 0fff00h
 CoreVoltage            = ((CoreAdjVOffset shl 16) or CoreAdjVStatic) or CoreMulti or (CoreFixed shl 20)
;-------------------------------------------------------------------------------------------------------

 CacheVOffset           = -50
 CacheVStatic           = 0
 CacheMulti             = 0
 CacheFixed             = 0

 if      CacheVOffset   > 999
         CacheVOffset   = 999
 else if CacheVOffset   < -1000
         CacheVOffset   = -1000
 end if

 CacheAdjVOffset        = ((CacheVOffset shl 15)/1000) and 0ffe0h
 CacheAdjVStatic        = ((CacheVStatic shl 18)/1000) and 0fff00h
 CacheVoltage           = ((CacheAdjVOffset shl 16) or CacheAdjVStatic) or CacheMulti or (CacheFixed shl 20)
;-------------------------------------------------------------------------------------------------------


section '.text' code executable readable

main:
 
 mov [Handle], rcx         ; ImageHandle
 mov [SystemTable], rdx    ; pointer to SystemTable

 lea rdx, [_Start]
 mov rcx, [SystemTable]
 mov rcx, [rcx + EFI_SYSTEM_TABLE.ConOut]
 sub rsp, 4*8
 call [rcx + SIMPLE_TEXT_OUTPUT_INTERFACE.OutputString]
 add rsp, 4*8

 mov     eax,0
 mov     edx,0
 mov     ecx,8bh
 wrmsr
 mov     ecx,0
 inc     eax
 cpuid
 cmp     eax,306f2h
 jne     WrongCPU
 mov     ecx,8bh
 rdmsr
 cmp     edx,0
 jne     MicroCodePresent
 mov     ecx,194h
 rdmsr
 bt      eax,20
 jc      OC_Locked

 mov     ecx,150h                ; Get Core OC ratio and capabilities
 mov     edx,80000001h
 mov     eax,0
 wrmsr
 rdmsr
 cmp     dl,0
 jne     MailBoxError
 mov     [TopCore],al

 mov     edx,80000201h           ; Get Cache OC ratio and capabilities
 mov     eax,0
 wrmsr
 rdmsr
 cmp     dl,0
 jne     MailBoxError
 mov     [TopCache],al

 mov     eax,CoreVoltage     ; Set Core voltage
 mov     al,[TopCore]
 mov     edx,80000011h              
 wrmsr
 rdmsr
 cmp     dl,0
 jne     MailBoxError

 mov     eax,CacheVoltage     ; Set Cache voltage
 mov     al,[TopCache]
 mov     edx,80000211h
 wrmsr
 rdmsr
 cmp     dl,0
 jne     MailBoxError

 mov     al,[TopCore]            ; Set turbo ratio's
 mov     ah,al
 mov     dx,ax
 shl     eax,16
 mov     ax,dx
 mov     edx,eax
 mov     ecx,1adh
 wrmsr
 inc     ecx
 wrmsr
 inc     ecx
 or      edx,80000000h
 wrmsr

 mov     ecx,620h                ; Set Cache Min/Max Ratios
 rdmsr
 mov     al,[TopCache]
 mov     ah,al
 wrmsr

 mov     ecx,194h
 rdmsr
 bts     eax,20
 wrmsr

 lea rdx, [_Success]
 jmp Text_Exit

WrongCPU:
 lea rdx, [_WrongCPU]
 jmp Text_Exit

MicroCodePresent:
 lea rdx, [_MicroCodePresent]
 jmp Text_Exit

OC_Locked:
 lea rdx, [_OC_Locked]
 jmp Text_Exit

MailBoxError:
 lea rdx, [_MailBoxError]
 jmp Text_Exit

Text_Exit:
 mov rcx, [SystemTable]
 mov rcx, [rcx + EFI_SYSTEM_TABLE.ConOut]
 sub rsp, 4*8
 call [rcx + SIMPLE_TEXT_OUTPUT_INTERFACE.OutputString]
 add rsp, 4*8
 mov rax, EFI_SUCCESS
 retn

section '.data' data readable writeable

TopCore    db ?
TopCache   db ?

Handle      dq ?
SystemTable dq ?

_Start              du 13,10,'Xeon v3 All-Core Turbo Boost EFI Driver by C_Payne v0.1 (Single CPU)',13,10,0
_WrongCPU             du 'Failure - Wrong CPU.',13,10,0
_MicroCodePresent     du 'Failure - Microcode present.',13,10,0
_OC_Locked            du 'Failure - Overclocking Locked.',13,10,0
_MailBoxError         du 'Failure - Mailbox Error.',13,10,0
_Success              du 'Success.',13,10,0
 

kjboughton

Senior member
Dec 19, 2007
330
118
116
@C_Payne, this is interesting. Please tell me what you think.

CPU0 is running a max all-core multi of 33x under AIDA64 stress load.
CPU1 is running a max all-core mulit of 34x.
I chose AIDA64 as it's one of the few programs out there that seems to stress the number a logical cores in this machine (72).

See if you can spot the difference in these pics:

CPU0_AIDA64_stress.jpg


CPU1_AIDA64_stress.jpg

Are we sure these Vcore offsets are being applied to both CPUs?
 

C_Payne

Junior Member
May 16, 2017
22
11
41
Are we sure these Vcore offsets are being applied to both CPUs?

Most certainly not. It is single CPU like v3.efi right now.
What puzzles me is why u have high multiplier on both CPU's... You have multi setup in Bios? or did you load another efi before?
 

Dufus

Senior member
Sep 20, 2010
675
119
101
@Dufus, I take it then the powercut fundamental has been applied to this EFI driver? I ask because my xeon are both reporting less than a watt usages with only a small proportional change with change in load (a few milliwatts at most).

It's looking more and more like something we haven't seen and can't touch: EDP.
I wrote the first one (EFI driver) just as a proof of concept, it was never meant to be the solution. A BIOS mod was and still is the recommended choice IMHO. What the first EFI driver did was fully outlined in an earlier post. There wasn't any power modifications in that one, perhaps someone else added them later on.
 
  • Like
Reactions: sciff

CANONKONG

Member
Jul 11, 2017
98
62
46
I wrote the first one (EFI driver) just as a proof of concept, it was never meant to be the solution. A BIOS mod was and still is the recommended choice IMHO. What the first EFI driver did was fully outlined in an earlier post. There wasn't any power modifications in that one, perhaps someone else added them later on.
But only the first V3.EFI &V3-1.EFI can package into the bios,and the othor EFI cannot.
 
Last edited:

CANONKONG

Member
Jul 11, 2017
98
62
46
Most certainly not. It is single CPU like v3.efi right now.
What puzzles me is why u have high multiplier on both CPU's... You have multi setup in Bios? or did you load another efi before?
Thank you very much!I am trying to package into the bios.