cpufreq driver behavior is not as expected

[hshen1]

My CPU is Ivy-bridge i3770K. My OS is ubuntu 12.04 with kernel 3.5.0-34.


I used to play cpufreq driver a lot. But its behavior is weird on this machine.
It has four physical cores. With SMT enabled, there are totally 8 logical CPUs. And CPU 0 and CPU 4 are in the same physical core.


First I echo 'userspace' to the scaling_governor of all cpus. Then I echo '1600000'(minimum frequency supported) to scaling_setspeed of all cpus. Then I echo '3400000' to scaling_setspeed of CPU0 and CPU4.


What should be expected is that physical core 0 (consists of CPU0 and CPU4)'s frequency should be changed to be 3.4GHz while other cores' frequency should stay at 1.6GHz. However, by repeatedly checking cpuinfo_cur_freq of all the cpus, I have found all cpus' frequency are jumping with 1.6GHz and 3.4GHz all the time.


However, if I set all the cpu's frequency to be 3.4GHz, then they will all stay in 3.4GHz.


This is not expected. I believe though the voltage change probably is socket level, the frequency change should be core level,right?

 

[Cerb]

Seems reasonable to me. You have 1 CPU. The normal way they work is that every core runs at the same speed, but might not always be on. IE, running at 3.9, only 1 core is active, but 3.4 and below, all cores may be active. Running different cores at different speeds would be a nightmare, for performance. Some AMD CPUs I know support it (I don't recall if the FX series do or not), but it really serves no purpose other than to drive the OS insane, if controlled by the user (they would reduce core speeds based on being under low load, then automatically increase them again if loaded, transparently to the user).

 

[JoeRambo]

Scaling governors are out of fashion in linux. Knowledgeable ppl are advising to use new infrastructure like pstates driver in 3.9 kernel ( but you will need to build your own).

Discussion about pstates, intel_idle etc is here:
https://plus.google.com/117091380454742934025/posts/2vEekAsG2QT

 

[Jodell88]

Scaling governors are out of fashion in linux. Knowledgeable ppl are advising to use new infrastructure like pstates driver in 3.9 kernel ( but you will need to build your own).

Discussion about pstates, intel_idle etc is here:
https://plus.google.com/117091380454742934025/posts/2vEekAsG2QT

In the OP he says he's using the 3.5 kernel. Also, the pstate governor only works with sandy bridge cpus, ivybridge will be supported in the 3.10 kernel.

 

[hshen1]

Seems reasonable to me. You have 1 CPU. The normal way they work is that every core runs at the same speed, but might not always be on. IE, running at 3.9, only 1 core is active, but 3.4 and below, all cores may be active. Running different cores at different speeds would be a nightmare, for performance. Some AMD CPUs I know support it (I don't recall if the FX series do or not), but it really serves no purpose other than to drive the OS insane, if controlled by the user (they would reduce core speeds based on being under low load, then automatically increase them again if loaded, transparently to the user).

Yes.What you said might be true. I am a researcher and I just need to do some research on how to control the frequency of different cores though different frequency of different cores might not be useful as you said.
But anyway, I think the hardware now should be like different core has separate PLLs while one chip has only one voltage regulator module. So as long as the cpufreq driver is there working, I should be able to configure it to set different cores running at different frequency. Right...

 

[hshen1]

Scaling governors are out of fashion in linux. Knowledgeable ppl are advising to use new infrastructure like pstates driver in 3.9 kernel ( but you will need to build your own).

Discussion about pstates, intel_idle etc is here:
https://plus.google.com/117091380454742934025/posts/2vEekAsG2QT

Thanks. That post is very helpful. I will take more time to read. But maybe scaling governors should be still working as long as it is in the kernel? Above all, from what I learnt, to change a frequency of one core, substantially, all I need to do is to write something into the MSR of that core to configure the PLL multiplier value

 

[hshen1]

In the OP he says he's using the 3.5 kernel. Also, the pstate governor only works with sandy bridge cpus, ivybridge will be supported in the 3.10 kernel.

Thank you. Actually I have done a lot of research about the CPU frequency scaling in Linux. This is the first time I have heard about the "pstate governor"... So maybe I should try to build 3.10 kernel by myself. But I think the old governors probably should still be supported

 

[Jodell88]

Thank you. Actually I have done a lot of research about the CPU frequency scaling in Linux. This is the first time I have heard about the "pstate governor"... So maybe I should try to build 3.10 kernel by myself. But I think the old governors probably should still be supported

I wouldn't worry about building a new kernel. It will be included in Ubuntu 13.10 so I'll just wait till then.

 

[Cerb]

Yes.What you said might be true. I am a researcher and I just need to do some research on how to control the frequency of different cores though different frequency of different cores might not be useful as you said.
But anyway, I think the hardware now should be like different core has separate PLLs while one chip has only one voltage regulator module. So as long as the cpufreq driver is there working, I should be able to configure it to set different cores running at different frequency. Right...

I think AMD's later Stars cores supported doing that, but I'm not sure about Intel (I'm not even sure exactly which AMD cores supported it, but I do know some did, as part of their CnQ implementation).

Part of your problem is that the trend is towards the CPU managing itself, with the OS only managing power very coarsely. Parts of each CPU will be running slower than max, even if set to a high frequency.

There's also a possibility, however, that you might be able to get it working with HT off. It could be that is causing it issues.

 

[JoeRambo]

I wouldn't worry about building a new kernel. It will be included in Ubuntu 13.10 so I'll just wait till then.

Kinda interesting advice for a guy who is a researching into clock setting and is not afraid to tinker with settings

My advice is opposite, go ahead and build kernel ( even on 3.9 ivy is supported with tiny change to one file as described in that thread). In the progress you will probably learn more about infrastructure of Linux kernel, and who knows maybe you will contribute back.

Waiting for ubuntu to mouth feed you is not research friendly for sure.

 

[jvroig]

I think AMD's later Stars cores supported doing that, but I'm not sure about Intel (I'm not even sure exactly which AMD cores supported it, but I do know some did, as part of their CnQ implementation).

This is true, and works for both Deneb and Thuban. I need to manually set (through a bash script I made for the purpose) all 4 or 6 cores to full-speed before running some apps (particularly, when benchmarking) because, as I found out, using the lm_sensors-based gnome sensors applet only affects 1 core out of 4 or 6, unless I duplicate the applet 4 or 6 times and manually assign each duplicate to a unique core.

So it does indeed work as the OP expected, at least on Deneb / Thuban. I have not yet had the opportunity to test the same on my Sandy Bridge-based machine. It only runs Windows 7 at the moment.

 

[hshen1]

I think AMD's later Stars cores supported doing that, but I'm not sure about Intel (I'm not even sure exactly which AMD cores supported it, but I do know some did, as part of their CnQ implementation).

Part of your problem is that the trend is towards the CPU managing itself, with the OS only managing power very coarsely. Parts of each CPU will be running slower than max, even if set to a high frequency.

There's also a possibility, however, that you might be able to get it working with HT off. It could be that is causing it issues.

Yes.Your HT suggestion sounds promising. I will look into that. I just started to learn the Intel-pstate driver yesterday from the google+ post. But still believe the old acpi driver should at least work. Probably HT causes the problem

 

[hshen1]

Yes.Your HT suggestion sounds promising. I will look into that. I just started to learn the Intel-pstate driver yesterday from the google+ post. But still believe the old acpi driver should at least work. Probably HT causes the problem

At last I have found what happened. It has nothing to do with HT.

When the cores are idle, the frequency is always minimum as long as not all frequencies are set to be a fixed value. So results are only meaningful when the CPU is loaded.

After I ran a busy loop and pin it on specific cores, I have found it looks like my CPU implements a chip-level frequency&voltage change even though I can change it differently in the sys directory. All cores' frequency will be equal to the highest frequency among all the loaded cores. For example , if I set core0's frequency to be 3G and core1's frequency to be 2G and load the two cores, all cores' frequency will be 3G.
Voltage will be the one supporting 3G(from lm-sensors).

I have also found the Intel_idle driver is pretty powerful. The idle state also seems to be chip level. Disable one core will also disable all cores on the chip. When I disabled all idle states(totally four), the power doubles.