Moderate CPU loads are the issue. Why power up a high amp core when you can get buy with a very low amp one?
Intel has other problems, they can never match the die size of the ARM cores due to their rather excessive baggage they lug around for legacy support. On desktop chips where neither die size nor power are that big of an issue, it doesn't matter. For mobile parts the 5 CPU core Tegra 4 devotes most of its die space to GPU cores and still comes in smaller then Intel's parts which have two functional CPU units and a dated and tiny GPU in comparison.
Intel's only hope is the one thing they are always better at then everyone else. Fabrication technology. Given enough of a lead in fab tech, the rest is just noise.