That argument falls apart when you have more cores and a fixed power budget (I am making the assumption that most computers won't keep allowing more CPU power forever as we add more cores). To add more cores in the same power budget, the big cores no longer are efficient. If there was no power budget, then yes the big cores are better and we should have all big cores. See the right side of the image below.
But, and this is the point that I am trying to emphasize, you cannot add more cores and just ignore the power consumed. To stay within power budgets as you add more cores, you have to start turning down the frequencies. Soon the little cores are actually more efficient (right side of the image below).
View attachment 85842
I could post more images, this is just a representative one. There is a crossing point. As more cores are added, we cross over to the little cores doing more work per joule.