They're (Almost) All Dirty: The State of Cheating in Android Benchmarks

[Eug]

Unbelievable

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization.

BTW, Google's Nexus line isn't "boosted" either.

P.S. I like the title of the table.

 

[ChronoReverse]

Wow. It's almost comic how sad that is.

 

[poofyhairguy]

Wow, another reason to buy Nexus.

 

[Eug]

Coincidentally, the mobile devices in our household are Apple, Motorola, and Google. Looks like I chose well.

 

[Ns1]

Wow. It's almost comic how sad that is.

this

 

[MrX8503]

Wow I didn't know it was this bad. Good on Anand for discovering this. So Apple/Motorola was handicapped in benchmarks. Unbelievable.

 

[slackerinabox]

Wow I didn't know it was this bad. Good on Anand for discovering this. So Apple/Motorola was handicapped in benchmarks. Unbelievable.

Yet the A7 still whooped everyone - kudos Apples, kudos AMERICA (google/moto)!

 

[Red Storm]

I remember going through this with video cards back in the day, what's sad is the very small performance gains they get from doing this. Personally it's never impacted any decision I've ever made regarding phones as I don't really pay attention to benchmarks. I'm glad to see that Moto and (obviously) the Nexus line don't resort to these shenanigans.

 

[cliftonite]

Pathetic.

 

[KentState]

I have never heard a single mention by an individual of benchmarks outside of Anandtech reviews or the forum. Not sure why phone manufacturers think this is a big enough deal to cheat.

 

[openwheel]

Nexus is the way to go

 

[QueBert]

What's the point of cheating if it doesn't put you ahead? The 5s out benchmarks the SG4 in just about every CPU/GPU related benchmark. Seem's stupid to bullshit benchmarks when they still don't even place your scores above your biggest competitor.

 

[cronos]

What's the point of cheating (if it doesn't put you ahead? The 5s out benchmarks the SG4 in just about every CPU/GPU related benchmark. Seem's stupid to bullshit benchmarks when they still don't even place your scores above your biggest competitor.

Because when they did it for the SG4 they had no idea how the 5S would score?

 

[Enrique2SIM]

Hi everyone,


I'm a newbie. I don't want to sound dumb, but I have one question based on the article posted...


That means that most of the manufacturers are lying about the real performance of their CPU?


Thanks in advance.

 

[Subyman]

Makes the 5S performance even more impressive. Wonder if any mainstream media outlets will run stories?

 

[PowerYoga]

I still think it's funny that we're benchmarking phones of all things.

 

[Ns1]

I still think it's funny that we're benchmarking phones of all things.

why? the note3 is more powerful than some user's desktops/laptops.

 

[ponyo]

why? the note3 is more powerful than some user's desktops/laptops.

Say what?

 

[Ns1]

Say what?

I know it's shocking, but there are users with less than 2.3ghz/3gb ram under the hood.

See: Netbook/MBA users

 

[slackerinabox]

bah

 

[Skurge]

I know it's shocking, but there are users with less than 2.3ghz/3gb ram under the hood.

See: Netbook/MBA users

It doesn't work that way.

 

[Ns1]

What are you doing on a tech forum when you're comparing a mobile soc and a desktop/laptop cpu based solely on megahertz. An intel i5/i7 (found in a mba0 is in a totally different league than the snapdragon 800 (a great mobile proceossor)

I think you'll come to find that a large portion of users on this "tech forum" are not uber hardcore tech geeks. I most certainly only saw "ghz vs ghz" & "core vs core" comparison. So please, enlighten me.


And to answer your question, I arrived at AT because I wanted to know how to overclock my celeron366 -> celeron550.

Note: I realize I'm eating foot in mouth and I don't care, educate me ATMD&G. Help me understand the difference in computing power. Preferably in laymens terms.

 

[Crono]

I still think it's funny that we're benchmarking phones of all things.

why? the note3 is more powerful than some user's desktops/laptops.

Good luck using all that power. Not only do most app developers design their programs and games for the midrange devices most of the time (unlike the PC world, last I checked there is no Crysis-level game bringing phones to their knees), but right now trying to run your phone at the max will rapidly drain your battery and/or overheat your phone.

The reason why these companies "cheat" - not that I'm justifying it, just explaining - is because their CPU cores run at lower clock speeds most of the time (which is how mobile CPUs are designed to operate) to save battery and minimize heat. So they effectively throttle for most tasks but run at max for benchmarks. Somewhere along the way one of these companies decided to do it, so now a number of them do it. It's a bit misleading and unfortunate, but it is understandable why.

In the desktop space you very well could run your CPU or GPU at max frequencies very happily for video encoding, gaming, or mining, but it's less likely to occur on the mobile side, except in bursts ("race to idle"). I'm not saying there aren't any CPU or GPU intensive apps out there, and more and more phones will in the future be capable of the same types of tasks as desktops, but right now what most users want is a fluid, smooth experience on smartphones, and software/GUI matters a whole lot more in that area when you are comparing current generation phones. There is a little benefit talking about 2.2 GHz versus 1.5 GHz or whatever comparison you can make, but it is only a little benefit even when all other things are equal. It's not as dramatic as on the desktop side when you can run at full tilt all the time if you really want to.

Benchmarks are useful for comparison to a small degree, and maybe are beneficial for shopping for a device for the sake of longevity (i.e. you want your phone to be good for 3 years or more), but we tend to give them a little too much weight as mobile enthusiasts on the whole.

 

[lopri]

I don't think this matters to individual buyers (does anyone buy a phone/tablet based on Antutu score?), but I am glad that this issue is brought up early. Hopefully reviewers' time will be better spent in the future instead of heralding useless tests such as 3DMarks.

 

[Eug]

Good luck using all that power.

Both the 5S and the 5 feel very fast for basic OS navigation and app usage. However, video export on the iPhone 5S is twice as fast as on the 5.

So yeah, even today, having more performance helps.