They're (Almost) All Dirty: The State of Cheating in Android Benchmarks

[golem]

Could very well be.

The smartphone market has changed very quickly from something just above a featurephone to the small portable computers we have today.

Its hard to pin down a usage pattern to test really.

At the moment (and I've said this for awhile) its best just to look at the SOC and use a bit of common sense as to what you should expect.

Smartphones seem to be making good moves in all the areas we need. Raw performance is moving forwards, battery life is better than ever, IO performance seems to be the new thing.

All in all times are pretty good.


Oh and thanks for a very civil, enjoyable discussion. :thumbsup:

Thank you also!

 

[Mopetar]

I thought Apple people didn't care for benchmarks

Some don't, but I imagine that a few do. A benchmark result on its own isn't necessarily useful, but it can give a rough idea of performance gains. Faster javascript and web benchmarks mean that the browsing experience will be better and pages will load more quickly, so it's worth caring about.

Graphics benchmarks mean that games can run more smoothly and use better graphics, so for anyone who likes playing cutting edge games on their phone, those benchmarks are important.

The benchmark numbers aren't necessarily useful, but eventually the performance gains enable features like the burst mode for the camera that can shoot 10 8 MP shots every second and then do some post processing to pick out which images are likely the best.

Most phone users don't care at all about some benchmark number, but it's those numbers that enable the kinds of features that people do actually care about. Developers probably care more, just because they give them some idea about the limitations of the hardware that they're working with. If you're trying to make a 3D game, you probably care a lot about the graphics benchmarks.

 

[Puddle Jumper]

I thought Apple people didn't care for benchmarks

They care a lot about any benchmarks Apple happens to win, otherwise not so much.

 

[MrX8503]

They care a lot about any benchmarks Apple happens to win, otherwise not so much.

So the reverse isn't true? Lol. How about people
going as far as defending the OEMs for cheating on benchmarks?

The lesson is real world performance is more important.

 

[Eug]

So the reverse isn't true? Lol. How about people
going as far as defending the OEMs for cheating on benchmarks?

Indeed. I'm surprised that anyone at AT would defend this behaviour, but nonetheless there have been people here that have.

 

[openwheel]

They care a lot about any benchmarks Apple happens to win, otherwise not so much.

That's for sure.

 

[annomander]

That's for sure.

And android fans arn't the same?

Its laughable, last few years its all being about cores and mhz of the chips, as soon as they're losing its suddenly cpu;s don't matter, lol

 

[shady28]

Hi everyone,


I'm a newbie. I don't want to sound dumb, but I have one question based on the article posted...


That means that most of the manufacturers are lying about the real performance of their CPU?


Thanks in advance.

Correct, with the notable exceptions of Motorola and Apple. Also Google branded 'Nexus' phones, but the manufacturers of those devices lie about the performance of their other devices.

So basically, the only 3 honest companies in the phone markets are Google, Apple, and Motorola. Google happens to own Motorola.

I suggest people vote with their checkbook.

 

[Mart_L]

You guys are idiots, this is just anti android propaganda from anandtech and pure sensationalist "journalism"

The article is pathetic, I for one thank the Android manufactures for doing this because it saves me having to force the performance governor myself when I run these ridiculously short benchmarks.

 

[MrX8503]

You guys are idiots, this is just anti android propaganda from anandtech and pure sensationalist "journalism"

The article is pathetic, I for one thank the Android manufactures for doing this because it saves me having to force the performance governor myself when I run these ridiculously short benchmarks.

Anandtech wasn't the first to report boosted benchmarks. Also, WOW, lol.

 

[annomander]

I am shocked and astounded by a lot of people on her, at the vitriol sprouted towards Anad about this issue, really a lot should be ashamed of themselves.

I mean on the front page, are ddriver and beautyspin and others actually paid by samsung due to the amount off effort they are going to badmouth the sit?

Anand is better then me, I'd have IP blocked them.

 

[bearxor]

I thought Apple people didn't care for benchmarks

I find them fascinating from a purely theoretical point of view. Especially now that Apple designs their own processors.

However, the fundamental differences between Android and iOS are so significant that comparing SoC speed across platforms should be pretty far down on the list if people are trying to choose platforms.

They do make for a good device comparisons on Android. Perhaps the manufacturers should work on making their software more efficient at waking and sleeping the processor as opposed to detecting an app and running the processor full throttle.

 

[thedosbox]

I am shocked and astounded by a lot of people on her, at the vitriol sprouted towards Anad about this issue, really a lot should be ashamed of themselves.

I mean on the front page, are ddriver and beautyspin and others actually paid by samsung due to the amount off effort they are going to badmouth the sit?

Anand is better then me, I'd have IP blocked them.

There are always going to be people who think that they need to defend a company whose products they've bought - post purchase rationalization and all that. It applies just as much to android, apple, PC hardware (see Intel vs AMD, AMD vs nvidia), games etc etc. It shouldn't be surprising.

 

[WelshBloke]

...
They do make for a good device comparisons on Android. Perhaps the manufacturers should work on making their software more efficient at waking and sleeping the processor as opposed to detecting an app and running the processor full throttle.

I thought that that was the problem.

Being serious though, I don't really see a lot of point in benchmarking different devices that use the same SOC and OS (talking benchmarks that test the SOC here). There really shouldn't be a huge difference and if there is it's indicative of a problem rather than potential performance.
About the only place it was potentially interesting was between exynos and non exynos versions of the same phone but Samsung were able to make them perform exactly the same so that killed that.

 

[bearxor]

I thought that that was the problem.

Being serious though, I don't really see a lot of point in benchmarking different devices that use the same SOC and OS (talking benchmarks that test the SOC here). There really shouldn't be a huge difference and if there is it's indicative of a problem rather than potential performance.
About the only place it was potentially interesting was between exynos and non exynos versions of the same phone but Samsung were able to make them perform exactly the same so that killed that.

You're exactly right. That's the reason to bench them. And to see if the different manufacturers have deficiencies in their software that makes the phone not perform as well as it should. So there is still good reason even of the SoC is the same.

But cross platform benching means almost nothing outside of a theoretical discussion. I can find it fascinating that a 1.3ghz Apple chip can go toe to toe with a 2.3ghz Qualcomm chip all day long. But that means almost nothing in the decision to choose one OS platform over another.

 

[WelshBloke]

You're exactly right. That's the reason to bench them. And to see if the different manufacturers have deficiencies in their software that makes the phone not perform as well as it should. So there is still good reason even of the SoC is the same.

The software is pretty similar across vendors, even the GE phones benchmark similar to the non GE ones. Point I was making is you'd basically be bug testing rather than performance testing.

But cross platform benching means almost nothing outside of a theoretical discussion. I can find it fascinating that a 1.3ghz Apple chip can go toe to toe with a 2.3ghz Qualcomm chip all day long. But that means almost nothing in the decision to choose one OS platform over another.

It would be more interesting if the benchmarks were more aimed at a true workload rather than spoolup-shortworkload-spooldown. If it was all day long I'd agree with you!

 

[bearxor]

It would be more interesting if the benchmarks were more aimed at a true workload rather than spoolup-shortworkload-spooldown. If it was all day long I'd agree with you!

Read that as: I can be fascinated by it all day long.

 

[WelshBloke]

Read that as: I can be fascinated by it all day long.

Yeah I know but it tied in with my main problem with these benchmarks so I thought I'd coopt it.

 

[lothar]

The software is pretty similar across vendors, even the GE phones benchmark similar to the non GE ones. Point I was making is you'd basically be bug testing rather than performance testing.

No.

 

[slackerinabox]

Wow, defending overclocking cheaters - who are the sheep now? And why deny that the a7 is the faster mobile processor out right now? It won't be in 6 months and the cycle resets, it's how it's always been! Why deny that by claiming the greatly designed A7 is somehow inferior and all marketing, it just makes you look ignorant since tech journalists and cpu super nerds all agree that it's great tech (which will be superceded by samsung in a few months, I'm sure).