AnTuTu and Intel

Discussion in 'CPUs and Overclocking' started by Exophase, Jul 10, 2013.

  1. Idontcare

    Idontcare Elite Member

    Joined:
    Oct 10, 1999
    Messages:
    21,130
    Likes Received:
    10
    110% agree, especially with the bolded part.

    I remember well when SUN broke the SPEC benchmark by some compiler optimization that improved scores in just one test out of the suite by something like 10x (or was it even more).

    So they scored the same in 14 out of 15 tests, but in that one test their score went from something like 8.7 to an astonishing 93 (IIRC), which then pulled the weighted average in such a way that it seemed like they were suddenly kicking Power6's ass in Spec.

    Completely broke the purpose of the benchmark, and the compiler trick wasn't really applicable to real world software.

    Everyone knew it, the subtest results were there for all to see...but that didn't stop SUN marketing from hyping and grandstaging their uber processor for its spec scores :rolleyes:

    (and yes, the irony never escaped me that while we were SUN's foundry, I knew their chips were crappy and delivered bottom-tier performance, the Atom's of the big-iron world :D)
     
  2. BLaber

    BLaber Member

    Joined:
    Jun 23, 2008
    Messages:
    184
    Likes Received:
    0
    Optimizing for games & apps that people buy systems to run on makes sense, its worth the effort , but whats going on here is not even remotely comparable to games & apps optimization.
     
  3. Abwx

    Abwx Diamond Member

    Joined:
    Apr 2, 2011
    Messages:
    8,343
    Likes Received:
    194
  4. beginner99

    beginner99 Diamond Member

    Joined:
    Jun 2, 2009
    Messages:
    3,513
    Likes Received:
    229
    I agree that if a compiler gets optimized for s specific benchmark it's cheating and idiotic. However the thing with the JIT is ARMs problem and they must solve it. Because if the like it or not but the software is part of the whole stack and if your CPU and ISA rocks and is 10x times faster if properly optimized for it's still useless if no software does it or your your compiler sucks. I guess you get the point.

    If not defending intel but on the other hand you can't always say Intel CPU is only faster due to the better compiler. Well, your free to make a compiler for your CPU if it is from scratch or by contributing to GCC.
     
  5. galego

    galego Golden Member

    Joined:
    Apr 10, 2013
    Messages:
    1,099
    Likes Received:
    0
    I bolded relevant parts. This Intel engineer claims that the Geekbench developer agrees. Therefore, it all of this is right, this would be a case where the benchmark needs to be adjusted/improved, not one where the benchmark has been deliberately cheated to cripple competence. Nobody cheats a benchmark and then admits that did. It is stupid.

    But history shows that OEM have been fooled before. From the Intel-AMD FTC complaint (verified in the settlement):

    I find reasonable that some mobile OEMs could be cheated again.

    I am one of those complaining. Yes, those optimizations are improvements, but you are ignoring the true problem.

    The true problem is that optimization is made for some few games (for instance half dozen of them) and then, curiously, those few good-performing games are used in reviews ever and ever. The final user gets the false impression of that is the general performance of the hardware, when he is not aware that that performance is not achievable with the 99% of games for which no improvement was made in the driver.
     
    #30 galego, Jul 11, 2013
    Last edited: Jul 11, 2013
  6. SlimFan

    SlimFan Member

    Joined:
    Jul 5, 2013
    Messages:
    77
    Likes Received:
    0
    It's not clear to me at all that phone OEMs know anything about performance, benchmarks, or what's important to end-user experience. Quad core phones are pretty much useless in today's market with today's workloads. The only things that the additional cores make faster are the benchmarks that we're talking about here.

    ARM has been selling Dhrystone performance for a very long time, and DMIPS/MHz, DMIPS/mW, and total DMIPS score (reached by multiplying by the number of cores, even though it's a single threaded benchmark). This is what much of the ARM ecosystem has been using to make decisions. These parts typically went into a closed system (before App stores) that made it almost impossible to run 3rd party software to figure out how fast they were. This was all wonderful, because nobody really cared. As long as the preloaded software ran at an acceptable performance level, all was well.

    In today's market, you have app stores where arbitrary code is now run on the products. Suddenly you can download new code that may or may not run very well. The inherent performance of the product is now visible to the end user in a brand new way.

    This is now a new world for the OEMs. I wouldn't assume that they are that much more evolved about performance, benchmarks, and what portions of the platform matter to the end user experience. This is not the PC/server space where you've had Intel, AMD, IBM, Sun, etc., fighting over benchmarks, multiple versions of SPEC, and virtually limitless software that can be run on each and every platform.

    Now the only thing that anyone can run are these toy benchmarks with crazy results. End users and reviewers run them, and this is now publicly seen by people looking to make a purchase. An OEM that ignored these would be taking a big risk and either assuming that their customers are smarter than the reviewers, or that none of their customers would ever read these reviews.
     
  7. mrmt

    mrmt Diamond Member

    Joined:
    Aug 18, 2012
    Messages:
    3,976
    Likes Received:
    0

    You are not quoting the settlement, you are quoting the complaint.
     
  8. beginner99

    beginner99 Diamond Member

    Joined:
    Jun 2, 2009
    Messages:
    3,513
    Likes Received:
    229
    lol. epic fail. but what else is to be expected from current troll number one.
     
  9. AnandThenMan

    AnandThenMan Diamond Member

    Joined:
    Nov 11, 2004
    Messages:
    3,791
    Likes Received:
    60
    I just don't think the phone makers give a hairy rats *** about a bench like this, and neither does the target buyer. The form factor, screen, "must have" appeal etc. etc. are much higher on the list. If you ask just about any smart phone owner, hey did you see the latest AnTuTu bench? Your next phone has to be the one that does the best on it, they will not understand a word you're saying let alone even begin to care.
     
  10. mrmt

    mrmt Diamond Member

    Joined:
    Aug 18, 2012
    Messages:
    3,976
    Likes Received:
    0
    How much would it cost to AMD/ARM to develop their own compiler/benchmarks and why didn't do this before?
     
  11. sontin

    sontin Diamond Member

    Joined:
    Sep 12, 2011
    Messages:
    3,054
    Likes Received:
    17
    I guess nobody takes a benchmark from an IHV serious...
     
  12. simboss

    simboss Member

    Joined:
    Jan 4, 2013
    Messages:
    46
    Likes Received:
    0
    ARM and AMD do develop their own compilers :rolleyes:

    The trick (well, one of them) from AnTuTu and/or Intel is that they use a compiler that no one else can or want to use on Android, whereas the other benchmarks use GCC, which is the officially supported compiler for android for ARM and x86.

    As for benchmark, how much credibility would you give to a benchmark developed by ARM, AMD or Intel? :hmm:
    If they were open source, it would still be vaguely reliable, but closed-source benchmark would be pretty much useless.
    I have even done one myself ;)

    Code:
    If (my_arch) score = 100000000000; 
    else score = 1; // let's not give them 0
    
     
  13. AnandThenMan

    AnandThenMan Diamond Member

    Joined:
    Nov 11, 2004
    Messages:
    3,791
    Likes Received:
    60
  14. Genx87

    Genx87 Lifer

    Joined:
    Apr 8, 2002
    Messages:
    40,128
    Likes Received:
    147
    Oh you werent around on the video card forum about a decade ago then? :D Any optimization was considered cheating. I agree with you if the optimization can be used in real world applications then it is legit. But if it is a trick that only works within the benchmark then it is crap and misrepresenting the capabilities of the processor.
     
  15. AnandThenMan

    AnandThenMan Diamond Member

    Joined:
    Nov 11, 2004
    Messages:
    3,791
    Likes Received:
    60
    If there a more slippery slope than benchmarking? Probably not. The responsibility to keep things as fair as possible ends up falling on the review sites, they are the last line of defense against vendors trying to cheat their way to the top. And you can be sure if the respective vendors can, they will.
     
  16. SlimFan

    SlimFan Member

    Joined:
    Jul 5, 2013
    Messages:
    77
    Likes Received:
    0
    I think you're right that these benchmarks shouldn't be involved in any phone makers or phone buyer's decision process. But I think you're wrong in assuming they don't. Review sights haven't really caught onto the worthlessness of the benchmarks. No, nobody says "did you see the latest AnTuTu score" but if a phone comes out with bad scores relative to others people say "yeah, that phone sucks."

    Why else is there a phone SOC arms race? Because nobody cares? How do you think OEMs measure who's "winning" the race?
     
  17. Nothingness

    Nothingness Golden Member

    Joined:
    Jul 3, 2013
    Messages:
    1,667
    Likes Received:
    111
    Did you miss the ABI Research report or the recent leak of Bay Trail AnTuTu score? :)
     
  18. krumme

    krumme Diamond Member

    Joined:
    Oct 9, 2009
    Messages:
    5,193
    Likes Received:
    961
    Intel is acting like they did in the old days fighting amd. The difference this time is they are not the big gorilla but a small player, and most sites for phones is not dependant on info and good relationship with Intel.

    Its not like a Intel power engineer just happen to come by gsmarena or t3 with a voltmeter, sunspider and ohms law. What do they care? They just need the newest pink fast and will eat the pr from Samsung or Apple.

    The end result might even backfire and the result be some arm bend benchmarks dominating not showing the potential of the new Atom core. After all its a little in apple and samsung interest to show a few of the customers that have already bougt their phones, it was the best choice, as nobody can tell the difference anyway.

    Intels best bet to pull this antutu is after all the oem have more important problems to fight. Like what looks more and more like a stagnating high end phone market.
     
  19. Kidster3001

    Kidster3001 Junior Member

    Joined:
    Jul 11, 2013
    Messages:
    1
    Likes Received:
    0
    While I agree that most benchmarks favor one platform or another, I think the OP is missing the point about compilers.

    Compilers aren't there to convert your source code directly into machine code in the same sequence so that the same steps run in the same order on all platforms. The whole idea behind compilers is to create the most efficient code possible for the target platform. If a given compiler can figure out how to make work easier, for example if you are dealing with loops and constants then good for the compiler.

    If you create and initialize an array would you prefer your compiler to use a loop to set the contents or would you prefer your compiler to call memset() in libc? The second compiler here is going to generate code that performs much faster.

    Why not use the compiler that generates the best code for each platform? If you don't then you're really just benchmarking the compiler.
     
  20. sefsefsefsef

    sefsefsefsef Senior member

    Joined:
    Jun 21, 2007
    Messages:
    219
    Likes Received:
    0
    This is actually OP's complaint. AnTuTu on Intel is just benchmarking ICC, not AnTuTu.
     
  21. jfpoole

    jfpoole Member

    Joined:
    Jul 11, 2013
    Messages:
    43
    Likes Received:
    0
    John from Primate Labs here (the company behind Geekbench).

    I wanted to provide some details about what's going on with the floating point workloads the Silvermont architect referenced. Two of the Geekbench 2 floating point workloads (Sharpen Image and Blur Image) have a fencepost error. This error causes the workloads to read uninitialized memory, which can contain denorms (depending on the platform). This causes a massive drop in performance, and isn't representative of real-world performance.

    We only found out about this issue a couple of months ago. Given that Geekbench 3 will be out in August, and fixing the issue in Geekbench 2 would break the ability to compare Geekbench 2 scores, we made the call not to fix the issue in Geekbench 2.

    If you've got any questions about this (or about anything Geekbench) please let me know and I'd be happy to answer them. My email address is john at primatelabs dot com if you'd prefer to get in touch that way.
     
  22. KompuKare

    KompuKare Senior member

    Joined:
    Jul 28, 2009
    Messages:
    573
    Likes Received:
    67
    That, plus as Exophase said in the OP the kind of optimizations which ICC has done to AnTuTu is not something which they can do without knowledge of the benchmark. In other words, Intel seems to specifically targeting this benchmark to look good. Seems like a lot of trouble but Intel do have past form for this kind of thing. So yes, the Intel compiler is genuinely a very good compiler but a certain percentage of Intel compiler budget seems to be set aside for these kind of shenanigans.

     
  23. TuxDave

    TuxDave Lifer

    Joined:
    Oct 8, 2002
    Messages:
    10,577
    Likes Received:
    3
    Nice of you to chime in. Thanks for your comments. :)
     
  24. Nothingness

    Nothingness Golden Member

    Joined:
    Jul 3, 2013
    Messages:
    1,667
    Likes Received:
    111
    You'll be happy then to know that recent versions of gcc just do that to one part of the Stream benchmarks: it's changed into a call to memcpy.

    The problem here is that the code that icc transforms into a form of memset certainly doesn't look like a memset. The transformation is really impressive and is probably useless outside of that particular loop in that particular benchmark.

    Exophase also mentions the optimization in icc only happened recently and conveniently just before AnTuTu v3 release. Call me paranoid if you want...

    My experience with icc is the same as many people I know: if your code can't be vectorized icc isn't significantly faster than gcc, if at all. Ah well except if your code looks like a benchmark for which Intel already has tweaked icc :rolleyes:
     
  25. StrangerGuy

    StrangerGuy Diamond Member

    Joined:
    May 9, 2004
    Messages:
    8,362
    Likes Received:
    78
    I'm sure ARM licensees are going to bend over their asses to Intel over doctored benchmarks so they can join the line of Intel slaves like Asus to suffer stuff like the RDRAM fiasco.