Anandtech's new battery life ratings

Discussion in 'Mobile Devices & Gadgets' started by Pataling, Nov 3, 2012.

  1. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    lulz @ people thinking Apple is giving AT direct orders. Nobody thinks that. A new battery life testing method? Sure, no problem. It's just... odd that they decided to go with a new method for the new iPhone review and they left out the one phone that for sure would have scored better than the iPhone no matter what methodology you use. It's not a conspiracy theory, it's a poor job of reviewing (like the time they benched an overclocked Nvidia card against a stock AMD card release).
     
  2. Pataling

    Pataling Junior Member

    Joined:
    Nov 3, 2012
    Messages:
    9
    Likes Received:
    0
    Hmmm...

    I noticed this chart in Anandtech's Galaxy Note II review:

    [​IMG]

    I checked with a friend that owns the original Galaxy Note and Anandtech's test flies in the face of his experience. I know it's anecdotal but I can't find any other online review reporting the Galaxy Note as having worse battery life than the Galaxy Nexus. I'm quite confused.
     
  3. TuxDave

    TuxDave Lifer

    Joined:
    Oct 8, 2002
    Messages:
    10,577
    Likes Received:
    1
    http://blog.gsmarena.com/samsung-galaxy-note-ii-battery-life-test-is-complete-here-are-the-results/

    The order is slightly swapped but they're both close together and both at the bottom. There's probably some margin of error but the general idea is there, that both phones are pretty terrible for constant web surfing.

    What experience did your friend have when he focused on surfing?
     
  4. Pataling

    Pataling Junior Member

    Joined:
    Nov 3, 2012
    Messages:
    9
    Likes Received:
    0
    At least 4 hrs, usually past 5 hrs. And, of course, awesome standby time due to that huge battery. Maybe they are using different browsers?

    gsmarena's arena tests are done on WiFi though. I find their ratings useful but I'm much more interested in 3G and LTE (gsmarena is based in the UK) since I use my phone most often during a commute.
     
  5. dagamer34

    dagamer34 Platinum Member

    Joined:
    Aug 15, 2005
    Messages:
    2,592
    Likes Received:
    0
    The problem is that the standard of the test are NOT how people p use their devices, it's designed to be a consistent test to compare devices against each other, not compare their results with what you experience (because that wouldn't be repeatable).

    Throwing ups chart and complaining about it does no one any good, and a test that says "I used it until it died" is unscientific. The test isn't perfect, but it's methodology allows it to be repeated, which is what's really important when devices get updates and new devices come out.
     
  6. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    Then they need to find a new method that more closely resembles real world experience. Someone not very technical will take one look at the chart and think (for example) "Wow, the Galaxy S3 has horrible battery life!" when that's actually not true. In my own experience it's had the best battery life of any phone I've owned except the Razr Maxx.
     
  7. Eug

    Eug Lifer

    Joined:
    Mar 11, 2000
    Messages:
    21,299
    Likes Received:
    46
    No, that chart says the Galaxy S III has average battery life, and it is nowhere near as good as the iPhone 5. Judging by the experiences of my colleagues, that's probably true.
     
  8. Mopetar

    Mopetar Diamond Member

    Joined:
    Jan 31, 2011
    Messages:
    3,554
    Likes Received:
    6
    Who's real world experience do we use? There are some people who barely touch the phone all day and others who will need to have anything plugged in before the day is through.

    Maybe Anand could consider adding multiple versions of the test in order to show how light vs. heavy usage, but that just makes it take longer for reviews to come out and people already complain about that.
     
  9. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    Here's some of my own anecdotal "evidence": I have an S3, and I used an iPhone 5 for two weeks since we got several of them for our supervisors and other employees. The iPhone was better on some days, but overall was not as good as the S3 has been for me.
     
  10. vshah

    vshah Lifer

    Joined:
    Sep 20, 2003
    Messages:
    18,975
    Likes Received:
    14
    it would be great if there was another battery life suite based around app usage, which is what a lot of people do on their phones vs. just web browsing.


    something like:
    unplug phone
    10 mins twitter
    10 mins facebook
    5 mins email
    turn screen off for an hour
    take a few pictures
    find a restaurant on yelp
    use navigation for 15 minutes
    play music with screen off for an hour

    /repeat

    or something like that. i know this would be possible on android with the testing tools built in to the SDK - not sure about iOS, but I suspect it would be possible as well.
     
  11. Pataling

    Pataling Junior Member

    Joined:
    Nov 3, 2012
    Messages:
    9
    Likes Received:
    0
    Add push notifications and background pooling too (email, twitter, skype, whatsapp etc). Almost everybody uses those but Anand's test doesn't seem to take that into account.

    Well to solve the light usage vs usage we would need to get a standardized usage ratings for cellphones. Compare the situation to the eink ereaders where the battery life can be measured in page turns. Then it's very easy to determine how your own usage (light = 100 pages per day, heavy = 500 pages per day) will result in battery life.

    But that's not the problem with Anand's battery life though. It just doesn't make sense:
    For the iPhone 5 Apple's own specs page claims up to 8 hrs battery life for both 3G and LTE but Anand's test says it gets 4.55 hrs on 3G and 8.19 hrs on LTE.
    For the HTC One X Anand's old test said it got 9.65 hrs on 3G and 5.517 Hrs on LTE. Anand's new test says it 4.98 hrs on 3G and 6.95 hrs on LTE.

    So after the iPhone 5 came out LTE suddenly became more efficient that 3G? I'd like to see the tests on 2.5G (Edge) as well although at its slow data rates the phones would probably be drained in an hour.

    It's also worth pointing out that the battery test doesn't match the review's real world usage:
     
    #36 Pataling, Nov 13, 2012
    Last edited: Nov 13, 2012
  12. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    And that is the big problem with AT's testing methodology. People don't care how well a phone does in a specific type of testing method that doesn't reflect real world usage, they want to know how the phone will do in typical day to day use. I simply cannot trust AT's review not because it isn't accurate, but because it has no correlation to how people actually use phones. The numbers do not make sense.
     
  13. Mopetar

    Mopetar Diamond Member

    Joined:
    Jan 31, 2011
    Messages:
    3,554
    Likes Received:
    6
    Just use AT's numbers as a scale. If a phone gets around Nexus 4 levels of battery life on the chart, and that phone can last the day with typical use, then anything that has a better battery life on the chart will probably last longer.

    Otherwise typical use is about impossible to measure and still suffers from the same problem in that someone can just claim that it's not actually representative. Also, there's no easy way to get a standardized usage rating for cell phones, it would be incredibly difficult to test many (e.g. using turn-by-turn navigation) in a repeatable fashion.

    AT's tests might not be perfect, but they provide a good way to gauge performance, especially considering that talk time is usually included as well. It's also repeatable and isn't highly specific to some arbitrary definition of normal use.
     
  14. Pataling

    Pataling Junior Member

    Joined:
    Nov 3, 2012
    Messages:
    9
    Likes Received:
    0
    If this were true then I wouldn't have created this thread. I don't trust that anything on that chart with a better battery life than the Nexus 4 will actually last longer. The iPhone 5 on LTE has the best battery life on that chart and my experience has forced me to question that result. I'm not even sure if all the devices on that chart are measure using the new test. I don't think AT's test provide a good way to gauge performance. AT's test has its own arbitrary definition of normal use which, for example, apparently doesn't include the usage of anyone who has to commute each day.

    Talk time is no longer a meaningful way to rate the total battery performance for a modern smartphone just like music playback hours wouldn't be used to rate the smartphone. Placing phone calls and listening to music are just two of many functions that the smartphone is used for each day.
     
  15. DLeRium

    DLeRium Lifer

    Joined:
    Feb 19, 2001
    Messages:
    20,102
    Likes Received:
    2
    How can you say that? All the phones were tested the same way. Just because you don't like the N4 results doesn't mean it isn't true. The fact that the phones are tested the same way means that the relative performance of the phones stand. You might not agree with the testing methodology, and you think the N4 should last 10 hours instead of 6, but if your use gets 10 hours instead of Anand's 6, then the other phones such as the SGS3, One X, iPhone 5 should also see a proportional increase. The point is the numbers are a relative performance indicator.

    It's like saying I can't get Anand's exact framerates in a GPU test, but I use Anand's GPU tests to understand that upgrading to a Radeon 7850 will give me huge benefits over my Radeon 4870.

    My iPhone 5 does ridiculously well against my SGS2 in a day of use. Granted I've only had a few days with my iPhone 5. My gf's SGS3 which seems to do better against the N4 according to Anand sits somewhere between my iPhone and SGS2 in terms of standard use. I'd say his results make sense to me.

    But Anand doesn't replicate real world daily use. He doesn't test standby battery. One thing I'd like to see him test is to use mobiledata and a certain amount of sync that's representative to a typical user. Email, Facebook, Twitter? Go for a day and see the 24 hour battery drop. I think this is where the iPhone does very well.

    Some people forget the primary purpose of a phone is to make calls. A smartphone may have changed things, but many people still make calls. I make conference calls a lot. Talk time is still a figure quoted on phones, so I think that's fine.

    None of Anand's tests really represent something you'll be doing on a daily basis. I don't see people tethering their phones til they die. I don't see people surfing consistently for 6 hours straight or whatever Anand simulates, but I think people WILL surf for maybe an hour spread through the day.

    The point is Anand's benches show a systematic way of benchmarking phones. How they perform in standby is a different story, but most likely you can use the relative performance of phones to judge how battery life will be. The iPhone in my experience has been great to me on battery, and my 3 Android phones have never performed spectacular--though each one has gotten better, so I expect the N4 to do better than my SGS2.

    I think we're going into a different discussion now. At first you were talking about the 3G vs 4G issue which I find to be the biggest issue about the new benchmarks. The fact that 3G and 4G flipped from previous benches is still weird to me. I really want an explanation of how the tests were done before and how they're done now. More importantly I want to see how the new tests are more representative of smartphone use and how the old tests are not. Because the issue for me is that the general population still believes 3G battery > 4G battery. It's not because of Anand's old reviews either. It's an industry wide accepted belief. Thus when Anand challenges the whole industry belief with a new set of numbers, I'd really do some digging.

    This is almost like when HardOCP introduced real world benchmarks or whatever the hell where your GPU tests were bottlenecked by CPU. I'm not say this is as big as a foulup as that, but it's a potentially controversial new way of testing that produces very different results.
     
    #40 DLeRium, Nov 14, 2012
    Last edited: Nov 14, 2012
  16. Pataling

    Pataling Junior Member

    Joined:
    Nov 3, 2012
    Messages:
    9
    Likes Received:
    0
    Er, I created the thread long before the N4 results came out.


    Anand says his test extends to how people use the device:
    I'm still talking about the 3G to 4G issue.
     
  17. Mopetar

    Mopetar Diamond Member

    Joined:
    Jan 31, 2011
    Messages:
    3,554
    Likes Received:
    6
    They're all the same, and for what it's worth, it doesn't claim that it's a normal usage pattern. It says it's for web browsing over 3G/LTE/WiFi. Just like the chart for talk time is just a chart measuring talk time.

    Which is why you can't create a good battery life test to measure normal usage. There are too many definitions, and it's almost impossible to construct an automated repeatable test that can measure many of the different metrics used.

    Instead we get a few tests that show web browsing (data usage) and talk time, which are some of the most frequent tasks performed on a phone. Video and audio playback times might be useful, but because audio playback would take so long to test it's probably just best to use the published figures. From those 4 tests, you can probably derive a pretty good estimate of what you could expect to get out of the device.

    The only other thing missing is app usage, but once again, that's pretty much impossible to test as there are so many variables that you can't control for it. Also, if the apps are using data there's another factor thrown in as well.
     
  18. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    Small nitpick. From the AT review:

    But it does not seem like it does. Definitely not to me and what I've experienced.

    And the Maxx is still missing. You'd think they'd consider it pretty important to have the top performer in the category on the chart. It's like leaving out the highest end Nvidia card in an FPS benchmark chart...
     
    #43 Red Storm, Nov 14, 2012
    Last edited: Nov 14, 2012
  19. TuxDave

    TuxDave Lifer

    Joined:
    Oct 8, 2002
    Messages:
    10,577
    Likes Received:
    1
    Even better would be just an app that generates a usage profile that a bunch of people can download and run. And then Anand get go average it down and generate some benchmark based on that. So you'd have to give up privacy for a week but at least you'd get something.
     
  20. Mopetar

    Mopetar Diamond Member

    Joined:
    Jan 31, 2011
    Messages:
    3,554
    Likes Received:
    6
    It's 4 hours of web browsing on 3G. That's a long time. I honestly doubt that most people spend that much time loading new web pages, even if they're using their phone a lot. Even looking at other phones using 3G the results aren't much better (usually around an extra hour tops, but much less than that usually) so it's not as though the test is somehow biased against the Nexus 4. At worst, it makes 3G look worse than LTE, but that's about it.

    And for what it's worth, AT might be right in saying that this benchmark is more indicative of normal use. It both uses data and the CPU to some degree. Most of the popular apps that people have mentioned (Twitter, Facebook, etc.) are going to require grabbing data and some CPU use as well.

    Obviously, without actual tests to back this up, it can't be determined how good of a proxy AT's benchmark is for the other similar use cases. However, 4 hours or straight use where both the CPU and cellular radio are being utilized isn't a bad result, especially compared to other phones. In an average day, if you're using your phone actively for 4 hours, that constitutes about 25% of the time you're awake. Most people probably don't use their phone that much.

    I think too many people have looked at the result, saw four hours of use time, and flipped out without actually taking it into context. 4 hours of active and steady use is a lot of time. Most people probably don't come anywhere near that so the Nexus 4 is going to be fine for their needs.

    Already been over this in the thread. Unless they have a MAXX handy for testing with the new benchmark, there's a perfectly reasonable explanation for not including it. It's probably that none of the staff at AT have one as a personal device, so outside of getting a loaner, there's no easy way for them to test it. If Motorola thinks their device would kick ass and look good, they should probably send one over so it can be included. That's what you pay your PR people for.
     
  21. Red Storm

    Red Storm Lifer

    Joined:
    Oct 2, 2005
    Messages:
    13,183
    Likes Received:
    2
    I guess I just feel it would have been common sense to include the Maxx. Don't have one? Then get one and include it or hold off on releasing this new method until you do. It kinda is their job after all...

    I still don't agree with their methodology, but I admit it's tough to create a quantitative test. I think it does need to be some kind of app that many users can run and the stats get uploaded and recorded. But the "Loading desktop web pages over and over until the battery dies" just doesn't mimic real world usage at all IMHO.
     
Loading...