[TECH Report] As the second turns: the web digests our game testing methods

Discussion in 'Video Cards and Graphics' started by Final8ty, Jan 2, 2013.

  1. Black Octagon

    Black Octagon Golden Member

    Joined:
    Dec 10, 2012
    Messages:
    1,402
    Likes Received:
    0
    Ok good point. Carry on (and THANKS for what you are undertaking)
     
  2. VulgarDisplay

    VulgarDisplay Diamond Member

    Joined:
    Apr 3, 2009
    Messages:
    6,194
    Likes Received:
    1
    Maybe you have more posts per page than I do, but the only thing I see at the top of this page is someone trolling me.
     
  3. VulgarDisplay

    VulgarDisplay Diamond Member

    Joined:
    Apr 3, 2009
    Messages:
    6,194
    Likes Received:
    1
    This sounds really promising. It's exactly what I was talking about doing, and I'm glad someone has to skills to actually make something work.

    If you get your simulation to the point where you can adjust a slider or something to increase or decrease the latency spikes then a tester could do a bunch of blind tests and come up with an average number for when people actually see the issue.

    It's obvious to me that the goal should be completely consistent frame times from both vendors that are as low as possible, but if the actual worst case scenario before visible issues are easily apparent then both vendors have a number to shoot for and gamers win.
     
  4. Lonbjerg

    Lonbjerg Diamond Member

    Joined:
    Dec 6, 2009
    Messages:
    4,426
    Likes Received:
    0
    The "average" won't cut it...no one size fits all in human perception.
     
  5. VulgarDisplay

    VulgarDisplay Diamond Member

    Joined:
    Apr 3, 2009
    Messages:
    6,194
    Likes Received:
    1
    That's why you get an average, because one person isn't enough to draw a conclusion...

    They're looking for the point where NONE of those tested can sense anything strange in the motion. That's the point where they can say what is acceptable and what isn't.
     
  6. BrightCandle

    BrightCandle Diamond Member

    Joined:
    Mar 15, 2007
    Messages:
    4,763
    Likes Received:
    0
    Exactly this. By no one we probably mean 99.999%. The data is likely to come out as a bell curve of some description but with some odd charateristics (huge spike around 17ms if its not more noticeable to most below that). Because for a product that sells millions to only have 10's to 100's of people see a problem is probably OK. More than that and you have a problem.
     
  7. BoFox

    BoFox Senior member

    Joined:
    May 10, 2008
    Messages:
    689
    Likes Received:
    0
    Hmm, that's interesting - I always saw ABT articles popping up pretty high with some searches..

    Like if I type this in google:
    hd 7970 frame time

    ABT's article comes up at #5, after 4 links to techreport articles.

    :awe:

    Honestly, what was wrong with ABT's article?

    It didn't make HD 7970 look all that terrible - just that it performed abysmally in Hitman: Absolution - which can be clearly shown with youtube videos of this and GTX 680.

     
  8. Elfear

    Elfear VC&G Moderator
    Super Moderator

    Joined:
    May 30, 2004
    Messages:
    6,475
    Likes Received:
    4
    I don't think the issue was with that particular article. More just the general attitude at ABT and how trustworthy the reviews are.
     
  9. Lonbjerg

    Lonbjerg Diamond Member

    Joined:
    Dec 6, 2009
    Messages:
    4,426
    Likes Received:
    0
    Have you read any threads here...a lot of people have issues with microsttuer:

    [​IMG]
     
  10. Elfear

    Elfear VC&G Moderator
    Super Moderator

    Joined:
    May 30, 2004
    Messages:
    6,475
    Likes Received:
    4
    Again, I don't think the people posting about ABT in this thread were specifically calling out the ms article. Just seeing the article was from ABT was what some posters questioned. In a similar vein as seeing an article from Geforce.com or AMDZone.com.
     
  11. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    I would blind test the people who claim they see it first.
     
  12. VulgarDisplay

    VulgarDisplay Diamond Member

    Joined:
    Apr 3, 2009
    Messages:
    6,194
    Likes Received:
    1
    If done right it really wouldn't matter who was tested. They need to come up with an animation of something like pong. Simple, objects moving across the screen, and have a way to increase/control frame latencies on this simulation.

    Just tell the observer to watch the movement and once they find the movement to be no longer be perfectly smooth have them click a button or something to record the exact amount of frame latency that has been introduced.

    It would actually probably be better to test people that have no idea what frame latency is whatsoever.
     
  13. Rikard

    Rikard Senior member

    Joined:
    Apr 25, 2012
    Messages:
    428
    Likes Received:
    0
    We need all types of people. Those who claim that they are ultra sensitive, average gamers, and average Joe as control group. It is also important that the degree of stuttering is not gradually changing but come in random order. Otherwise it is too easy to guess right.
     
  14. Rikard

    Rikard Senior member

    Joined:
    Apr 25, 2012
    Messages:
    428
    Likes Received:
    0
    Isn't the typical RMA rate at percent level for computer components already? It is hard to claim that it is serious if it is much smaller than that.
     
  15. Lonbjerg

    Lonbjerg Diamond Member

    Joined:
    Dec 6, 2009
    Messages:
    4,426
    Likes Received:
    0

    Yeah, because no way we could see the fluctuations that the data shows.

    Funnny thing:

    This was first reported by people...before any data showed what was up.

    This is really low damage control :|
     
  16. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    The question was asked, "Who would you test?" It makes sense to test people who say that they can see stuttering on cards with these recorded latency numbers. It would make no sense to test people who say they can't see a difference. Joe Blow off of the street might not have a clue what he/she is looking for. Who would you suggest we test?

    Do you have a problem with sitting people down at gaming rigs with 7950's and 660ti's who say that they can see a difference and compare the 2 for smoothness without knowing which card they are using?

    Why don't you explain where my logic is faulty, rather than making accusations of my motives?
     
  17. Lonbjerg

    Lonbjerg Diamond Member

    Joined:
    Dec 6, 2009
    Messages:
    4,426
    Likes Received:
    0
    Your sentecne about people "claiming" to see it was loaded.

    I see it...it's not a claim, it's a fact.
     
  18. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    The whole point is to verify that it's visible during normal gameplay. Until it's verified it's not a fact.
     
  19. Lonbjerg

    Lonbjerg Diamond Member

    Joined:
    Dec 6, 2009
    Messages:
    4,426
    Likes Received:
    0
    Still in denail eh...
     
  20. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    I'm not denying anything. I don't know what you're talking about.

    Here's the scenario for you. If you did a double blind test like I described and can tell a difference then it proves there's a visible flaw in the reproduction. No problem, as I never said there wasn't one. Just that I'd like it tested under real world conditions without any chance of bias (which as humans we all have) dictating the outcome. Of course, if you couldn't tell which card was which with any statistical significance, then you are wrong. You are the only one between us who has predicted an outcome. It's no wonder you don't like the idea.
     
  21. Keysplayr

    Keysplayr Elite Member

    Joined:
    Jan 16, 2003
    Messages:
    21,120
    Likes Received:
    22
    So if the people used in the double blind test are people who cannot for whatever reason perceive microstutter or hitchy performance, than it doesn't exist?

    You keep saying: "I never said there wasn't one."
    But you won't say: "There is one"
     
  22. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    I'm saying to use people who say they can see it. Use more than one person. The more the better. It would be dumb to use people who say they can't see it.

    As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other? Just benchmarks don't visually demonstrate anything. Neither does videos in slo-mo. Real world testing will demonstrate if it can be seen in regular gameplay. Testing with multiple people, multiple games, different settings, etc... The more thoroughly it's tested the better we can understand it.
     
  23. Keysplayr

    Keysplayr Elite Member

    Joined:
    Jan 16, 2003
    Messages:
    21,120
    Likes Received:
    22
    Slo Mo is actually very good in seeing if you are getting what you expect. The Slo Mo was done by Scott Wasson, presumably because he got pummeled with complaints from those who could not see the stutter in real time. He did them a favor, although some didn't think so. Smooth gameplay and a real representation of the fps you are said to be getting in any given game. If you are told that you'll get a peak framerate in Hitman Absolution or Skyrim or Crysis2 and you buy a card on that premise (after looking at pure fps benches and reviews), but what you really get is a really uneven delivery of frames, then you aren't really getting what you paid for.

    Lastly: "As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other?"
    If AMD acknowledged the problem, I don't see any reason for you to deny the problem exists. AMD thinks it has been shown to be a problem. It's up to you now. Continue the pointless battle or appreciate that AMD is working to rectify the problem. Your call.
     
    #673 Keysplayr, Jan 24, 2013
    Last edited: Jan 24, 2013
  24. 3DVagabond

    3DVagabond Lifer

    Joined:
    Aug 10, 2009
    Messages:
    11,730
    Likes Received:
    2
    If it has to be shown in slo-mo to be seen then what does it matter?

    Nobody, not me, AMD, or anyone else is saying that the measured frame latencies don't exist. Of course AMD is going to be concerned. It's a benchmark that shows their card in a negative light compared to the competition. Please link to AMD saying there's a visible stuttering problem. They have said they are addressing the frame latencies, not stuttering. At least that's all I have seen. Maybe I missed it?
     
  25. Keysplayr

    Keysplayr Elite Member

    Joined:
    Jan 16, 2003
    Messages:
    21,120
    Likes Received:
    22
    Look, I can see why you chose to take this line in the debate. If done well, it would provide a certain level of deniability of the issue if it can't be seen. Unfortunatly for you, you're not doing it well.
    You've just claimed that because AMD did not claim to see it, it isn't there?
    Your position is quite weak my friend. I don't know why you continue with this line. All I know is that you have been one busy beaver the last week.
     
Loading...