HardOCP says anandtech has bad methods

[Gary Key]

Originally posted by: Citizen86

Running the other resolutions now but this is amazing as there are no differences to speak of with fraps running.

Yes I just registered today, but I think you guys are misunderstanding a little bit. The HardOCP results are not simply the difference between FRAPS and what the Benchmark says, it's the difference between running the demo in REALTIME and running it as a Timedemo. You guys should read, because it's explained on the same page you you link to:

The ?Real Time Timedemo FRAPS? data you see is gleaned from running the canned GPU timedemo in real time, and recording the framerate with FRAPS. The ?Traditional Timedemo Benchmark? results are as you might expect from running in timedemo mode where the recorded demo runs as fast as it can till completion then gives you your benchmark scores.

So to put it simply, one is the canned GPU demo run real time and the other is the demo run in timedemo benchmark mode.

Now what you will immediately notice is that the two sets of results using the Crysis canned GPU demo are not even close to the same. Simply running the timedemo as a traditional ?timedemo benchmark? gives us a 38% increase in average framerate over running the canned demo at real time speed using the 3870 X2. Average framerate increased 38% going from a real time canned demo to a traditional ?fast as it can draw it? timedemo benchmark. Same demo, same settings, same hardware, same driver.

Am I missing something? I could be completely wrong, but I believe it's all explained right there.

What is "real time" playback when utilizing Crysis, using the demo or play console commands results in the same numbers being generated by FRAPS or the included Benchmark_GPU test (averaging the four test runs). Trying to actually recreate the GPU benchmark in "real-time" is impossible, the variability between test runs is totally outside any scope of repeatability.

We can spend an extra five seconds during the forest overfly or simply point the weapon up an additional 10 degrees and get FRAPS results higher than the GPU benchmark, lower the weapon and spend additional time near the housing complex and it goes the other way.

The simple fact is that it is impossible to ensure repeatability between test runs using that methodology. Not going to debate the subject matter any further as more than enough has been stated already. However, sometimes it is nice to understand the definition of what a term like "real-time" actually means in the grand scheme of things.

 

[xxceler8]

Real world benchmarks to measure real world gaming. Who would have thought. It is troubling why so many here attack Kyle personally in lieu of offering a perspective. I think he makes some great points especially the one about doctored benchmarks in timedemos.

 

[lopri]

Originally posted by: xxceler8
Real world benchmarks to measure real world gaming. Who would have thought. It is troubling why so many here attack Kyle personally in lieu of offering a perspective. I think he makes some great points especially the one about doctored benchmarks in timedemos.

The problem is, while performing his 'real-world' game play for his readers, he tries to attack and denounce his competitions. He can surely 'evaluate' video cards in his style and just leave it at that. But of course that's not what he aims. He wants to be 'da man' and wants to justify the way his site has been going over the past few years. At one point the site was nothing more than BFG PR & customer-care center. He appears as a very typical 'red neck' in forums and didn't hesitate to swing ban sticks or use derogatory words towards anyone who objected his viewpoints.

In this silly occasion, he had to dig up a forum post (not an official article/review, or even a blog piece, but a freaking forum post) just to draw attention on the net. It speaks exactly what type of person he is.

In a nutshell, he can 'evaluate' or cook and eat video cards at his hearts content and try to appeal to his readers. I don't think anyone will have a say about it. The trouble begins when he starts finger-pointing others, not knowing how small his brain is. And that's what you're seeing here.

Originally posted by: Gary Key
What is "real time" playback when utilizing Crysis, using the demo or play console commands results in the same numbers being generated by FRAPS or the included Benchmark_GPU test (averaging the four test runs).

Thank you. I feel vindicated. Crysis time demo is designed to run as fast as a system allows and I don't even know what they mean by 'we ran it real time', other than using Fraps to count the fps. I guess devs can make a 'real-time' demo that runs at certain speed and let hardware render as many frames possible underneath.. but.. does that sound familiar? Yeah then we end up with another 'canned' benchmark, a la 3DMarks.

I would love to hear some explanation on this, but I'm pessimistic that we'll ever learn about the mystical way of running 'real-time' Crysis time demo. :laugh:

 

[jjyiz28]

Originally posted by: UltimaBoB
Obviously, objective benchmarking trumps subjective benchmarking *if it is possible*.

But if the tools don't exist to objectively benchmark actual gameplay, and if that which can be scientifically benched is being manipulated, then inherently subjective benchmarks - done as "objectively" as you can - may be the best you can hope for.

QFT. I agree.