HardOCP says anandtech has bad methods

apoppin

Lifer
Mar 9, 2000
34,890
1
0
alienbabeltech.com
Originally posted by: clickynext
Originally posted by: apoppin
thanks ... and welcome

i sent a PM to Derek ...


i sense a war brewing ... :)

i do have to agree with their intro, however:

HardOCP has been thoroughly mired ...

that is the correct term

:p
who's derek?

Derek Wilson, the one mentioned by HardOCP ... he is online ... i hope he catches it

he is quoted in the first few lines of the article:

Derek Wilson, Anandtech ? 01/28/08: ? I'll still stand by the fact that it is not necessary to look at gameplay situations in order to build an accurate picture of the relative performance of a graphics card.?
 

DerekWilson

Platinum Member
Feb 10, 2003
2,921
5
0
*sigh... thanks for the PM, but I don't think I could have avoided what amounts to an industry wide call out like that one.
 

ViRGE

Elite Member, Moderator Emeritus
Oct 9, 1999
31,516
167
106
It's basically a half-truth mixed with FUD. It's true, a timedemo isn't always going to produce results that are the same as playing a game (you try coming up with a timedemo that exactly replicates how every person will possibly play the game) but I also think it's the more scientific manner to do such testing. With timedemos you get exactly the same situation every single time, the only variable changed is the video card; with manual control in spite of your best efforts you're not getting the same test. And while it's acceptable given no other option (i.e. Oblivion), I find it more acceptable to stick to timedemos so that you are for sure only changing the video card as the variable.

Picking a fight with AnandTech specifically probably isn't the smartest thing either. It's one thing to do an article about this kind of stuff to explain why you do something the way you do it, but dragging in someone else is at the very least petty.
 

apoppin

Lifer
Mar 9, 2000
34,890
1
0
alienbabeltech.com
Originally posted by: DerekWilson
*sigh... thanks for the PM, but I don't think I could have avoided what amounts to an industry wide call out like that one.

i just wanted you to see it while it was still smoldering ;)


i would guess the OP is from HardOCP who brings the very latest

--it IS a call out and we are all making popcorn
 

jstatham102

Junior Member
Feb 11, 2008
2
0
0
actually im from neither. i read both sites, but i found this article to be interesting, and noted the call out made
 

ArchAngel777

Diamond Member
Dec 24, 2000
5,223
61
91
I'd just ignore the guy. HardOCP has the worst benchmarking method I have ever seen. Trying to make sense of their highly subjective benchmarks will just give you a migraine. Not with your time Derek.
 

apoppin

Lifer
Mar 9, 2000
34,890
1
0
alienbabeltech.com
the more i think about it ... *we* did this ,,, :p

there is no doubt that we have taken a lot of heat from various sources. Most of it is generated through ignorance and the human quality of being resistant to change. Some of it is generated by websites and forums that will proclaim that there is no need for tools beyond a timedemo or cutscene in order to understand graphics cards? performance.

Do you remember a certain current HardOCP member that we alll flamed for his defense of RW testing [as well as other things]? And how we simply dismissed out of hand their testing as "subjective" and even "biased"?

Well ... that is their response. mostly to us ... as they named our chief

well if it's war they want ... i'm all for it ... i can only expand on what i know ... i'd LOVE to question Kyle and get to REPLICATE some of his benches ...

he might even get a genuine apology from me
:Q

or not :p

BUT ... BEST of all *timing* is incredible ... i AM benchmarking and i'd love to ADD some of his methods ... as long as they can be scientifically set up and replicated on other rigs as in traditional benchmarking.

==========

Originally posted by: jstatham102
actually im from neither. i read both sites, but i found this article to be interesting, and noted the call out made

At ANY rate you are welcome here and thanks for the news
 

tuteja1986

Diamond Member
Jun 1, 2005
3,676
0
0
Hardocp are fools :! they had war with many tech website :!

Firingsquad fought hard and beat the f out of kyle :! FS made kyle look like a fool :!
 

DerekWilson

Platinum Member
Feb 10, 2003
2,921
5
0
Originally posted by: apoppin
the more i think about it ... *we* did this ,,, :p

there is no doubt that we have taken a lot of heat from various sources. Most of it is generated through ignorance and the human quality of being resistant to change. Some of it is generated by websites and forums that will proclaim that there is no need for tools beyond a timedemo or cutscene in order to understand graphics cards? performance.

Do you remember a certain current HardOCP member that we alll flamed for his defense of RW testing [as well as other things]? And how we simply dismissed out of hand their testing as "subjective" and even "biased"?

Well ... that is their response. mostly to us ... as they named our chief

you may be right ...

actually, i am sure you are right considering the quote on the first page of their article was mine from this forum, and because they specifically cite forums in their opening.

if that actually is the case, then i'm incredibly flattered.

i could understand H responding to an article i wrote... but to respond like this to our forum speaks VERY highly of all of you and how they see your influence in the community.

you should all be very proud.

I know I am.

:beer: here's to the AT Video Forum! you guys are famous!
 

Rusin

Senior member
Jun 25, 2007
573
0
0
Well in this HD3870 X2 case there was other sites that used same sort of testing methods as HardOCP and got similar results.

This for example:
http://plaza.fi/muropaketti/ar...deon-hd-3870-x2-r680,2

Well..at least they didn't test cut scenes like Anandtech did few times (This one AT editor did say that they didn't do this even when their review said they did)
 

DerekWilson

Platinum Member
Feb 10, 2003
2,921
5
0
Originally posted by: Rusin
Well..at least they didn't test cut scenes like Anandtech did few times (This one AT editor did say that they didn't do this even when their review said they did)

anand change the test while i was in the hospital and i didn't pay attention ... also, i acknowledged that i got that wrong. and it wasn't "a few times" it was one cut scene.

and cut scenes are still often very useful.
 

Zinn

Member
Sep 13, 2005
143
0
0
Originally posted by: DerekWilson:beer: here's to the AT Video Forum! you guys are famous!
Derek, you guys might be famous, but that's not such a great thing if you're famous for being wrong. I'd really be interested to see what you can say to refute the points Kyle brings up in this article.

Canned benchmarks might be repeatable, but what's the point of that when people are buying video cards to play games? When the 3870x2 beats an 8800 GTX in timedemo benchmarks, but is considerably slower in actual gameplay, what benefit are your readers getting by reading your results?

I think this is a big problem in the industry right now. AMD and NVIDIA focus too much resources on getting their numbers higher in canned benchmarks to the detriment of their performance in actual gaming applications. Kyle has shown that this is misleading to consumers, and I hope you can do better than just brushing him off.

 

Piuc2020

Golden Member
Nov 4, 2005
1,716
0
0
The thing I like about HardOCP is the maximum playable settings for each card, it gives you a more realistic image of what to expect with the card but AT's way of benchmarking is a better way of testing relative performance.
 

taltamir

Lifer
Mar 21, 2004
13,576
6
76
I don't think this evolve (devolve?) into a war. I am seeing anand taking the high road on this one (am I using the saying correctly?).

Hard made some interesting observation about how supposedly both AMD and nVidia cheat in the canned benchmarks in crysis. But that doesn't exactly refute the point that relative performance remains very similar. Which was dereks whole point, which they failed to refute.
They also compared the correctness of the "canned" benchmark to their "natural playtime" benchmark of running through a single level, but it is a well known fact that there are huge differences between performance in different levels using the same cards. For all we know had they played through the entire game rather then one level it might have matched the canned benchmark (percentage wise) more closely then their one level benchmark.

I can never really understand hard's results.. how big a difference does it make when they bump one out of ten options from medium to high on the better card to get 2.5 fps less but a "better quality picture". How much faster (if its faster), in percent, is that card then? (so that I could calculate its relative value per dollar by comparing how much more expensive it is, in percent)

I do like reading various sources though, without hardocp i wouldn't have known about the crysis benchmark cheating... And there is value to be gleaned from seeing the results of all testing methods... this means I can expect much lower performance in crysis engine based games with the GTS 512MB I just ordered... so I learned something from reading that article... :)

It is something to note though (perhaps in an article), that both AMD and nVidia cheat so aggressively in crysis benchmarking is deplorable and misleading to people buying those cards based on the canned benchmarks results they read online. The relative power of cards remains similar (should I go with a 3870 or an 8800GT is answered well by anands methods) but the hard method gives a better idea of what playable resolution might be at (the anand result of 30fps at 1600x1200 is due to both companies cheating, neither is playable at anything above 1280x1024).

Anand trys and succeeds in showing relative performance of cards compared to eachother. (I can tell, in percents, how much faster is a 8800GTS 512MB then a 8800GT).
Hard trys and succeeds in showing realistic and subjective playable performance of specific cards with specific games, but without substantial comparison between the cards. (I can tell how fast the 8800GT will be in crysis, I can't tell how it will compare to the 3870)
 

apoppin

Lifer
Mar 9, 2000
34,890
1
0
alienbabeltech.com
Originally posted by: Zinn
Originally posted by: DerekWilson:beer: here's to the AT Video Forum! you guys are famous!
Derek, you guys might be famous, but that's not such a great thing if you're famous for being wrong. I'd really be interested to see what you can say to refute the points Kyle brings up in this article.

Canned benchmarks might be repeatable, but what's the point of that when people are buying video cards to play games? When the 3870x2 beats an 8800 GTX in timedemo benchmarks, but is considerably slower in actual gameplay, what benefit are your readers getting by reading your results?

I think this is a big problem in the industry right now. AMD and NVIDIA focus too much resources on getting their numbers higher in canned benchmarks to the detriment of their performance in actual gaming applications. Kyle has shown that this is misleading to consumers, and I hope you can do better than just brushing him off.

well, i do benchmarking as a hobby and i play EVERY single game [or demo, in the case of Crysis] that i bench for this forum. If there is a difference in a "canned" benchmark from gameplay [like STALKER 'short/buildings'] i will run it and report on it as being different from regular gameplay.

i think i can be of use to people who play these games after i report on the results with MY rig in "real world" as well as "canned" repeatable benches.

in my *own* particular peculiar case - i love to play all my games fully maxed - every in-game setting 'ultra' if posible - and with 4xAA and 16xAF if at all possible and i tend to upgrade my HW untill i get a level of comfort at maximum settings and decent FPS. And i STATE this so readers will get an "idea" or visualize what i am reporting.

i also notice - in this article - that Kyle is much more thorough in explaining the *details* of his set-up in his current benchmarking. That IS what i asked him to do [on our forum] and i appreciate it [being conveyed to him with a favorable response to the suggestion].

as i said before, i would like to see formal reviews of both RW and canned testing ... the more reliable info, the better to make a decision

 

Cookie Monster

Diamond Member
May 7, 2005
5,161
32
86
So when do we pick up our pitch folks and burning torches? :D

Both kinds of reviews (canned/timedemos/real world gameplay) have their pros and cons. Its fairly hard to say who is using the right method. But for kyle to make such a bold statement on AT's method on reviewing video cards is pretty colorful.
 

Earfoam

Member
Oct 28, 2007
100
0
0
Hard as it is to believe, HardOCP is not the supreme entity they seem to self-project. At times, reading their reviews give me the same bad taste in my mouth as reading Tom's Hardware reviews.......
 

chunkylafanga

Junior Member
Jun 14, 2007
15
0
0
From Hard OCP "we can run the canned demo in REAL TIME and record the framerate with FRAPS."

Just a question, how exactly is this done?

I would like to replicate this test using my 8800gt (same setup as H)

thanks
 

apoppin

Lifer
Mar 9, 2000
34,890
1
0
alienbabeltech.com
Originally posted by: Cookie Monster
So when do we pick up our pitch folks and burning torches? :D

Both kinds of reviews (canned/timedemos/real world gameplay) have their pros and cons. Its fairly hard to say who is using the right method. But for kyle to make such a bold statement on AT's method on reviewing video cards is pretty colorful.

Do you think we hurt his feelings?

:D

He is obviously making a bit for attention ... and for credibility ... he is the one that admitted he benchmarked games without playing them in the past ... his famed "real world" testing is just to make his site different:

A couple of days before I was to be in San Francisco someone asked me, ?How did you like gaming on the card?? I had no idea. I had never actually used the video card to play games, I had only "benchmarked" it using traditional methods. Here I am supposedly an ?expert? on video cards telling our readers which cards are best for them and I had never used the card for the primary application that it is sold for; playing 3D games. It was a ?moment of clarity.?

I truly knew at that moment HardOCP?s direction in video card reviews had to change. I was not sure what needed to change at that moment, but this clarity pointed me in the right direction

Call me Mr. Cynical but his moment of "clarity" was a bid to drive traffic to a minor site :p

Yes, the Crysis Benchmark is more of a demonstration than a real benchmark ... do you notice how MANY FarCry benches there are?

And i am not so sure i understand this ... all he appears to be doing is making a real time imitation of a *custom* canned benchmark that HE considers "representative" of the game:

We pick a level or portion of the game that we find will give us a worst case scenario in terms of stressing the graphics card. This is a process that takes quite of bit of skill and practice in order to produce data that is comparable across different video cards. The numbers that you see graphed and charted in our evaluations are pulled from actual gameplay. We use FRAPS to record the framerate as we travel through the game. The trick is being able to do it multiple times in ?exactly? the same way. You have to liken this to a race car driver that becomes comfortable with his course. With enough practice you can start pulling very tight lap times that are of course comparable. And as any of our video card editors will tell you, practice makes perfect.

Is every ?lap? in our real-world gameplay identical? No it is not, but we put a tremendous amount of time and effort into making sure we have data which we think is thoroughly comparable. We think it would be irresponsible to do otherwise. If that is a deal breaker for you, we understand, but please remember that we draw our conclusions about a video card based on real gameplay, not the data graphed on our evaluation pages.

yep, in a nutshell, ^this^ is the *deal breaker* for me


Post the video of your 'lap' on the 'net Mr. KyleB and let ME attempt to replicate it as closely as i can. You have to load a "save" to get to your benchmark. Give us the 'save' and let us run thru your benchmarks over-and-over so we will all have a lot more representation of what is really "playable" or not and at a multiplicity of settings.
 

Earfoam

Member
Oct 28, 2007
100
0
0
I have noticed MUCH bias in HardOCP reviews. They'll get all PO'd and cry foul if you mention it too, but hey, truth always hurts doesn't it.....

Oh yeah, just my $0.02 as always ;)