HardOCP says anandtech has bad methods

[NoStateofMind]

Originally posted by: tuteja1986

Originally posted by: PeteRoy
Isn't hardcop nvidia biased?

He is as fair and balanced like Rupert Murdock

:laugh: :thumbsup:

 

[NoStateofMind]

Originally posted by: Rusin

Originally posted by: DerekWilson

Originally posted by: Rusin
Well..at least they didn't test cut scenes like Anandtech did few times (This one AT editor did say that they didn't do this even when their review said they did)

anand change the test while i was in the hospital and i didn't pay attention ... also, i acknowledged that i got that wrong. and it wasn't "a few times" it was one cut scene.

and cut scenes are still often very useful.

One cut scene? Actually two..CoD4 was one and then The Witcher.

You wouldn't happen to be a disgruntled [H}ard member would you?

 

[Acanthus]

HardOCP has the worst testing methodology ive ever seen...

I stopped reading there and posting on the forums long ago.

 

[novasatori]

What ever happened to recording demos in games WHILE playing, or multiplayer time demos and playing them back for tests.

Oh yeah, that was quake, quake 2, and quake 3 days...

Do games even have that feature anymore??

That would be the only way I would accept the "gameplay" benchmarks. As mentioned the tests are not even representative or able to be reproduced. No matter how close he was to playing the same as the "timedemo" the difference is statistically insignificant when you factor in all the differences that could have happened.

 

[Acanthus]

Originally posted by: KristopherKubicki
As an ex-AnandTech employee, I sure remember having my share of poor reviews. That doesn't mean the methodology is incorrect.

Kyle's benchmarking method can justify whatever outcome he'd like. "Feel good" benchmarks are the easiest in the world since they can never be wrong. KillerNIC anyone?

The correct move here is to apologize and move on, but I wouldn't dignify any of this with a response. Tom's Hardware is already doing "gameplay" reviews. If AnandTech starts doing benchmarks like the rest of these guys, then you know the demographic of this whole industry has shifted away from the college educated techie to the high school gamer.

Derek: Unreplicable benchmarks are the first horsemen of the apocalypse for your industry -- whether its Kyle, Tom or you doing them. If any of us were still in (or ever went) to college we'd all fail trying to pass that sort of testing methodology off in a physics class.

As a science major i find that hilarious

The HardOCP method:

"In todays experiement will be will be burning fluids in the laboratory.
We will be changing the jar, fuel type, amount of fuel, amount of oxygen, temperature, and pee in some of the samples for good measure, and try to draw conclusions from that."

 

[nevbie]

I think the most enjoyable part of HOCP video card reviews are those graphs of FPS values. They just look so fun! I suppose they are made by utilizing FRAPS? The apples vs oranges comparison is otherwise useless to me, though, and mildly uninteresting. I would rather understand why better cards get worse FPS values in those graphs than older cards, at times. Perhaps it's the apples vs oranges.. bleh.

That HardOCP article made me think of the ways you can measure performance of a game. As I understand it now, "timedemo" plays the demo as fast as it can, and gives the time as a result. It is a function of the game. Every other measurement is made by FRAPS, correct? Or can the games measure average/min/max fps during a normal demo, for example? eh, it is slightly confusing, and I'd like to find a genius that tells me everything about these things in an article.

All these demo-based measurements can be replicated, heck, you could even replicate real game by making a certain type of map. As a quake 3 player/mapmaker/whatever I could make a simple map where you start in air and drop. No input needed. Of course such idea should be refined more than this..

The most interesting question is: How much you can try to mimic gameplay and still keep things replicateable? I'm not interested in apples vs oranges or subjective comparisons, but this instead. If the various "objective" measurements give DIFFERENT results, THEN I'd like to know why. And answering that could tell us that which is the most desirable way to measure video card performance under (certain) games.

Bashing other sites and most of the random posts in this thread are useless and boring (though I skipped a page), but there are also interesting things here.

 

[BFG10K]

Don't believe it, answer this - on a new driver update, are you going to hear that they improved Crysis performance by 20-30% or hear that they improved the framerate on UT2004 (WoW, Quake 4, etc...)?

Actually ATi's driver updates frequently contain information about performance gains in legacy games.

 

[Keysplayr]

Originally posted by: myocardia

Originally posted by: keysplayr2003
You've never heard of bracket racing?

I might understand a thing or two about bracket racing; I did it for almost ten years. What I don't understand is how bracket racing has anything to do with my post that you quoted. Care to explain?

I was actually quoting this:

"He needs to shut his trap, apples to apples benchmarks are where its at... I dont care about his methods, their benchmarks are just unnacceptable... When you wanna compare 2 cars by test driving them, are you gonna use different tracks? I dont think so! Wouldnt make much sense now, would it? It might or might not be the same as "real world experience" but the lack of variables make it a good benchmark either way... Meanwhile, Kyle happily goes through a light area with the card he likes, and then when benchmarking the other card, taxes it as much as he can... Oh but.. Its real world benchmarking"

And I can't see how you didn't put that together. It just happened to be someone you quoted, but hacked off the name. So I just quoted your post.

Anyway, I used to race at hampton raceway, and sometimes i would face a faster opponent and was granted a certain time to launch ahead of him/her, and other times a slower opponent, who would get the first break from the tree. The point of all this is, even though we launched at different times, we still usually ended up at the finish line very close.

But why should I have to explain this if you raced for 10 years? you should know. Yes?

 

[Keysplayr]

Originally posted by: ShadowOfMyself

Originally posted by: keysplayr2003

Pick a game. Decide what you think is a playable framerate for said game, and test them.

And how exactly does that help for the other 95% combinations of settings? Thats another problem... Ok sure, by reading HOCP I know Card 1 is playable at Settings A, and Card 2 is playable with only B settings... And? What if I wanna try settings A on Card 2 or vice versa? Thats something his methods dont cover, and its a pretty large mistake IMO

If am I thinking of buying a card I want to know how it performs in as many different situations as possible
to know Im covered

I cant get such info from HOCP benchmarking

95% huh? You mean reducing AA a notch? Or changing say, "soft shadows" to medium, or draw distance to 50% to get the same framerates as the faster card? Wow, that is a whole lot of information to wrap my head around. How could I have possibly discerned anything from those benchmarks.

It is so easy it is mind numbing. And my mind is numb from all this method bashing. People often do not like change, or something different. In ancient times, people would get stoned just for being different or having a different way of seeing things. Like what is happening here.

 

[ricleo2]

What an entertaining/amusing thread. I would hazard a guess that AT editors or employees think that also.

 

[SilentRunning]

Originally posted by: keysplayr2003

Originally posted by: myocardia

Originally posted by: keysplayr2003
You've never heard of bracket racing?

I might understand a thing or two about bracket racing; I did it for almost ten years. What I don't understand is how bracket racing has anything to do with my post that you quoted. Care to explain?

I was actually quoting this:

"He needs to shut his trap, apples to apples benchmarks are where its at... I dont care about his methods, their benchmarks are just unnacceptable... When you wanna compare 2 cars by test driving them, are you gonna use different tracks? I dont think so! Wouldnt make much sense now, would it? It might or might not be the same as "real world experience" but the lack of variables make it a good benchmark either way... Meanwhile, Kyle happily goes through a light area with the card he likes, and then when benchmarking the other card, taxes it as much as he can... Oh but.. Its real world benchmarking"

And I can't see how you didn't put that together. It just happened to be someone you quoted, but hacked off the name. So I just quoted your post.

Anyway, I used to race at hampton raceway, and sometimes i would face a faster opponent and was granted a certain time to launch ahead of him/her, and other times a slower opponent, who would get the first break from the tree. The point of all this is, even though we launched at different times, we still usually ended up at the finish line very close.

But why should I have to explain this if you raced for 10 years? you should know. Yes?

But when racing there may be differences between the lanes such as smoothness and how much rubber is layed down which contributes to the speed of the run. There may also be differences with how the drivers go through the gears. These are the types of issues similar that are not being measured when play testing as opposed to "canned" testing.

 

[apoppin]

Originally posted by: keysplayr2003

Originally posted by: ShadowOfMyself

Originally posted by: keysplayr2003

Pick a game. Decide what you think is a playable framerate for said game, and test them.

And how exactly does that help for the other 95% combinations of settings? Thats another problem... Ok sure, by reading HOCP I know Card 1 is playable at Settings A, and Card 2 is playable with only B settings... And? What if I wanna try settings A on Card 2 or vice versa? Thats something his methods dont cover, and its a pretty large mistake IMO

If am I thinking of buying a card I want to know how it performs in as many different situations as possible
to know Im covered

I cant get such info from HOCP benchmarking

95% huh? You mean reducing AA a notch? Or changing say, "soft shadows" to medium, or draw distance to 50% to get the same framerates as the faster card? Wow, that is a whole lot of information to wrap my head around. How could I have possibly discerned anything from those benchmarks.

It is so easy it is mind numbing. And my mind is numb from all this method bashing. People often do not like change, or something different. In ancient times, people would get stoned just for being different or having a different way of seeing things. Like what is happening here.

except i believe HardOCP is deliberately hiding their "saves" and fiddling with the 'variables" to produce deliberately skewed results

i'd be very glad to PROVE HardOPCP is a sham and a fan-site ... with Kyle's cooperation of course ... i want HIM - a big mouth troublemaker in the tech industry to allow us to TEST his methods.

either Kyle can

1) put up

or

2) stfu

 

[myocardia]

Originally posted by: keysplayr2003

Originally posted by: myocardia

Originally posted by: keysplayr2003
You've never heard of bracket racing?

I might understand a thing or two about bracket racing; I did it for almost ten years. What I don't understand is how bracket racing has anything to do with my post that you quoted. Care to explain?

I was actually quoting this:

"He needs to shut his trap, apples to apples benchmarks are where its at... I dont care about his methods, their benchmarks are just unnacceptable... When you wanna compare 2 cars by test driving them, are you gonna use different tracks? I dont think so! Wouldnt make much sense now, would it? It might or might not be the same as "real world experience" but the lack of variables make it a good benchmark either way... Meanwhile, Kyle happily goes through a light area with the card he likes, and then when benchmarking the other card, taxes it as much as he can... Oh but.. Its real world benchmarking"

And I can't see how you didn't put that together. It just happened to be someone you quoted, but hacked off the name. So I just quoted your post.

I was quoting ShadowOfMyself, but I put a comment in the middle of the quote, just like I'm doing in this post.

Anyway, I used to race at hampton raceway, and sometimes i would face a faster opponent and was granted a certain time to launch ahead of him/her, and other times a slower opponent, who would get the first break from the tree. The point of all this is, even though we launched at different times, we still usually ended up at the finish line very close.

But why should I have to explain this if you raced for 10 years? you should know. Yes?

Yeah, I know how bracket racing works. Ten years is actually quite a long time. I just had no idea what your point was, at that point. What I don't understand is how anyone on earth would think that it's fair to benchmark two cards at completely different settings. Do you think that an 8800 Ultra is slower than a 2600 Pro? Because if I want to, using the same logic, I can make a 2400 Pro faster than an 8800 Ultra in any gaming benchmark, including Crysis.

Don't believe me? How many FPS do you think an 8800 Ultra could do in Crysis @ 2560x1600, with 16x SSAA/16x AF, at high or ultra high in-game settings? It's gotta be pretty low, since it only gets 24.4 FPS at 1920x1200, 0x AA/1x AF. Now, how do you think a 2600 Pro would do @ 800x600, 0x AA/0x AF, with low in-game settings? Still think benchmarking that way is fair?

 

[richwenzel]

the dude from voodoo actually suggests in one of his CPU magazine articles that benchmarks in general are not good (comparing them to bench press as a measure of strength in actual sports) and that real life experiences are...

i don't really read hardocp, and to be honest i have seen other places have low opinion of AT reviews (esp. the psu ones)...and i don't really care...i come to AT for the boards and the good opinons/experience you can get there....i don't really every go do hardocp..

but subjective reviews do have their place...

real world experience is valuable...

ill give you an example...

when plasmas came out, the 1080 ones cost over 2x the 480 ones....i believe it was hometheater magazine would do SBT (or maybe it was DBT) of a group of 480 and 1080 resolution panels...

on paper the expensive sony and necs looked better, and they truly delivered 1080 performance...

for like 3 years in a row, you know which one did the panel of reviewers (who did not know the brands they were looking at, just some code letter/number [can't remember right now]) picked? the 480 panasonic...

why? well cause the panasonic had "better" blacks...not measure by contrast (not that contrast is a measure of blacks, and i dont want to get into explanation of this), but just better...

what did the writer of the article conclude? that viewers are more sensitive to light/dark changes then fine detail....but if you looked at canned tests (i.e. resolution, on/off contrast, lumens, etc), the one that won would not have been the panasonic...

it didn't win just for best value, it won for best picture...and from a "benchmarking" perspective, it would have lost...

does this make the panel of AV experts opinion invalid because it was subjective? no...i would go by that...

if a group of 10 people played 1 hour each of two different cards (say a 3870 and 8800gt) with whatever settings and not know which card is which, lets just say max setting where min FPS doesn't get below 30, and 8+ of the viewers agreed that one card had a better experience, i would be really interested in that...

Rich

 

[Endgame124]

Originally posted by: myocardia
Don't believe me? How many FPS do you think an 8800 Ultra could do in Crysis @ 2560x1600, with 16x SSAA/16x AF, at high or ultra high in-game settings? It's gotta be pretty low, since it only gets 24.4 FPS at 1920x1200, 0x AA/1x AF. Now, how do you think a 2600 Pro would do @ 800x600, 0x AA/0x AF, with low in-game settings? Still think benchmarking that way is fair?

Unless, of course, you aimed for a minmum framrate based on the game (say 20fps for game 1, 40 fps for game 2), and then bumped up the game detail until you hit that minimum.

Like this:

Card 1, Game 1: 1600x1200, 2x AA, 8x AF, High Details (hits 20 fps min)
Card 2, Game 1: 1600x1200, 0x AA, 8x AF, Med Details (hits 20 fps min)

Card1, Game 2: 2560x1600, 4x AA, 16x AF, Ultra Detail (Hits 40 fps min)
Card 2, Game 2: 1600x1200, 4x AA, 16x AF, Ultra Detail (Hits 40 fps min)

Gee, it sure looks like card 1 wins.

Then you say: "But why play the game? Its not 100% reproducable (only 94%), better run the timedemo". But the timedemo results look like this:

Card 1, Game 1 Timedemo: 1600x1200, 2x AA, 8x AF, High Details (hits 20 fps min)
Card 2, Game 1 Timedemo: 1600x1200, 4x AA, 16x AF, Ultra Details (hits 20 fps min)

Card1, Game 2 Timedemo: 2560x1600, 4x AA, 16x AF, Ultra Detail (Hits 40 fps min)
Card 2, Game 2 Timedemo: 2560x1600, 8x AA, 16x AF, Ultra Detail (Hits 40 fps min)

Gee, card 2 cheats the timedemo because the driver is optomized for timedemo runs of game 1 and game 2. Sure it sucks if you actually play the game, but we don't care about that because its not scientific enough? I'm not getting your point...

 

[apoppin]

yeah "display" reviews and "audio" reviews are very subjective ... you *expect* that when you hear things like "warmth" and "depth" to describe the auditory experience.

But Benchmarking Computer HW does have an established scientific testing method
-What HardOCP is attempting to do is to cast doubt on SCIENCE in favor of Kyle's obfuscated voodoo "methods"
---INSTEAD of questing established methods ... let him PROVE *his* methods are sound. i want to *examine* his methodology ... to attempt to *replicate* his results
the Burden of Proof is on HardOCP ... not on established testing.

Even Bracket Racing has established rules. Kyle is running according to HIS rules.
--and i 'think' he is going to be exposed if he dares show us REALLY how he does it:

He *can* be vindicated IF he chooses and IF he is right - Post a video of his benchmark run and the Crysis save ... or shut up
-i will kiss his hairy butt over at HardOCP and apologize profusely IF he IS right
:Q

without his cooperation - without his attempt to PROVE his method - anything else is a BS useless discussion here

it's ALL up to Kyle ... i am looking forward to Part 2 of his article

 

[richwenzel]

i submit that i don't know as much as you, so i am asking...

is it possible with the same settings, that one card at fps X looks differently than another card at the same FPS?

if so, then science doesn't capture it all...

 

[apoppin]

Originally posted by: richwenzel
i submit that i don't know as much as you, so i am asking...

is it possible with the same settings, that one card at fps X looks differently than another card at the same FPS?

if so, then science doesn't capture it all...

no the cards look the same
-the scene looks the same the details are the same the settings are the same everything is IDENTICAL ... the difference is the SPEED the demo runs at - that IS the difference between the cards ... one is faster or slower or they are equal - why screw with setting on ONE card or set DIFFERENT setting or each card like the primate testing we are being asked to swallow without any proof.
--apples to monkeys --forget bananas

and the children that populate the HardOCP forums are sending Crap to my Junk e-mail account ... very mature ... like your fearless leader
:roll:

 

[richwenzel]

so there is no IQ difference between ATI and nVidia? I have read that some say there are...

i literally don't know, nor do i truly care....i just buy something that that masses say are pretty good value, look at my game and decide if it is good enough and go on with my life...

 

[aussiestilgar]

Way to blow it out of proportion. FPS results are always going to be approximated and possibly skewed. Thats why in some games the 3870 beats the 8800GT, and in other games vice versa. If you look at the big picture to compare graphics cards, using a bunch of different games running timedemos OR playing from a save point STILL gives you a GENERAL IDEA of how a card performs overall. No one ever said you could count on any particular benchmark. There is nothing wrong with AT's methods. HOCP is just a different method... maybe it shows different results, maybe not, doesn't mean its right or wrong. Different sites showing different results is nothing new. All said and done, you still get an overall approximation of a card's performance.
You know what is more important than all this? How much you enjoy reading the article. Thats why I like AT.

 

[Zap]

Originally posted by: BFG10K

Don't believe it, answer this - on a new driver update, are you going to hear that they improved Crysis performance by 20-30% or hear that they improved the framerate on UT2004 (WoW, Quake 4, etc...)?

Actually ATi's driver updates frequently contain information about performance gains in legacy games.

This makes me torn between GPUs. I like Nvidia GPUs because I can never get picture scaling to work how I want it to work on ATI GPUs (if an older game doesn't support widescreen, I WANT the black bands on the sides). However, I've seen a couple times when older games just no longer work on latest Nvidia cards without dinking with settings. Examples: Command & Conquer: Generals (and Zero Hour expansion) doesn't seem to work on latest Nvidia hardware/drivers without changing settings. I haven't heard such an issue with ATI cards. A much older title, Crimson Skies has issues with Nvidia but not with ATI.

Originally posted by: richwenzel
i don't really read hardocp, and to be honest i have seen other places have low opinion of AT reviews (esp. the psu ones)...

Yeah, the PSU reviews here aren't bad and are definately a notch above all those other review sites that just slap the PSU into their gaming rig, find that they can still run their games, and give it a 9/10 Editor's Award. However, I've heard complaints about the more esoteric stuff like ripple tests and such. Let's just say that AT PSU reviews are very much above "average" but not quite the "cream of the crop." I would say though that any PSU review that slaps the bad boys onto a synthetic load tester to find out if the PSU can actually put out what it claims on the label is already at 80% of a good review. Props to AT for getting that far, and here's to hoping for more and better AT PSU reviews. :beer:

 

[theprodigalrebel]

This is my awesome proposal for benchmarking games:

1. Record a demo during an intense firefight, the part of the game where all hell breaks loose
2. Run the same demo (in real-time) on all the cards being tested @ the three popular resolutions (1280x1024, 1680x1050, 1920x1200)
3. Record frame-rate data using FRAPS (and not the timedemo command)
4. Report performance similar to the in-built test in FEAR ( x% frames < 25FPS, y% frames between 25-40FPS, z% frames > 40 FPS)
5. ...
6. PROFIT!

 

[Tempered81]

isn't the reason his "real time gameplay benchmarks" are lower than the "canned benches" because he is using Fraps? I remember using fraps and it knocking like 30% off my framerates while i had it running

 

[myocardia]

Originally posted by: Endgame124
Gee, card 2 cheats the timedemo because the driver is optomized for timedemo runs of game 1 and game 2.

No offense, but you obviously know nothing about video cards. If you did, you'd know that both nVidia and ATI cheat equally in these canned benchmarks. They have for many years now, and it's not about to stop anytime soon.

Originally posted by: richwenzel
is it possible with the same settings, that one card at fps X looks differently than another card at the same FPS?

There's not really any difference in image quality. The last time there was a noticeable difference in IQ between the two companies was back in the GeForce 5x00 vs the Radeon 9x00 days. The Radeon 9x00's had a pretty sizeable advantage in IQ. Then nVidia came out with the 6x00 series, and retook both the performance and IQ lead. ATI's next generation retook the performance lead, and caught up in IQ, etc, etc. The differences these days in IQ are so small that not even a fanboy of either company could tell the difference. The real differences between the cards are their features, more than anything else, but also their drivers, like Zap has already said.

 

[Adam12176]

Web traffic must be down over there.

Once in a while it's nice to just look at a bar graph of cards in the SAME settings and make conclusions. Not this apples and oranges crap. At least provide standard benchmarks for people to make their own conclusions.

And when was the last time hardocp even did anything 'extreme.' Place is more of a glorified IGN site than anything.