Originally posted by: taltamir
.........................................................
I do like reading various sources though, without hardocp i wouldn't have known about the crysis benchmark cheating... And there is value to be gleaned from seeing the results of all testing methods... this means I can expect much lower performance in crysis engine based games with the GTS 512MB I just ordered... so I learned something from reading that article...
It is something to note though (perhaps in an article), that both AMD and nVidia cheat so aggressively in crysis benchmarking is deplorable and misleading to people buying those cards based on the canned benchmarks results they read online. The relative power of cards remains similar (should I go with a 3870 or an 8800GT is answered well by anands methods) but the hard method gives a better idea of what playable resolution might be at (the anand result of 30fps at 1600x1200 is due to both companies cheating, neither is playable at anything above 1280x1024).
Anand trys and succeeds in showing relative performance of cards compared to eachother. (I can tell, in percents, how much faster is a 8800GTS 512MB then a 8800GT).
Hard trys and succeeds in showing realistic and subjective playable performance of specific cards with specific games, but without substantial comparison between the cards. (I can tell how fast the 8800GT will be in crysis, I can't tell how it will compare to the 3870)
I agree
Hmm...... surprised nobody brought up the fact that they used ONE example, and one example only. Who's to say that it's simply not just a matter of Crysis' timedemo benchmark being flawed? In order to flat out say something is flawed, you need multiple examples to prove it. Yes, by using the example they did, it shows there might be a flaw with the "traditional" method, but it doesn't show that it is utterly useless in every title out there.
It would be the same way as writing a new video card review and using one game as the official benchmark for the card - that wouldn't be too reliable now, would it?
It would be the same way as writing a new video card review and using one game as the official benchmark for the card - that wouldn't be too reliable now, would it?
wow , this coming from hardocp.
they are easily "bought" and have 0 integrity.
for example when evga came out with motherboards (which were made by jetway) and bfg came out with boards (made by chaintech, especially the NF4 socket 939) hardocp gave them glowing reviews, when the chaintech and jetway versions got mediocre reviews everywhere else for being at best "value" boards at best, while the twice as expensive evga and bfg versions got great reviews at hardocp.
they are easily "bought" and have 0 integrity.
for example when evga came out with motherboards (which were made by jetway) and bfg came out with boards (made by chaintech, especially the NF4 socket 939) hardocp gave them glowing reviews, when the chaintech and jetway versions got mediocre reviews everywhere else for being at best "value" boards at best, while the twice as expensive evga and bfg versions got great reviews at hardocp.
Inspector Jihad
Lifer
- Apr 17, 2005
- 13,465
- 3
- 81
You are seeming to forget that other sites that used same methods that HardOCP did got same sort of results.
For example:
http://plaza.fi/muropaketti/ar...deon-hd-3870-x2-r680,2
-----
Now they are attacking against Anandtechs testing methods in their new article. The point they make is that there was huge performance difference when they tested with time demos and actual game play. Yes, 8800 ultra was faster on time demos as well, but it didn't see that large increase as did HD3870 X2.
For example they also tested Crysis with Anandtech's quality settings and in actual game play HD3870 X2 lost time after time against 8800 ultra with good margin..while in build in benchmark HD3870 X2 won.
For example:
http://plaza.fi/muropaketti/ar...deon-hd-3870-x2-r680,2
-----
Now they are attacking against Anandtechs testing methods in their new article. The point they make is that there was huge performance difference when they tested with time demos and actual game play. Yes, 8800 ultra was faster on time demos as well, but it didn't see that large increase as did HD3870 X2.
For example they also tested Crysis with Anandtech's quality settings and in actual game play HD3870 X2 lost time after time against 8800 ultra with good margin..while in build in benchmark HD3870 X2 won.
Valid points in that article, no doubt.
I love how there's a 60% rise in AMD's benchmark in Crysis fly-by while a meagre 1-2 FPS in crease in gameplay. WAY TO GO AMD! That's when you know that a timedemo or whatever isn't cutting it.
I love how there's a 60% rise in AMD's benchmark in Crysis fly-by while a meagre 1-2 FPS in crease in gameplay. WAY TO GO AMD! That's when you know that a timedemo or whatever isn't cutting it.
If you look at the history of hard ocp, they love to bring on the drama for page hits.Must be a slump in page hits, so its time to needlessly flame other websites in order to get page hits and forum registrations.
That used to be one of my favorite sites until they started doing that.I also think their "real world" benchmarking is more subjective to the reviewers tastes and does not reflect what others will experience while playing with the same equipment at the same settings.
Several times I have tried to play the same games they supposedly play, on the same hardware, at the same settings and found the play to be very choppy and difficult to enjoy.I have not been able to duplicate their results.
On the other hand,websites using traditional benchmarking methods,I have been able to duplicate results 100%.
So I believe hocp "real world" benchmarks should be taken with the whole salt shaker and not just with a single grain of salt.
They rely far too much on the personal preferences of the reviewer and not enough on a duplicateable method.
I think Anandtech should just point this out then ignore them. They are just looking for drama,attention,and page hits.
That used to be one of my favorite sites until they started doing that.I also think their "real world" benchmarking is more subjective to the reviewers tastes and does not reflect what others will experience while playing with the same equipment at the same settings.
Several times I have tried to play the same games they supposedly play, on the same hardware, at the same settings and found the play to be very choppy and difficult to enjoy.I have not been able to duplicate their results.
On the other hand,websites using traditional benchmarking methods,I have been able to duplicate results 100%.
So I believe hocp "real world" benchmarks should be taken with the whole salt shaker and not just with a single grain of salt.
They rely far too much on the personal preferences of the reviewer and not enough on a duplicateable method.
I think Anandtech should just point this out then ignore them. They are just looking for drama,attention,and page hits.
milesl:
Other sites with similar methods as HardOCP gets same sort of results you know? It's not just one site against everyone else anymore.
Other sites with similar methods as HardOCP gets same sort of results you know? It's not just one site against everyone else anymore.
As an ex-AnandTech employee, I sure remember having my share of poor reviews. That doesn't mean the methodology is incorrect.
Kyle's benchmarking method can justify whatever outcome he'd like. "Feel good" benchmarks are the easiest in the world since they can never be wrong. KillerNIC anyone?
The correct move here is to apologize and move on, but I wouldn't dignify any of this with a response. Tom's Hardware is already doing "gameplay" reviews. If AnandTech starts doing benchmarks like the rest of these guys, then you know the demographic of this whole industry has shifted away from the college educated techie to the high school gamer.
Derek: Unreplicable benchmarks are the first horsemen of the apocalypse for your industry -- whether its Kyle, Tom or you doing them. If any of us were still in (or ever went) to college we'd all fail trying to pass that sort of testing methodology off in a physics class.
Kyle's benchmarking method can justify whatever outcome he'd like. "Feel good" benchmarks are the easiest in the world since they can never be wrong. KillerNIC anyone?
The correct move here is to apologize and move on, but I wouldn't dignify any of this with a response. Tom's Hardware is already doing "gameplay" reviews. If AnandTech starts doing benchmarks like the rest of these guys, then you know the demographic of this whole industry has shifted away from the college educated techie to the high school gamer.
Derek: Unreplicable benchmarks are the first horsemen of the apocalypse for your industry -- whether its Kyle, Tom or you doing them. If any of us were still in (or ever went) to college we'd all fail trying to pass that sort of testing methodology off in a physics class.
I have not been able to duplicate their results in the past and that is all I need to know.The "real world benching" is totally subjective to the reviewers tastes.If other sites have reviewers with the same tastes as hocp then good for them.I know I do not from personal experience.
apoppin
Lifer
apoppin
Lifer
my *challenge* to my KyleB is to post his 'saves' and setting and lets us attempt to replicate them ... he is living in a "imitation" real-world fantasy land
all HE is doing is creating a *canned benchmark* that he plays in real time while controlling the variables unknown to everyone but himself ... do i believe he "finishes" every game ?
-imo his credibility has dropped below Zero if that is at all possible
imo there IS some value to his testing IF it can be duplicated - for that is the SCIENTIFIC method ... HardOCP is a sham and a fake and their members like to believe their FUD as truth
---surprisingly, there IS Truth hidden in the very first few words of his silly article:
yeah ... stuck in a rut of unscientific unrepeatable biased benches
:thumbsdown:
He reminds me of a creationist trying to defend as "scientific" the world being remade in a week a few years ago
:roll:
My step-father, who was a senior marketing executive for John Deere, always told me that when you mention your competition in ads or articles, especially trying to 'prove' that your product is better than theirs, you're simply scared of the competitor. 
I'll stand by AT's and Tom's reviews. HardOCP, for the most part, is poorly written and poorly reported. Regardless of flawed methodology or what have you, H's reviews are unbearable to read. This is coming from a journalist
I'll stand by AT's and Tom's reviews. HardOCP, for the most part, is poorly written and poorly reported. Regardless of flawed methodology or what have you, H's reviews are unbearable to read. This is coming from a journalist
I think both methods have their places. I think Crysis is a good example though for the article considering that the canned benchmarks DO run faster then when playing the game so hocp does have a point on that.
ATI and Nvidia are both guilty of doing things like "optimizing" time demos/benchmarks. Most recently found ATI's "60% increase on benchmark frame rates" that had minimal effect on real game play funny. I buy graphic cards to play actual games so I want to know what each game does while playing the game so I can see where Hardocp is coming from.
I think people should look at all methods, not just one and base their decision on that because each method does have pros/cons.
ATI and Nvidia are both guilty of doing things like "optimizing" time demos/benchmarks. Most recently found ATI's "60% increase on benchmark frame rates" that had minimal effect on real game play funny. I buy graphic cards to play actual games so I want to know what each game does while playing the game so I can see where Hardocp is coming from.
I think people should look at all methods, not just one and base their decision on that because each method does have pros/cons.
"is totally subjective to the reviewers tastes."
Every testing method out there requires that reviewer should be as neutral as possible. If you test the way AT does you can still favor some opponent if you aren't honest.
----
Muropaketti also try to find best playable settings, but with different way than OCP. They usually have that card which is being tested as reference. They try to find settings where it runs at 30-40 fps minimum and then compare it to other cards with same settings. If there's big enough difference to their direct competitors they might test that and perhaps some other card also with higher settings to demonstrate how big that difference is. Their test results usually gets out pretty late because they test so many times these games. They do this to get better picture on what that performance is on average. I mean that one "incident" wouldn't have that big impact on results. This testing method ain't different in that light that it requires tester to be neutral. So far I haven't seen any problems in that sector.
Actually this Muropaketti's article about Anantech's testing method is pretty untypical for them. If they would have had typical approach to this they would have just reported that there's disagreement with testing methods between OCP and Anandtech. Then they would have told us what arguments both would have and few links there.
Every testing method out there requires that reviewer should be as neutral as possible. If you test the way AT does you can still favor some opponent if you aren't honest.
----
Muropaketti also try to find best playable settings, but with different way than OCP. They usually have that card which is being tested as reference. They try to find settings where it runs at 30-40 fps minimum and then compare it to other cards with same settings. If there's big enough difference to their direct competitors they might test that and perhaps some other card also with higher settings to demonstrate how big that difference is. Their test results usually gets out pretty late because they test so many times these games. They do this to get better picture on what that performance is on average. I mean that one "incident" wouldn't have that big impact on results. This testing method ain't different in that light that it requires tester to be neutral. So far I haven't seen any problems in that sector.
Actually this Muropaketti's article about Anantech's testing method is pretty untypical for them. If they would have had typical approach to this they would have just reported that there's disagreement with testing methods between OCP and Anandtech. Then they would have told us what arguments both would have and few links there.
apoppin
Lifer
Originally posted by: angry hampster
My step-father, who was a senior marketing executive for John Deere, always told me that when you mention your competition in ads or articles, especially trying to 'prove' that your product is better than theirs, you're simply scared of the competitor.
I'll stand by AT's and Tom's reviews. HardOCP, for the most part, is poorly written and poorly reported. Regardless of flawed methodology or what have you, H's reviews are unbearable to read. This is coming from a journalist
your step father is a wise man ... HardOCP is a fringe fan site making a bid to appear credible to someone other than their forums
let's look at HardOCP carefully > UN-scientific ... unrepeatable ... flawed methods ... poorly written ... exaggerated ... if that is "real world" then the entire rest of the tech world is wrong
:Q
--i don't *think* so
... perhaps the best thing to do would be to IGNORE them
If you were walking down the street, and some beggar cussed at you... would you waste your time beating his face with a tire iron, or would you just ignore the attention seeker?
As a longtime AT reader I've seen some decline in Anand's reviews, unfortunately. Derek and Jarred have done a great job in breathing some life into AT's review-style in recent years and its gotten to the point I'd rather see one of them do a product review than Anand himself. Anand still writes the best tech write-ups you'll find anywhere though. :thumbsup:
As for using canned vs. non-canned, there's flaws with both. Personally I don't mind canned benches as long as they're indicative of actual gameplay. I don't have any problem with cutscenes as long as they're fully rendered with the game engine and mimic gameplay, but the problem is they're often simplified with less going on so they won't stress a GPU like normal gameplay would. In some games, they even change the resolution (Witcher for example goes to 1920x1080).
With non-canned, you introduce human error and gameplay variables that you can't control that you may or may not be able to replicate. It does a better job of stressing components in real-time, but more often than not they tend to stress CPU/memory/storage subsystems rather than the GPU itself. While this is great for showing real-world performance, it tends to skew the performance of the GPU if that was your intention.
Anand still writes the best tech write-ups you'll find anywhere though. :thumbsup:As for using canned vs. non-canned, there's flaws with both. Personally I don't mind canned benches as long as they're indicative of actual gameplay. I don't have any problem with cutscenes as long as they're fully rendered with the game engine and mimic gameplay, but the problem is they're often simplified with less going on so they won't stress a GPU like normal gameplay would. In some games, they even change the resolution (Witcher for example goes to 1920x1080).
With non-canned, you introduce human error and gameplay variables that you can't control that you may or may not be able to replicate. It does a better job of stressing components in real-time, but more often than not they tend to stress CPU/memory/storage subsystems rather than the GPU itself. While this is great for showing real-world performance, it tends to skew the performance of the GPU if that was your intention.
apoppin
Lifer
i would never use a tire iron
:Q
i never ran across a tech savvy beggar on the street ... it appears they are all on the 'net
I think their reviews are perhaps less scientific, but a hell of a lot more relevant and I'll take that over whatever anyone else does. I mean, take that Crysis example where frame rates increased a hell of a lot in the fly-by but only a little in the real game. Is this a moot point to you guys? This bears no relevance at all?
Also, if you read the article carefully and in full, you'll notice that he says that he doesn't use the graphs he provides as the basis for his conclusion. He claims to play the game in entirety ( this may be a point of contention, but I dont' think he's a liar) and adjusts the settings until he gets the best image quality coupled with playability. He then picks out a particular stressful scenario from the game and displays the FPS to support his conclusion. Perhaps this isn't repeatable by you or me, but I'll take it over the canned stuff I suppose because it may lead to "cheating" like the 60% increase in FPS with AMD's hotfix.
Also, if you read the article carefully and in full, you'll notice that he says that he doesn't use the graphs he provides as the basis for his conclusion. He claims to play the game in entirety ( this may be a point of contention, but I dont' think he's a liar) and adjusts the settings until he gets the best image quality coupled with playability. He then picks out a particular stressful scenario from the game and displays the FPS to support his conclusion. Perhaps this isn't repeatable by you or me, but I'll take it over the canned stuff I suppose because it may lead to "cheating" like the 60% increase in FPS with AMD's hotfix.
Similar threads
TRENDING THREADS
-
Discussion Intel current and future Lakes & Rapids thread- Started by TheF34RChannel
- Replies: 20K
-
Discussion Speculation: Zen 4 (EPYC 4 "Genoa", Ryzen 7000, etc.)- Started by Vattila
- Replies: 13K
-
-
-
Facebook
Twitter
Instagram