Anyone think more Objectivity is needed?

[z3R0C00L]

I have a question for you all.

It has come to my attention that many sites seem to not post the whole story. They omit game results that would show their sponsors in a bad light (at least that's m observation). Now I don't want to make any accusations just yet but take the info I'm giving you bellow at face value. Before I start I have to make a confession. I currently only own Geforce 8 series cards (2x8800GTX + 1x8800GTS) I thought I should mention this beforehand.

Perfect examples are:

HardOCP
FalconNorthwest (there was an article on DailyTech regarding this)
Fudzilla
Bjorn3D

What I mean is that.. how many of you play S.T.A.L.K.E.R? How many of you play Flight Simulator X?

Would you deem those two games to be more popular then:
Quake 4
Battlefield 2
Doom 3
Half Life 2
and
F.E.A.R ?

I don't think so.. not if you look at the sales. Sales figures are posted on a monthly basis and S.T.A.L.K.E.R./Flight Simulator X have never even been on the radar. As such most gamers have the other titles I've mentioned above and play those instead.

So why does HardOCP only test on those games which are known to show the Radeon HD 2900XT in a bad light? (games which are the least popular as well) And why does HardOCP not test those other games I mentioned where the Radeon HD 2900XT 512MB is on par or faster then an 8800GTS 640MB? (this is a question coming from a Geforce 8 series owner).

Here's the HardOCP review in question:
Click Me! with the link I just provided above.

Now have a look at this:
Click Me!

Both those two reviews painted a single one sided story each using titles that performed best on each respective architecture.

Now for some Objectivity look here:
Click Me!

Notice anything?

It's quite apparent that HardOCP and Fudzilla are both misleading the public from their own respective ends. Oddly enough nVIDIA card partner banners are all over HardOCP's site whereas ATi/AMD banners are all over Fudzilla's site. Coincidence? Maybe. I've also heard some folks over at HardOCP claim that HardOCP does benchmarking different. Well.. how so? They claim HardOCP uses AA+AF + High resolutions. Guess what.. so did Firingsquad.

It seems that Kyle Bennett (yes I got in a HUGE argument with him over this) won't admit to using un-popular games that seem to only paint one side of the story.

Now taking it further I visited the sites in question and took screenies.

These images are large so bare with me (i've made thumbnails):

Fudzilla's Advertising=
Click Me!
They lean ATi/AMD in their articles..

HardOCP's Advertising=
Click Me!
Click Me!
Click Me!
They lean nVIDIA in their articles..

Firingsquad's Advertising=
Click Me!
Click Me!
They don't lean to either side and paint a fair and balanced picture testing MANY games. They clearly show that as a whole the Radeon HD 2900XT is a match for the GeForce 8800GTS 640MB (unlike what HardOCP show) but that it trails the GeForce 8800GTX 768MB (unlike what Fudzilla show). And their advertising is nVIDIA thus clearly highlighting objectivity.
So what do you think about Objectivity in today's PC Hardware reviews?

I just gave you the information to consider what conclusions you come too are your own. I expect that fanboyism won't poison this thread and instead we can discuss this as adults.

You decide.

PS- This is in no-way geared towards Anandtech who do a bang up job to remain objective in their reviews.

 

[hardbench]

THis has and always will be an ongoing problem...

 

[Wreckage]

Just read 5 or so sites and take an average from that.

 

[z3R0C00L]

I'm thinking that "WE" the consumers need to demand change. Sites that practice this dishonesty need to be exposed.

At least that's my opinion.

I'd love to see Anandtech and Tom's (the two biggest sites) do up some major updated reviews just to expose this behavior by the likes of Fudzilla and HardOCP.

THis has and always will be an ongoing problem...

Yes it has been and it's pissing me off..lol

 

[nitromullet]

Don't believe everything you read... Personally, I don't really care if some sites are biased. We all are to some extent. You just have to work on gathering information and arriving at your own conclusions. My biggest peeve is the lack of Vista benchmarking. Apparently, all these review sites think that people who spend big bucks on "next gen" hardware also like to use last year's OS...

Edit: I do agree with you on the DailyTech posting of that editorial from the Falcon Northwest PR guy... That was a serious low for DailyTech (and Anandtech by association)... Not only should a PR guy from a hardware company not ever get that kind of space on a supposed tech site, but the benchmarks were not conclusive at all with only two games benched with vaguely described settings such as "default" and "high". ...not exactly the clear cut victory Mr. Falcon NW was going on about IMO.

 

[apoppin]

Originally posted by: z3R0C00L
I'm thinking that "WE" the consumers need to demand change. Sites that practice this dishonesty need to be exposed.

At least that's my opinion.

I'd love to see Anandtech and Tom's (the two biggest sites) do up some major updated reviews just to expose this behavior by the likes of Fudzilla and HardOCP.

THis has and always will be an ongoing problem...

Yes it has been and it's pissing me off..lol

We already do this "in house"; we can benchmark anything the review sites can ... except for the stuff under NDA.

In House HD2900XT vs. 8800GTS 640

it is the VERY LATEST info from Sunday with quite a few new/old games and the very latest drivers for nvidia and AMD

STUDY posts 1-3 and number 5. In [long] Number 5 i compared my HD2900xt to a 8800GTS-640 OC in both Win Vista32 and WinXP with a detailed set of benchmarks at 14x9 and 16x12 with all in-game setting completely maxed - apples to apples.

and search for n7's review here of the 8800GTX OC vs. the 1GB oc'd 2900xt

the truth is in here

 

[z3R0C00L]

Originally posted by: apoppin

Originally posted by: z3R0C00L
I'm thinking that "WE" the consumers need to demand change. Sites that practice this dishonesty need to be exposed.

At least that's my opinion.

I'd love to see Anandtech and Tom's (the two biggest sites) do up some major updated reviews just to expose this behavior by the likes of Fudzilla and HardOCP.

THis has and always will be an ongoing problem...

Yes it has been and it's pissing me off..lol

We already do this "in house"; we can benchmark anything the review sites can ... except for the stuff under NDA.

In House HD2900XT vs. 8800GTS 640

it is the VERY LATEST info from Sunday with quite a few new/old games and the very latest drivers for nvidia and AMD

STUDY posts 1-3 and number 5. In [long] Number 5 i compared my HD2900xt to a 8800GTS-640 OC in both Win Vista32 and WinXP with a detailed set of benchmarks at 14x9 and 16x12 with all in-game setting completely maxed - apples to apples.

and search for n7's review here of the 8800GTX OC vs. the 1GB oc'd 2900xt

the truth is in here

Thanks man.

See that's a better picture.

 

[bryanW1995]

I read both apoppin's and n7's reviews. Both are solid. Toms and AT periodically do reviews that seem very impartial to me, too.

 

[nullpointerus]

Originally posted by: z3R0C00L
It has come to my attention that many sites seem to not post the whole story. They omit game results that would show their sponsors in a bad light (at least that's m observation).

Welcome to the media. That's how things are done...on major technology news sites, on major television news networks, in major newspapers and magazines. The problem is that everyone seems to be working with different sets of facts. Everything becomes a petty he-said/she-said argument instead of a reasonable debate where people can agree on a set of facts and have an honest discussion about them.

Now I don't want to make any accusations just yet but take the info I'm giving you bellow at face value. Before I start I have to make a confession. I currently only own Geforce 8 series cards (2x8800GTX + 1x8800GTS) I thought I should mention this beforehand.

...
...
...
You need to take a course in journalism. NOW. Seriously, this is exactly what's needed in the media--people who try to pass information to the general public as accurately as possible and then be honest about their own biases. The I'm-a-servant-of-the-people-my-views-are-valid-I-can-say-anything-I-want-truth-is-all-relative attitude is for the birds. People in the media need to take responsibility and adopt some genuine humility.

Perfect examples are:

Examples are everywhere. That's why I no longer want to hear anything even remotely related to U.S. politics. There is no "closure"...no way to stem the nonsense. As soon as the general public gets even remotely close to the truth, a new crisis comes up and everyone forgets about pinning down the truth on the old crisis.

My point is that politics in the abstract sense occurs wherever people form emotional attachments to the subject matter, as one can see in the GPU flamewars. It's the exact same problem, just a different pile of poop and different teams with tattered flags. And I've become equally sick of it. The arguments are like black holes: meaningless loss of energy.

EDIT: The cynicism isn't directed at anyone here. I'm just tired of politics. There is no end to it...nothing to be finished. Well, maybe a bunch of insightful optimists will come together to create a good media. ... God, I must be losing my mind.

/apathetic cynicism

 

[Spoit]

The problem with the way that HardOCP tested the HD 2900XT wasn't necessarily their game selection, but the fact that their testing revolves around playing at an arbitrary "good enough" framerate, which exacerbated ATI's problems with running AA. The problem with all the games that you listed for alternatives is that they're all years old, so they hardly stress either card.

 

[mtl]

Yes for example X-bit labs always disables Nvidia's default settings(anything with the word optimization in it gets turned off) when they test the 8800 series, incurring a 10-12 per cent penalty in performance with no image quality gain. Many other reviews have shown the default image from the 8800 series to be at least equal if not slightly better than the 2900 xt so why do they do this? Nvidia spends a great deal of time and money developing drivers and settings to give the best image quality and performance. The reviews indicate in the 8800 series they succeeded.
It might appear as if X-Bit labs is trying to equalize the 8800's superiority versus the 2900xt, yet they are constantly quoted on this forum as a reliable source.

Personally I really like Hardocp's methods to show real world gaming. They have strongly recommended many ATI cards in the past and I am sure they are honestly trying to be objective.

 

[Cookie Monster]

Not sure about kyle bennett but i trust brent at .

However just like mtl said, xbitlabs method of "apple to apple" comparison via turning off opts on NV cards only really makes sense on 6/7 series cards. With the introduction of the G80, we already know that its default IQ is "technically" superior than the R600s e.g the G80's AF algorithm is technically better than the R600's AF algorithm.

 

[rise]

so, is there a way to standardize testing or is this a bitch fest

 

[nitromullet]

However just like mtl said, xbitlabs method of "apple to apple" comparison via turning off opts on NV cards only really makes sense on 6/7 series cards. With the introduction of the G80, we already know that its default IQ is "technically" superior than the R600s e.g the G80's AF algorithm is technically better than the R600's AF algorithm.

That is a good point. Review sites really should re-evaluate optimizations with each new generation and compare them to each other. Interesting that no one has complained of any shimmering with the 8-series, yet they are benched with optimizations off. I'm actually guilty of this myself. I haven't even given my own card the chance to run with the optimization turned on - I guess I was well trained by the 7-series.

edit: well, I guess it's a good thing that I turned the opts off and that xbit does too because I can definitely see the difference in some cases. Most notably (and not surprising) is World of Warcraft. There is definitely a shimmer present at the default settings that isn't with the opts turned off.

 

[mtl]

If image is helped by turning off that's fine. But many games seem to be fine with opts on so why incur the performance hit ?

 

[imaheadcase]

I have a idea, don't be a hardware reviewer site and takes ANY ad money or put ads on site that deal with hardware at all. Lots of sites do this,they put ads on site based on IT tech in general,but not specific hardware itself.

Stupid sites like wimpyOCP are so hardheaded in ways they do stuff, they could care less, lots of sites like them though just they are the main one.

 

[s44]

Originally posted by: Spoit
The problem with all the games that you listed for alternatives is that they're all years old, so they hardly stress either card.

Bingo. No hidden motives necessary.

 

[Keysplayr]

There isn't anything wrong with HardOCP's benchmarking methods. It's only different. Details are given on each test as to what was turned down or not. To this day, nobody has proven their benches to be false or misleading. I can read and understand them easily. All I hear are complaints like, "it exacerbated ATI's AA problems.". The translation or non PR meaning of this would be, "It shows ATI's weakness running AA.". Apoppin and I have seen first hand the R600's problem with AA.

As far as playable framerates being arbitrary? According to who? You didn't trust the reviewer to understand what playable frame rates are for a given game? hehe.

 

[tuteja1986]

Firingsquad is very reliable source. Hardocp and Fudzilla are rubbish website.

 

[CP5670]

As far as playable framerates being arbitrary? According to who? You didn't trust the reviewer to understand what playable frame rates are for a given game? hehe.

Everyone has a different idea of what playable or smooth is. The concept is totally meaningless. I consider HardOCP's "playable" framerates to be slideshows. I see this as the biggest problem with their methods.

 

[n7]

Originally posted by: keysplayr2003
There isn't anything wrong with HardOCP's benchmarking methods. It's only different. Details are given on each test as to what was turned down or not. To this day, nobody has proven their benches to be false or misleading.

Of course not.
Kinda hard to when you review things in a completely messed up unreproduceable way.
So here we have apples, oranges, pears, some plums, & some bananas...

I used to actually read their reviews for information.

But then they decided to do everything "real world".
Problem with their idea of real world is that it's not reliable.

Like how when Intel's Core 2 Duo came out, what'd we see there?

A review of the two systems, an A64 X2 & a C2D @ 1600x1200, maxed graphics settings.
Their conclusion as i recall it was that it doesn't matter which high end CPU you have, as GPUs are the main limiting factor.
Well no sh!t sherlock :roll:

Anyway, HardOCP really isn't worth arguing about, so i'll shut up.
But one thing i don't want to have to keep seeing is more Wreckage + pro-nV reviews every few days.
It's getting very old, & everyone here knows his agenda.

 

[will889]

If you know what your looking at you can learn from the jades <sold out> sites, it's just that like one person mentioned above you have to read many sources before you can finally come to a basic conclusion about any given card.

 

[apoppin]

Originally posted by: keysplayr2003
There isn't anything wrong with HardOCP's benchmarking methods. It's only different. Details are given on each test as to what was turned down or not. To this day, nobody has proven their benches to be false or misleading. I can read and understand them easily. All I hear are complaints like, "it exacerbated ATI's AA problems.". The translation or non PR meaning of this would be, "It shows ATI's weakness running AA.". Apoppin and I have seen first hand the R600's problem with AA.

As far as playable framerates being arbitrary? According to who? You didn't trust the reviewer to understand what playable frame rates are for a given game? hehe.

i am still going to disagree with you on this one.

it is easy to portray either GPU in a better or worse light by not defining what is 'playable'. Yes, we have seen the "problem" with AA and it is pretty easy to manipulate the settings to make it look worse- or better - than it is - depending on the reviewer.

i will stick with "apples to apples" comparisons - everything compared at the SAME setting... HardOC's testing criteria or methods are NOT published and the settings are arbitrary - at the reviewer's discretion.

at the best, HardOCP is for novelty [and nvidia fans] ... no other credible site supports their extra-low figures for the 2900 or ANY AMD card.

 

[Extelleron]

I agree with the OP that HardOCP is not a reputable site and definately favors nVidia, but I don't agree about only testing those "popular" games.

I don't know about you, but I haven't touched Battlefield 2 in a while, it's been a month or so since I've played Doom 3, I haven't touched F.E.A.R. in quite some time either. I have, however, played STALKER in the last few days.

Sites should test with a wide variety of games based on different engines, but they should definately test newer games. I'm probably not going to go back and play a 2 year old game on my new video card, but I will play a new game like STALKER, so I care about its performance.

The reason HardOCP is not reputable is because their way of benchmarking allows for benchmarks to be "fixed" based on which card they like better. Each card has strengths and weaknesses. HardOCP knows the HD 2900XT sucks at AA, so they use AA in their tests (which I actually agree with.) Let's say the HD 2900XT sees a big drop from "High Shadows" and the 8800GTS sees a minor drop. However, the 8800GTS sees a huge drop from "Ultra Lighting" but the HD 2900XT sees a small drop. HardOCP will use "High Shadows", but not "Ultra Lighting". Their test method is flawed. Reputable sites should use MAXIMUM IN GAME SETTINGS and not fool around with individual settings in-game. It's just too hard for bias to take over when you can fool around with individual settings.

 

[will889]

Pops I agree, Apples to Apples is the real thing. You take the same identical setups and you use default drivers settings and benchmark the cards in various games and 3d apps at various common resolutions. It's not that hard of a concept, just hard work to get it up and published with relative comments to the products, and thanks to you and keys for doing just that.