Anyone feeling disenchanted with (most) review sites?

[Lonyo]

It can't be just me, but there are lots of problems and limits to a lot of reviews on most sites.
Now, obviously reviews take time and effort etc, but a lot of sites are "lacking" in their reviews, and I can't think of a single review site which is 'perfect' and offers up everything.

I'm going to mention a few 'issues' I have with various sites, but I doubt this is an exhaustive list, and maybe we can air out some of our grievances and maybe some reviewers might even care!

In no particular order, and not an exhaustive list:
Most websites: Not including "enough" cards in their reviews. E.g. not including things like SLI/CF mid-range (GTX460/HD6800 cards) in their reviews of the GTX580 (to cite a recent example of this). 90% of the websites I looked at didn't include mid-range SLI/CF results with their GTX580 reviews.

Using older drivers for older cards in their reviews (e.g. Techpowerup, for both AMD and NV cards. http://www.techpowerup.com/reviews/NVIDIA/GeForce_GTX_580/4.html)

Xbitlabs: Some 'weird' things going on with some reviews: taking down all the results from the HD6800 reviews for a couple of days.
The Fallout: New Vegas review having suspect figures for the HD6850.

Most sites: Lacking discussion of image quality and IQ issues (such as with the 10.9 vs 10.10 AMD cards), as well as general IQ comparison (e.g. Anandtech only looked a little bit at AF filtering in the HD6800 review), and most sites only namedropped morphological AA (MLAA) without looking at it in detail (and talking about the many problems it has).

Many sites: Lacking minimum frame rates for most games/cards. Obviously a number of sites do try to include minimum FPS, but all too often it's lacking.

Number of tests/choice of tests: Many sites don't test many games (some as few as about 6 IIRC), or they do specific tests which again comes back to IQ issues.
They run tests at specific settings, and don't analyse features, e.g. how cards scale with different AA settings. AT is one example of being guilt of this, for instance Crysis Warhead benchmarks. All with "Gamer" settings, no tests with higher quality shaders, and all with 4xAA.

Multi-GPU testing: often doesn't do much in depth discussion, e.g. talking about potential issues such as microstutter. If you read the AT GTX580 review, you would think that the GTX580 SLI was perfect (maybe it is), but that's not really talked about. Maybe it should be "common knowledge", but "common knowledge" changes as drivers and hardware changes and (hopefully) improves.


When it's come to (specifically for me) HD6800 reviews, there was a lack of depth on any given site, and even when looking over many sites it wasn't all that much more helpful.

It feels like a lot of sites aren't really giving the full picture of everything. Maybe I just miss the days of paper launches when we got info about cards a long time before release, and websites had more time to benchmark and analyse the cards.
Anyone else feel on the whole hardware sites are letting us down in many ways?

 

[i990]

some did take money from manufacturer
it's proverbial and well-known

 

[happy medium]

I heard Appopin complain about not given the cards in time to give them a indepth review, mabe thats part of it?

i think mabe Ryan here at AT was waiting for 6850's also?

I do agree though, it seems I have to read 3 reviews to get my satisfaction and to make a conclusion.

 

[Aristotelian]

I've been feeling this way for a while. I think it started getting bad with a few of the [H] reviews that were done over 2-3 games. I realize that it takes a long time, but I presume that the reviewers who review on Anandtech etc. are paid to do so. I don't see a reason why, when the hardware is free especially, for the reviews not to be far more thorough as you suggested.

 

[Lonyo]

When we kind of had a mid-range SLI/CF vs single high end card thread, I checked I think maybe 5 sites. Anandtech was the only one which had 6800/GTX460 CF/SLI on the same graph as a GTX580, for everything else you had to check maybe 2 or 3 different reviews that site had done for those numbers (e.g. the GTX580 review for the GTX580 numbers + the 6800 review for CF numbers).

Stupid. They had the numbers, they just hadn't got them all together.

 

[blastingcap]

AT and HardOCP are all I need to know

 

[ronnn]

Probably hardocp is the best because you just get a nice subjective opinion - which is what the rest seem to be anyways.

The other issue is that even midrange cards perform so well, that we are now defining the best by marginal differences or games are mainly console crap - so one blocked out comic book looks much like any other - unless they add a few chainlink fences to dummy down the comparison.

 

[blastingcap]

[H] for the playthroughs and fps timecharts, AT for the repeatability (even if they are canned benchmarks) and in-depth architectural commentaries.

The other sites are posers compared to them.

 

[apoppin]

I heard Appopin complain about not given the cards in time to give them a indepth review, mabe thats part of it?

i think mabe Ryan here at AT was waiting for 6850's also?

I do agree though, it seems I have to read 3 reviews to get my satisfaction and to make a conclusion.

i find it very difficult to get a thirty page review out - TWELVE THOUSAND WORDS; using 25 benchmarks with 6 card configurations at two or three resolutions - all with FRESH benchmarks - in 7 days from receipt of card to publication when NDA ended

The OP and the rest of you need to realize that i spent 20 hours a day for 7 days working on it. Do you guys work this hard at your own jobs?

Did you want more?
- i'm working on cloning myself
:biggrin:

 

[RussianSensation]

[H] for the playthroughs and fps timecharts, AT for the repeatability (even if they are canned benchmarks) and in-depth architectural commentaries.

The other sites are posers compared to them.

I disagree with that. Hard OCP's videocard reviews are too subjective. For example, I am not happy playing a racing game below 60fps, but I am perfectly fine at 35 fps in Crysis. Kyle might be OK with 45 fps in Dirt 2 but may want 60 fps in Crysis.

I want average and minimum framerates at the same settings and I will decide myself what I consider "playable". Because he tests videocards at completely different settings to arrive at what he considers playable (not what I consider playable), his tests have questionable value. What is the point of testing high end cards only at 2560x1600 4AA in Metro, to get 25-30 fps? That's not how I am going to play the game. I am going to run at with Tessellation at 1680x1050 or with noAA/no tessellation at 1920x1080 and HardOCP is not going to provide with me either of these scenarios!! HardOCP is getting way too lazy. They never test enough games.

They got a 1.7 fps disadvantage with HD5870 compared to GTX480 at 2560x1600 with Tessellation ON in Metro 2033 and only 2.2 fps disdvantage in Civilization 5. Those results are completely off. It takes 5 min to check TechSpot, LegionHardware, Xbitlabs, Tom's Hardware, Tech Report, etc. to see that GTX480 is far faster than HD5870 in both of those games.

 

[Teizo]

Probably hardocp is the best because you just get a nice subjective opinion - which is what the rest seem to be anyways.

Subjective = opinion. Just give me numbers, stfu, and let the cards speak for themselves. I can't tell you how many times I've read some reviewer say one card blows another card away...and the difference is only 2-3 fps, some times even as low as .5-1.5..(which are made to appear worse by misleading graphs). I mean, ffs...that's plain insulting to the reader's intelligence.

 

[cusideabelincoln]

[H] for the playthroughs and fps timecharts, AT for the repeatability (even if they are canned benchmarks) and in-depth architectural commentaries.

The other sites are posers compared to them.

I like BFG's, of AlienBabel, high quality tests in older games. But the reviews do suffer from a couple of Lonyo's complaints. One being he only tests a few cards at a time. It would be nice to see what a mid range card from each vendor can do in comparison to high end cards.

 

[blastingcap]

I disagree with that. Hard OCP's videocard reviews are too subjective. For example, I won't play a racing game below 60fps. Kyle might be OK with 35 fps. I want average and minimum framerates at the same settings and I will decide myself what I consider "playable". Because he tests videocards at completely different settings to arrive at what he considers playable (not what I consider playable), his tests have questionable value. What is the point of testing high end cards only at 2560x1600 4AA in Metro, to get 25-30 fps? That's not how I am going to play the game. I am going to run at with Tessellation at 1680x1050 or with noAA/no tessellation at 1920x1080 and HardOCP is not going to provide with me either of these scenarios!! HardOCP is getting way too lazy. They never test enough games.

HardOCP also has apples to apples graphs at the bottom.

And I believe you were lobbying hard for the case that only the heavy hitters need be considered for reviews since the lightweight games are going to be maxed out by modern cards anyway.

Although to be fair you did criticize them for benching some lighter games. But what do you expect, they asked readers what games they'd like to see benched and some of the popular choices were for lighter games. Were they going to just dismiss their readers' popular choices?

Anyway, between AT and HardOCP--note I didn't say read only one and not the other--I know pretty much all I need to know. The other sites are for filling in the odd hole here or there, like an SLI comparison or oc'ing stats or stuff like that.

 

[RussianSensation]

And I believe you were lobbying hard for the case that only the heavy hitters need be considered for reviews since the lightweight games are going to be maxed out by modern cards anyway.

Agreed, but HardOCP misses some big games like AvP, Just Cause 2, STALKER:CoP and instead chooses to test Medal of Honor at 2560x1600..... I wish HardOCP included more resolutions and more games though.

I mean take a look at their GTX580 review - every game tested at 2560x1600 resolution...I don't exactly own a 2560x1600 monitor :biggrin: So what am I going to take away from that review exactly? F1 tested at 2560x1600 at 35-39 fps.... How meaningful are those results for PC racing fans? 35 fps "playable" in a racing game on a PC is an insult to anyone who plays racing games. He considered 38.4 and 40.4 fps playable on 5870 and GTX480, respectively, at 2560x1600 in BF:BC2. I am pretty sure people who play BF:BC2 in multiplayer would not consider that playable.

 

[blastingcap]

Agreed. I wish HardOCP included more resolutions and more games though. I don't exactly own a 2560x1600 monitor :biggrin:

Me neither but I know what you mean, 1680x1050 and 5040x1050 need more love.

 

[RussianSensation]

I think Lonyo and blastingcap should start their own review website. It would be have everything, time graphs, minimum frames, eyefinity, 8AA, SSAA/TrSS, DX9/10/11 games averages split and most importantly, videocard round up on a game-by-game basis

 

[cusideabelincoln]

Agreed, but HardOCP misses some big games like AvP, Just Cause 2, STALKER:CoP and instead chooses to test Medal of Honor at 2560x1600..... I wish HardOCP included more resolutions and more games though.

A lot of other sites test those three games.

We don't need every site testing the same games. It's good to have different sites testing different games, since it is unfeasible for a site to test every game.

 

[dust]

If time is an issue before the launch then why can't the reviewer go back to the issue later on, adding another week of work to the previous one should probably get the work done completely.

I call BS on the whole thingy, we live in a world driven by the tip, you can't get anywhere and do anything without bribes.They might be able to survive on cheers from the readers but money is just much better. As long as they use a shaded approach in the reviews none can blame them outright for bias, one can only speculate, but hey there are giveaways, just like it was mentioned here before - not including CF/SLI results with the top of the line GPU, using old drivers, etc. These are all signs of bias and saying the time is of the essence is not really an excuse, since it should matter for the integrity of the review.

 

[apoppin]

I think Lonyo and blastingcap should start their own review website. It would be have everything, time graphs, minimum frames, eyefinity, 8AA, SSAA/TrSS, DX9/10/11 games averages split and most importantly, videocard round up on a game-by-game basis

Agreed. i want to see one of you guys do this with a brand new card - before it becomes legacy


Make sure you also test Surround and i would also like to see some 3D Vision results. And don't forget to test over and underclocks as well as LOWER the voltage for the "green" guys. You should also run the games at several clock speeds and use Core i7 vs Phenon II and don't forget to run dual core benches vs. quad.
:awe:

If time is an issue before the launch then why can't the reviewer go back to the issue later on, adding another week of work to the previous one should probably get the work done completely.

i do only when i have time; i broke my original GTX 480 vs HD 5870 review into 3 parts; stock, overclocked testing, and with 8xMSAA vs 4xAA

Usually there simply is too much new HW to review to go back and work on old ones. Right now i am writing an article - an important one - that was done with Cat 10-6! And today i got a 128 GB SSD from Kingston to evaluate. And i got a follow up coming to my HD 68x0 vs GTX 460 review coming out next week (EVGA FTW vs Galaxy OC GTX 460; including SLI). Not to mention an interview with AMD that should be up on Wednesday.
- besides my regular duties as a senior editor

When should i return to it? Only when i get another GTX 580 for SLI testing

 

[Paratus]

i find it very difficult to get a thirty page review out - TWELVE THOUSAND WORDS; using 25 benchmarks with 6 card configurations at two or three resolutions - all with FRESH benchmarks - in 7 days from receipt of card to publication when NDA ended

The OP and the rest of you need to realize that i spent 20 hours a day for 7 days working on it. Do you guys work this hard at your own jobs?

Did you want more?
- i'm working on cloning myself
:biggrin:

Cloned Apoppin?
So would that be Apoppin Crossfire or SLI? Maybe an X2 card?

 

[manimal]

Me neither but I know what you mean, 1680x1050 and 5040x1050 need more love.

The problem with those resultions is for the most part all the titles out there run well enough on midrange cards.

Minimum frame rates are all that matters to me. I dont care if two cards both go over 100 fps and another does 130... means nothing.....

besides 1080P is soo cheap these days there is no reason to play on anything smaller...and if you cant afford a 1080P monitor than what does it matter to you if a high end card performs well at that res?

Midrange cards need tests below 1080P.

I fell bad for suckers that buy GTX 580s and run them on 19 inch monitors......because they are suckers....

 

[happy medium]

i find it very difficult to get a thirty page review out - TWELVE THOUSAND WORDS; using 25 benchmarks with 6 card configurations at two or three resolutions - all with FRESH benchmarks - in 7 days from receipt of card to publication when NDA ended

The OP and the rest of you need to realize that i spent 20 hours a day for 7 days working on it. Do you guys work this hard at your own jobs?

Did you want more?
- i'm working on cloning myself
:biggrin:

I was actually defending you and Ryan.
I kinda like the way you split the reviews up and I really like your driver reviews.

 

[happy medium]

Cloned Apoppin?
So would that be Apoppin Crossfire or SLI? Maybe an X2 card?

Na Frakenpoppin. he he he he

 

[apoppin]

I was actually defending you and Ryan.
I kind like the way you split the reviews up and I really like your driver reviews.

i noticed. Thank-you.

i just went into detail for everyone who thinks a reviewer has an easy time of it when HW is under NDA. i am trying to work with both Nvidia and AMD to make it easier on themselves and on the reviewers.

You cannot believe how hard their PR guys work at card launch. The entire company comes together to pull it off; it is quite amazing. Some of the PR guys at the major companies deal with 100 tech sites from all over the world.

And i know for sure they hate reading my long reviews


.. and since it is uh, new, tech .. it would be cloneapoppin

 

[3DVagabond]

Me neither but I know what you mean, 1680x1050 and 5040x1050 need more love.

AMD needs to donate 3* 1900*1080 monitors to the top 25 review sites (or top 10 No. American, European, and Asian sites) and insist they test their cards in eyefinity. Not a very big expense, for the coverage it would generate. If AMD doesn't care enough though, what can you do?