Benchmarking Guidelines: After reading the 11ty billionth awful Video Card review ....i have a dream!

dug777

Lifer
Oct 13, 2004
24,778
4
0
*dug777 enjoys this pipe dream*

Seriously, most of the stuff DT links to makes your brain hurt :eek:

Maybe they need me to proof-read/work on their formats/review process...it really wouldn't be hard to produce an up to date/relevant review outline would it...;)

I've got one i knocked up the other day (roughly), the salient points are:

1. Clearly note all drivers/demos/settings/hardware used. Provide custom demos for download.
2. Benchmark a good range of heavily played games (both older and cutting edge).
3. Run the full gamut of resolutions with all cards.
4. Use the latest whql drivers, and get the review out the door quickly enough to make those driver results relevant.
5. Include the relevant competition at that price point.
6. DO overclock, with the associated YMMV warning, and provide results of said overclocking.
7. Ensure your format is clear, uncluttered, and logical (provide a clear article index on every page too)
8. Thoroughly vet the article for grammatical and spelling errors.



Those are the key points that i see ignored time and time again, any other suggestions?

:beer:

dug.
 

josh6079

Diamond Member
Mar 17, 2006
3,261
0
0
That's why an enthusiast/consumer normally has to search a broad amount of information in order to get the entire scope of a product. Multiple forum threads coupled with multiple reviews are really the only way to go, and I don't see anything changing soon.
 

Keysplayr

Elite Member
Jan 16, 2003
21,209
50
91
Hey Dug. Even if all your criteria is met for the benching, and you are totally happy and satisfied, there will be many others who will not feel the same way. As a reviewer, it is simply impossible to make everyone happy. A great job can be done on any given review that is very comprehensive, informative and done correctly, but there will always be someone saying, "I wish they would have done this, this way." And 11ty billion others with wishes of their own.
 

Munky

Diamond Member
Feb 5, 2005
9,372
0
76
My 2 major gripes with the reviews are not using AA/AF settings, and not using high quality driver settings. For example, AFAIK R3D did the only review of the g80 where they noted that by defauult it still does brilinear filtering, and thus switched to HQ settings. Not to mention all the benches that were done on the 7-series with shimmerfest settings. As for AA/AF, I have no reason to be interested in how a high end card of 2006 runs a modern game at 1999 settings, so benches w/o AA+AF enabled are just useless to me.
 

bobsmith1492

Diamond Member
Feb 21, 2004
3,875
3
81
However, AA feels like crap to me, so why would I want to know just about AA when I could get a slightly cheaper card and still drive my monitor at full resolution without it? Therein lies the benchmarker's dilemma. You simply cannot test 800x600, 1024x768, 1280x960, 1280x1024, 1440x900, 1680x1050, 1600x1200, 1920x1200, 1920x1080, 2056x1542, 3200x2400 and so on and so forth WITH 0xAA, 2xAA, 4xAA, 6xAA, 8xAA, xQ, SS, and 0xAF, 4xAF, 8xAF, 16xAF and all possible permutations in each of 5 new games, 10 modern games, and 5 classic games in any sort of relevant timeframe with new hardware.

Think of the number of runs... that's 11 resolutions, 5+ AA settings of 2-3 different types, at least 4 AF settings, and probably 20 games people would want. That comes to 13200 test runs, and for valid results, you really need at least 3 runs of each - that comes to 39600 benchmark runs. Now, use 4 classes of processors for the crowd that wants that, and 20 different video cards for each person that can't extrapolate between cards using other reviews, and you're up to a grand total of 3,168,000 test runs.

Well, that was an interesting rant - it's mainly for the people that complain "They didn't test the x1627PEx2 Pro at 1280x960 without AA! WHY???"
 

Munky

Diamond Member
Feb 5, 2005
9,372
0
76
I'd prefer to just test with 4xAA and 16x AF, and maybe with AAA/TRAA added if the reviewer was smart enough. No need to run low res tests at 800x600, and since 6xAA or 8xAA is not commonly supported on all cards, that's not necessary either for apples-to-apples benches.
 

CP5670

Diamond Member
Jun 24, 2004
5,639
738
126
I would like to see minimum framerate scores more than anything else. Most reviews don't show those, but they are far more important than the averages as far as I'm concerned. There are quite a few games out there that exhibit exceptionally low minimums even when the averages look good.
 

josh6079

Diamond Member
Mar 17, 2006
3,261
0
0
I believe reviewers like to run at least one AA/AF bench of each game and resolution to show the impact of AA/AF.

In either case, I think that 4xAA is already starting to be too low of an AA setting for the hardware we're seeing come out. Afterall, it has been the minimum amount of AA that most would use.
 

Hyperlite

Diamond Member
May 25, 2004
5,664
2
76
Originally posted by: bobsmith1492
However, AA feels like crap to me, so why would I want to know just about AA when I could get a slightly cheaper card and still drive my monitor at full resolution without it? Therein lies the benchmarker's dilemma. You simply cannot test 800x600, 1024x768, 1280x960, 1280x1024, 1440x900, 1680x1050, 1600x1200, 1920x1200, 1920x1080, 2056x1542, 3200x2400 and so on and so forth WITH 0xAA, 2xAA, 4xAA, 6xAA, 8xAA, xQ, SS, and 0xAF, 4xAF, 8xAF, 16xAF and all possible permutations in each of 5 new games, 10 modern games, and 5 classic games in any sort of relevant timeframe with new hardware.

Think of the number of runs... that's 11 resolutions, 5+ AA settings of 2-3 different types, at least 4 AF settings, and probably 20 games people would want. That comes to 13200 test runs, and for valid results, you really need at least 3 runs of each - that comes to 39600 benchmark runs. Now, use 4 classes of processors for the crowd that wants that, and 20 different video cards for each person that can't extrapolate between cards using other reviews, and you're up to a grand total of 3,168,000 test runs.

Well, that was an interesting rant - it's mainly for the people that complain "They didn't test the x1627PEx2 Pro at 1280x960 without AA! WHY???"

you do have a point. I agree with dug on some points, but then again it really would be impossible to please everyone.
 

Cookie Monster

Diamond Member
May 7, 2005
5,161
32
86
Well, i also like benches similiar to that of hardocp and bit-tech. Hardocp shows you the fps graph as well as low fps score, and also tests games at the highest playable settings. With the introduction of G80, the performance hits on 16xCSAA and 8xCSAA is so low that you could almost get away with those high AA levels in most modern games instead of enabling 4xAA. They give you the indication of the cards potentials.

However with differeing setups raging from CPU to memory to motherboard etc, benchmarks will always differ so its quite hard for the reviewer to test what is "right" for us.
 

CP5670

Diamond Member
Jun 24, 2004
5,639
738
126
Well, i also like benches similiar to that of hardocp and bit-tech. Hardocp shows you the fps graph as well as low fps score, and also tests games at the highest playable settings.

I don't really care for that style myself, as everyone has a different idea of what acceptable framerates are. I find most of the settings they suggest to be slideshows. :p However, the graphs that H displays are excellent. I think if a review showed several of those graphs at a range of settings and resolutions, it would be just about perfect.
 

dug777

Lifer
Oct 13, 2004
24,778
4
0
Originally posted by: keysplayr2003
Hey Dug. Even if all your criteria is met for the benching, and you are totally happy and satisfied, there will be many others who will not feel the same way. As a reviewer, it is simply impossible to make everyone happy. A great job can be done on any given review that is very comprehensive, informative and done correctly, but there will always be someone saying, "I wish they would have done this, this way." And 11ty billion others with wishes of their own.

I'm not entirely retarded and i can obviously appreciate that, but i think you miss the overall point i'm trying to make :p There are far too many reviews that don't provide the basics i mention in my OP, such as detailed driver settings, or testing the relevant competition, or not using graphs that make your eyes bleed...
 

Keysplayr

Elite Member
Jan 16, 2003
21,209
50
91
I hear ya. Didn't mean to imply that you didn't grasp anything. Yeah, some of those graphs are not even readable. Annoying to look at. That's one thing I wish they would never do again.
 

BenchZowner

Senior member
Dec 9, 2006
380
0
0
dug777 are you reading my mind boy ? :D
Most ( if not all ) of the things that you mentioned in your first post, made me re-think of my last decision on a bad decision of mine a year ago :)
You got some good points, and I'd like to add some more, but atm I'm really tired and will have to drop off.
But I'll add some later.

Keep your eyes open, there are some good things coming soon :)

[ I'm not related to Anandtech, and I'm not talking about anandtech ofcourse ]