Lately, I have been doing a lot of thinking (no, your car isn't burning oil, smarta$$!)
The way I see it, there are 3 main problems with the benchmark-centric graphics card reviews of today:
1) There is usually no way to measure "minimum" framerate
2) There isn't a standardized way to measure "average framerate under heavy stress"
3) Super-high framerates in non-detailed areas are given undue weight in a benchmark "fps average" score, when they don't really add anything to your gaming experience
As far as framerate is concerned, when does framerate matter while gaming? (gaming, not benchmarking)
It matters when it gets LOW. When we are maintaining a steady 100, 80 or maybe even 60 fps, we usually won't notice what our framerate is (although 60 is pushing it, IMHO). However, when we drop to 30 or 20 (blech!!!), our framerate becomes VERY VERY noticeable, kinda like a 3-year old throwing a tantrum. It gets our attention in a NEGATIVE way. Hitting 150 fps in a low detail scene does nothing for my gaming experience, but maintaining 45-50 fps in the heaviest of situations certainly does.
Are today's benchmarks really telling us how our systems will perform under heavy stress? I say "no" for the above 3 mentioned reasons.
re: # 1) - Not too many games today have a "minimum" framerate measure. Evolva and UT come to mind presently, but, I don't know of any OGL games *at all* that do. We haven't, in the past, had ANY that gave us an idea of how our systems performed when we needed them to, so we "resorted" to #2...
re: # 2) "Killer timedemos" - crusher.dm2 and massive1.dm2 used to be "the standard" for benchmarking. Why? Because if your graphics card could perform well under those stresses, then it could handle ANYTHING any OGL game (and most games, in general) would throw at you.
No one used q2demo1 as a benchmark, because it was "too weak" and didn't really represent the true strength of the card. Yet today, we are using "q2demo1-type" benchmarks for means of comparison. q3's demo001 and MDK2's default test really don't tell us a whole lot, do they? Having an average of 90 fps is nice, but that doesn't tell us that it includes those few seconds you jumped to 200+ fps in a room with no enemies (which didn't help you), it doesn't tell us about the time you dropped to 20 after gibbing that guy at point blank range (and then got fragged yourself cuz your system slowed to a crawl), nor does it tell you how your framerate will suffer in a close RA3 match in the castle map with a boatload of h0m0s trying to rocketjump off your head.
re: # 3) - When we do our benchmarking, today's "big 3" (5500, Radeon and GTS) are fast enough to break 100 fps quite easily in the vast majority of games at most resolutions and 125 fps in many situations. Now, I will argue tooth and nail against someone who says there's no difference between 60 fps and 30fps. They're grossly incorrect. I can tell quite easily the difference between 60 fps and 100 fps even. But past 100 fps, I start to lose track. Unless I have a framerate counter, I can't tell if I'm getting 150 fps or 125 fps or even 180 fps.
In Q3 demo001, for example, do you break 200 fps in the little corridor just before you gib that guy with the railgun? (/seta r_BraggingBitch 1) I do (/seta r_BraggingBitch 0). Does breaking 200fps help the smoothness of the game? Nope. Does it have a heavy weight in your final benchmarking score? Sure as hell does.
So what? Well, it goes without stating that maintaining framerate is one of the most important considerations when building your gaming rig. In-game, I want no hiccups or stutters. I want perfectly smooth gameplay.
Unfortunately, the way benchmarks are run today, it is no longer about "measuring smooth gaming", it's about "statistics" (of the "lies, damn lies, and..." fame)
(shaddup, I'm getting to the point!)
So what can we do about the 3 "issues" I've raised?
1) Not much we can do. Either the developers stick in the "minimum/average/maximum" framerate calculations, or they don't. If the game has it, then let's use it and report it, as that gives a MUCH better picture of performance than just a simple average framerate, IMHO.
2) Someone, preferably a large website host (HINT HINT HINT!!!) should develop one (or more) "killer demos" to be used for benchmarking, much like Brett Jacobs did with massive1 and crusher. If it's good, it'll become a standard. Just don't give yourself too much credit, or other websites won't want to use it <g>
When I was comparing graphics cards, I made my own demo. Q3tourney4, 10 bots, me on god mode (cuz I sUx0rZ), fraglimit of 50. It really told me QUITE a bit about how the cards performed when I needed them to. The level detail was high, the action was intense, and there was NO time when my framerate jumped real high, so I didn't get a 'biased' report of my framerate, I got a good representation of how my system performed in intense situations when I needed it to perform well for me.
The difference between an average of 80 fps and 90 fps on demo001 is minimal. The difference between averaging 30 fps and 40 fps on my homemade 'q3crusher' demo is another frag added to your total vs. respawning with the gauntlet and running frantically trying to find a "real" weapon. The killer demo tells you how your system will perform in worst-case scenario, when you really need it to come through for you.
3) In lieu of #1 and #2, let's do something "drastic". Let's benchmark with vsync ENABLED. yes, it goes against EVERYTHING we've ever thought, but if you have a monitor that can do 125 Hz, setting vsync means you are effectively capping your frames @ 125 fps. Does anyone REALLY NEED MORE???
Well, why bother capping your framerate @ 125?
Simple. A "lower low" can be made up for by a "higher high". The "lower low" has a negative effect on your gaming, but the "higher high" doesn't help out at all.
Simplified hypothetical example:
Card 1 - high = 145 fps
low = 25 fps
ave = 85 fps
Card 2 - high = 120
low = 50
ave = 85
Ideally, a benchmark will report that card 1 dropped down twice as low as card 2. Usually, all we'll see is the average score. The average might lead one to believe that both cards performed about equal, but that obviously isn't the case.
Now apply vsync.
Card 1 - high = 125
low = 25
ave = 75
Card 2 - high = 120
low = 50
ave = 85
By appying vsync, we give a more accurate report of the true noticeable performance of the card. Obviously the #'s are hypothetical and assumed, but you get the message (I hope)
Is this is good as being able to report a "high/low/average"? Probably not. Is this as good as having a "killer demo"? Probably not. But it's as close as it gets anymore.
This primarily applies to the "middle of the road" resolutions, like 1024x768 (depending upon the game), where you can have massively high spikes in low detail situations, but your framerate can drop rock-bottom in intense situations. The spikes don't affect your gaming at all, but the drops sure hurt it!
Another way to look at it is to measure how fast a car is. Hey, my little Dodge Neon can break 100 MPH while going down a nice hill no problem! Let's not give that too much weight when asking "can the car move, or what?"
Anyway, if you agree, then say so. If you disagree, then piss off. <G>
Now then, a request:
If you have one of the "big 3" cards (or hell, any card) and you have a monitor that can manage 125 Hz at 1024 and 1280, could you run some benchmarks with vsync enabled and report your scores? I'm wondering if this will reveal something interesting.
Q3, Evolva, MDK2, and UT all have downloadable demos with benchmarks, so even if you don't have the game, you can run the benchmark. Obviously, CPU and RAM type and speeds, mobo, OS, etc. are important side information.
Feedback appreciated, and discussion welcome, as always.
The way I see it, there are 3 main problems with the benchmark-centric graphics card reviews of today:
1) There is usually no way to measure "minimum" framerate
2) There isn't a standardized way to measure "average framerate under heavy stress"
3) Super-high framerates in non-detailed areas are given undue weight in a benchmark "fps average" score, when they don't really add anything to your gaming experience
As far as framerate is concerned, when does framerate matter while gaming? (gaming, not benchmarking)
It matters when it gets LOW. When we are maintaining a steady 100, 80 or maybe even 60 fps, we usually won't notice what our framerate is (although 60 is pushing it, IMHO). However, when we drop to 30 or 20 (blech!!!), our framerate becomes VERY VERY noticeable, kinda like a 3-year old throwing a tantrum. It gets our attention in a NEGATIVE way. Hitting 150 fps in a low detail scene does nothing for my gaming experience, but maintaining 45-50 fps in the heaviest of situations certainly does.
Are today's benchmarks really telling us how our systems will perform under heavy stress? I say "no" for the above 3 mentioned reasons.
re: # 1) - Not too many games today have a "minimum" framerate measure. Evolva and UT come to mind presently, but, I don't know of any OGL games *at all* that do. We haven't, in the past, had ANY that gave us an idea of how our systems performed when we needed them to, so we "resorted" to #2...
re: # 2) "Killer timedemos" - crusher.dm2 and massive1.dm2 used to be "the standard" for benchmarking. Why? Because if your graphics card could perform well under those stresses, then it could handle ANYTHING any OGL game (and most games, in general) would throw at you.
No one used q2demo1 as a benchmark, because it was "too weak" and didn't really represent the true strength of the card. Yet today, we are using "q2demo1-type" benchmarks for means of comparison. q3's demo001 and MDK2's default test really don't tell us a whole lot, do they? Having an average of 90 fps is nice, but that doesn't tell us that it includes those few seconds you jumped to 200+ fps in a room with no enemies (which didn't help you), it doesn't tell us about the time you dropped to 20 after gibbing that guy at point blank range (and then got fragged yourself cuz your system slowed to a crawl), nor does it tell you how your framerate will suffer in a close RA3 match in the castle map with a boatload of h0m0s trying to rocketjump off your head.
re: # 3) - When we do our benchmarking, today's "big 3" (5500, Radeon and GTS) are fast enough to break 100 fps quite easily in the vast majority of games at most resolutions and 125 fps in many situations. Now, I will argue tooth and nail against someone who says there's no difference between 60 fps and 30fps. They're grossly incorrect. I can tell quite easily the difference between 60 fps and 100 fps even. But past 100 fps, I start to lose track. Unless I have a framerate counter, I can't tell if I'm getting 150 fps or 125 fps or even 180 fps.
In Q3 demo001, for example, do you break 200 fps in the little corridor just before you gib that guy with the railgun? (/seta r_BraggingBitch 1) I do (/seta r_BraggingBitch 0). Does breaking 200fps help the smoothness of the game? Nope. Does it have a heavy weight in your final benchmarking score? Sure as hell does.
So what? Well, it goes without stating that maintaining framerate is one of the most important considerations when building your gaming rig. In-game, I want no hiccups or stutters. I want perfectly smooth gameplay.
Unfortunately, the way benchmarks are run today, it is no longer about "measuring smooth gaming", it's about "statistics" (of the "lies, damn lies, and..." fame)
(shaddup, I'm getting to the point!)
So what can we do about the 3 "issues" I've raised?
1) Not much we can do. Either the developers stick in the "minimum/average/maximum" framerate calculations, or they don't. If the game has it, then let's use it and report it, as that gives a MUCH better picture of performance than just a simple average framerate, IMHO.
2) Someone, preferably a large website host (HINT HINT HINT!!!) should develop one (or more) "killer demos" to be used for benchmarking, much like Brett Jacobs did with massive1 and crusher. If it's good, it'll become a standard. Just don't give yourself too much credit, or other websites won't want to use it <g>
When I was comparing graphics cards, I made my own demo. Q3tourney4, 10 bots, me on god mode (cuz I sUx0rZ), fraglimit of 50. It really told me QUITE a bit about how the cards performed when I needed them to. The level detail was high, the action was intense, and there was NO time when my framerate jumped real high, so I didn't get a 'biased' report of my framerate, I got a good representation of how my system performed in intense situations when I needed it to perform well for me.
The difference between an average of 80 fps and 90 fps on demo001 is minimal. The difference between averaging 30 fps and 40 fps on my homemade 'q3crusher' demo is another frag added to your total vs. respawning with the gauntlet and running frantically trying to find a "real" weapon. The killer demo tells you how your system will perform in worst-case scenario, when you really need it to come through for you.
3) In lieu of #1 and #2, let's do something "drastic". Let's benchmark with vsync ENABLED. yes, it goes against EVERYTHING we've ever thought, but if you have a monitor that can do 125 Hz, setting vsync means you are effectively capping your frames @ 125 fps. Does anyone REALLY NEED MORE???
Well, why bother capping your framerate @ 125?
Simple. A "lower low" can be made up for by a "higher high". The "lower low" has a negative effect on your gaming, but the "higher high" doesn't help out at all.
Simplified hypothetical example:
Card 1 - high = 145 fps
low = 25 fps
ave = 85 fps
Card 2 - high = 120
low = 50
ave = 85
Ideally, a benchmark will report that card 1 dropped down twice as low as card 2. Usually, all we'll see is the average score. The average might lead one to believe that both cards performed about equal, but that obviously isn't the case.
Now apply vsync.
Card 1 - high = 125
low = 25
ave = 75
Card 2 - high = 120
low = 50
ave = 85
By appying vsync, we give a more accurate report of the true noticeable performance of the card. Obviously the #'s are hypothetical and assumed, but you get the message (I hope)
Is this is good as being able to report a "high/low/average"? Probably not. Is this as good as having a "killer demo"? Probably not. But it's as close as it gets anymore.
This primarily applies to the "middle of the road" resolutions, like 1024x768 (depending upon the game), where you can have massively high spikes in low detail situations, but your framerate can drop rock-bottom in intense situations. The spikes don't affect your gaming at all, but the drops sure hurt it!
Another way to look at it is to measure how fast a car is. Hey, my little Dodge Neon can break 100 MPH while going down a nice hill no problem! Let's not give that too much weight when asking "can the car move, or what?"
Anyway, if you agree, then say so. If you disagree, then piss off. <G>
Now then, a request:
If you have one of the "big 3" cards (or hell, any card) and you have a monitor that can manage 125 Hz at 1024 and 1280, could you run some benchmarks with vsync enabled and report your scores? I'm wondering if this will reveal something interesting.
Q3, Evolva, MDK2, and UT all have downloadable demos with benchmarks, so even if you don't have the game, you can run the benchmark. Obviously, CPU and RAM type and speeds, mobo, OS, etc. are important side information.
Feedback appreciated, and discussion welcome, as always.
but at the same time it didnt have any unnecessary whining/ranting, eloquent and concise i must say 
Facebook
Twitter