-
We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.
A minimum average framerate petition Edit:HTML Graphs!! Anand's still might be watching. Keep it going!
Page 4 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.
Barnaby W. Füi
Elite Member
huh? excel document? i'm pretty sure excel can export to html, no? wouldnt that be a better choice?
Shalmanese
Platinum Member
I think a better measure would be the 10th percentile frame rate rather than the minimum, ie, at what fps does the game perform better than 90% of the time. That way, you eliminate situations where the game can drop to 2 fps for 1 single frame and then stay at 100 fps the rest of the time. Its pretty common practice to use the 3/5/10th percentile rather than the absolute minimum.
So, if you were playing a timedemo for 40s and the 10th percentile fps was 30, that means there was 4 seconds where it dropped under 30fps. I dont quite know how chaotic the frame rates are so perhaps a 5th percentile would be better.
So, if you were playing a timedemo for 40s and the 10th percentile fps was 30, that means there was 4 seconds where it dropped under 30fps. I dont quite know how chaotic the frame rates are so perhaps a 5th percentile would be better.
Woodchuck2000
Golden Member
No offence FishtankX, but it seems ludicrous to overlay two FPS/Time graphs and start pulling definitive conclusions about why cards dip in certain places from that. How exactly can we deduce that a spike in the FPS is caused by poor occlusion culling or bad prefetch algorithms?
And you still havn't countered the main point that we dissenters are arguing;
Minimum Framerate is important in real world gaming. Benchmark scores do not relate directly to real world scores. It is therefore impossible to draw any conclusions about the nature of the real world performance from benchmarks.
BTW, why is the thread title "A Minimum Framerate Petition" When you're not asking for minimum framerates to be shown any more?
And you still havn't countered the main point that we dissenters are arguing;
Minimum Framerate is important in real world gaming. Benchmark scores do not relate directly to real world scores. It is therefore impossible to draw any conclusions about the nature of the real world performance from benchmarks.
BTW, why is the thread title "A Minimum Framerate Petition" When you're not asking for minimum framerates to be shown any more?
okay, dug, maybe I didn't explain it well enough last time. I'll try and be *really* clear now, because i'm famous for having to explain things twice. >.<
Here we go.
during huge battles, everyone knows that the framerate is gonna drop. That's a given. Thus, you would see a drop in framerate that is *normal*.
If you put the results of 2 cards into the same graph, like you used to see with the CPU and linpack results, you can see where one card might be faltering and one might be puttering along at full speed. At the same spot. That would give you some idea of an anomoly. That's relativley easy to do, to put the results of 2 card's into the same graphs. And it would definatley shed light on expectable minimum framerates, any anomolies (like one card crashing where another is still going strong, which would point to either a driver bug or some Hyper-Z implmenetation helping out, or color compression or whatever), etc.. the key is comparing two cards with graphs ontop of one another. That would be very useful.
The way I see it, just looking at the results by themselves provide very little useful information. Thus a comparison is necescescary. A comparison is what will really make these graphs significantly more useful than numbers. Minimum framerate numbers by themselves, while midly useful, aren't exactly what i'm looking for. I'm looking for a linpack style comparison between graphics cards, CPU's, and maybe even drivers and platforms. This is the goal of this petition. Not specifically to provide numerical results (although those would natrually be thrown in, although I would believe it would be most useful if the highest and lowest numbers were thrown out, IMHO) but rather to provide numerical results that add to what the graphs already tell us. Min, max, average, mean... (which would all be trivial to drive from the fraps dump) with a graph, for each card. That way we could get some real in depth analysis, which wouldn't normally be possible.
Here we go.
during huge battles, everyone knows that the framerate is gonna drop. That's a given. Thus, you would see a drop in framerate that is *normal*.
If you put the results of 2 cards into the same graph, like you used to see with the CPU and linpack results, you can see where one card might be faltering and one might be puttering along at full speed. At the same spot. That would give you some idea of an anomoly. That's relativley easy to do, to put the results of 2 card's into the same graphs. And it would definatley shed light on expectable minimum framerates, any anomolies (like one card crashing where another is still going strong, which would point to either a driver bug or some Hyper-Z implmenetation helping out, or color compression or whatever), etc.. the key is comparing two cards with graphs ontop of one another. That would be very useful.
The way I see it, just looking at the results by themselves provide very little useful information. Thus a comparison is necescescary. A comparison is what will really make these graphs significantly more useful than numbers. Minimum framerate numbers by themselves, while midly useful, aren't exactly what i'm looking for. I'm looking for a linpack style comparison between graphics cards, CPU's, and maybe even drivers and platforms. This is the goal of this petition. Not specifically to provide numerical results (although those would natrually be thrown in, although I would believe it would be most useful if the highest and lowest numbers were thrown out, IMHO) but rather to provide numerical results that add to what the graphs already tell us. Min, max, average, mean... (which would all be trivial to drive from the fraps dump) with a graph, for each card. That way we could get some real in depth analysis, which wouldn't normally be possible.
And as for Woodchuck2000's insistnace that benchmarks have no real bearing on real world gamplay, I would think that would be what demo recorders are for, no?
And I didn't specifically say that you could pull a conclusion from a drop in framerate. But knowing about anomolies is the key to getting them solved. And further investigation into the issue, like, say, turning down the texture detail eleminated the anomoly, would automatically point to thrashing in one card. Or turning down the geometry detail, vastly boosting it's scores would be the indicator of a poor T&L implmentation... etc..
Example:In the future, R300 beats the NV30 in AA benchmarks when you take averages. But the NV30 seems to have a much higher framerate in intense scenes (Which would probably be labeled on the graph). This might point to the NV30 having a more effective color compression engine. Espically if turning down the texture detail evens it up. It might also point to the R300 texture thrashing. While this isn't anything concrete, it's something to consider, it's there, and it has a relevant impact on your purchase decision. While the reasons of the anomoly are not known, it's useful to know because if you use FSAA alot that means something to you.
While I don't that anand would take the time to do graphs for every game, minimum framerates (in percentiles, to keep from throwing off results, like Shalmanese said) would be a good adition to average framerates, to see which card pulls the bottom line better. And graphs would help in analaysis of strange behavior of a card, and detect anomolies. Alot of the reason why people say that Nvidia used to give a better gaming experience is because ATi's cards used to have erratic framerates due to a bug. That wasn't detected in your run of the mill benchmarks.
Anyways, i'm not gonna argue the point anymore, because if Anand chooses to include them for their worthiness, i'll be happy. If he decides they're worthless, i'll respect his decision. But I think it's a valuable thing to create a petition so that people can voice their ideas about how useful such a thing would be.
P.S. In reply to your question about what peak minimum FPS in a benchmark tells you, it tells you how low a certian card will drop in heavy action. And if the lentgh of the drop is significant, but one card has higher minimum framerate (possibly due to better Hyper-Z in overdraw laden areas) and one has higher maximum framerate (possibly due to geometry engine strentghs, or other things that could possibly cause something to have a higher max) and their average is close, it's easier to pick out which one you would rather have when the action got rough. I know I would rather have the card that had the lower average but the much higher performance in intense situations. Wouldn't you? Right now, we have no way of knowing which one is which.
And in reply to it's title, while graphs are useful, they're more time consuming. While minimum framerates are easy. It'd be much easier to implmenet minimum framerate's and I don't know if it's worthit to anand to implmenet graphs. I don't know if Anand will implmenet graphs, but if we can get minimum framerates in every benchmark, and one or two graphs, it'd be worthit to me. And essentially, minimum framerate is the most important part of the graphs. So the goal of this petition hasn't changed too much.
And I didn't specifically say that you could pull a conclusion from a drop in framerate. But knowing about anomolies is the key to getting them solved. And further investigation into the issue, like, say, turning down the texture detail eleminated the anomoly, would automatically point to thrashing in one card. Or turning down the geometry detail, vastly boosting it's scores would be the indicator of a poor T&L implmentation... etc..
Example:In the future, R300 beats the NV30 in AA benchmarks when you take averages. But the NV30 seems to have a much higher framerate in intense scenes (Which would probably be labeled on the graph). This might point to the NV30 having a more effective color compression engine. Espically if turning down the texture detail evens it up. It might also point to the R300 texture thrashing. While this isn't anything concrete, it's something to consider, it's there, and it has a relevant impact on your purchase decision. While the reasons of the anomoly are not known, it's useful to know because if you use FSAA alot that means something to you.
While I don't that anand would take the time to do graphs for every game, minimum framerates (in percentiles, to keep from throwing off results, like Shalmanese said) would be a good adition to average framerates, to see which card pulls the bottom line better. And graphs would help in analaysis of strange behavior of a card, and detect anomolies. Alot of the reason why people say that Nvidia used to give a better gaming experience is because ATi's cards used to have erratic framerates due to a bug. That wasn't detected in your run of the mill benchmarks.
Anyways, i'm not gonna argue the point anymore, because if Anand chooses to include them for their worthiness, i'll be happy. If he decides they're worthless, i'll respect his decision. But I think it's a valuable thing to create a petition so that people can voice their ideas about how useful such a thing would be.
P.S. In reply to your question about what peak minimum FPS in a benchmark tells you, it tells you how low a certian card will drop in heavy action. And if the lentgh of the drop is significant, but one card has higher minimum framerate (possibly due to better Hyper-Z in overdraw laden areas) and one has higher maximum framerate (possibly due to geometry engine strentghs, or other things that could possibly cause something to have a higher max) and their average is close, it's easier to pick out which one you would rather have when the action got rough. I know I would rather have the card that had the lower average but the much higher performance in intense situations. Wouldn't you? Right now, we have no way of knowing which one is which.
And in reply to it's title, while graphs are useful, they're more time consuming. While minimum framerates are easy. It'd be much easier to implmenet minimum framerate's and I don't know if it's worthit to anand to implmenet graphs. I don't know if Anand will implmenet graphs, but if we can get minimum framerates in every benchmark, and one or two graphs, it'd be worthit to me. And essentially, minimum framerate is the most important part of the graphs. So the goal of this petition hasn't changed too much.
Woodchuck2000
Golden Member
You are yet to define minimum frame-rate. If calculated using the lowest decile as shalmanese suggests, they will have some bearing on the consistency of the card in a specific benchmark.
You're drawing buzzword-laden conclusion from a hypothetical situation within a hypothetical situation. Without masses of tests eliminating certain factors, there's no way to tell if a 2 second drop in frame rate is due to a system Anomaly, a flaw in the Game Engine, Driver issues, or indeed a problem with the card.
BFG10K
Lifer
Minimum framerate is x/t where x (framerate) is as low as possible and t (time) is as high as possible.
Exactly and during the hectic action is when you're most likely to get your minimums which is why both minimums and averages are important during benchmark runs.
Averages show how well your rig does overall in a given situation.
Minimums show how poorly your rig does when things get tough, and whether it's powerful enough to stay above the required threshold of smoothness.
Because it gauges overall performance over a specific situation in actual gameplay. While I agree that benchmarks are only a small window into the actual gameplay (case and point: the effects of 64 MB VRAM over 128 MB VRAM - benchmarks don't show the whole story), it is possible to get really good ones by running very long tests with a wide range of diverse situations and/or to run a wide range of different benchmarks. Serious Sam in particular does this very well because it has six benchmarks to try and all of them are quite comprehensive.
Also when testing it's best to aim for the worst case scenarios and deliberately go after the situations that will slow your system down. That way you never delude yourself into thinking your system is running a game better than it really is. A 60 FPS average on an easy map will absolutely blow on a tough map which is yet another reason why the "you only need X FPS arguments are flawed".
I dunno how relevant min FPS measurements are. If I'm pulling 60fps except for a 10 second drop to 30fps once in a while, I think the 60fps number is far more relevant than the 30.
Besides, most of the time the minimum framerate is due to the CPU, not the video card, and even the fastest card in the world can't keep up a minimum fps without dropping when paired with a slow cpu.
Besides, most of the time the minimum framerate is due to the CPU, not the video card, and even the fastest card in the world can't keep up a minimum fps without dropping when paired with a slow cpu.
Woodchuck2000
Golden Member
From a purely mathematical point of view, X/T where X is FPS (unit Frames*S^-1) and T is Time (Units S) would have units Frames*S^-2. Given that framerate already includes a time component (being Frames*S^-1,) How is dividing FPS by T (without specifying which time either) going to produce a meaningful result?
I'm yet to receive a decent explanation of what minimum FPS represents, unless it is simply the lowest (peak) value that the FPS reaches i.e. 1/t (t being the longest time taken to render any single frame.)
PS. this isnt' meant to be insulting/inflammatory, I simply don't understand the definition you've given.
Allright. I'd like to present an example of a useful metric of minimum framerate.
I'd like to present minimum framerate as a possible average of the lowest 10% (framerate wise) of the sequence.
Thus, you'd take the lowest 1/10th of the total (or the 10% margin) and average it. There you go. A minimum framerate average. While this is still an average, yes, it shows the worst case situation in a given benchmark. Just the slowest frame [to draw] itself might be deceptive, so you must have some sort of an average.
This would be a pretty useful metric, given it's much more important to know how a card will perform in the most hectic of situations, than to know the average of the whole situation. Honestly, if you can push that number over 50 in the most intense situations, you'll be allright in the rest of the game. Because honestly I believe the most intense situations in benchmarks are far worse than what you would encounter in any form of real gameplay.
Do you have any objections, woodchuck? Hopefully this metric, if included into our benchmarks, would give us some sort of an idea on how a card would hold up during the worst of firefights. And frankly, that, with the average of the whole sequence, with your normal statistical mean, mode, etc.. would be more than enough for me to judge a card's performance merits.
I'd like to present minimum framerate as a possible average of the lowest 10% (framerate wise) of the sequence.
Thus, you'd take the lowest 1/10th of the total (or the 10% margin) and average it. There you go. A minimum framerate average. While this is still an average, yes, it shows the worst case situation in a given benchmark. Just the slowest frame [to draw] itself might be deceptive, so you must have some sort of an average.
This would be a pretty useful metric, given it's much more important to know how a card will perform in the most hectic of situations, than to know the average of the whole situation. Honestly, if you can push that number over 50 in the most intense situations, you'll be allright in the rest of the game. Because honestly I believe the most intense situations in benchmarks are far worse than what you would encounter in any form of real gameplay.
Do you have any objections, woodchuck? Hopefully this metric, if included into our benchmarks, would give us some sort of an idea on how a card would hold up during the worst of firefights. And frankly, that, with the average of the whole sequence, with your normal statistical mean, mode, etc.. would be more than enough for me to judge a card's performance merits.
Cerb
Elite Member
Here's the thing in a pretty simple way:
-You know average framerates.
-You know they aren't what you'll get.
-On the other hand, you know how it will relate to what you'll get compared to what you have now.
Min/Max FPS gives the same info, just more of it. I don't know about most of you, but I'd rather get a range of 40-80 FPS than 10-110, even though they both average to 60.
-You know average framerates.
-You know they aren't what you'll get.
-On the other hand, you know how it will relate to what you'll get compared to what you have now.
Min/Max FPS gives the same info, just more of it. I don't know about most of you, but I'd rather get a range of 40-80 FPS than 10-110, even though they both average to 60.
powerMarkymark
Platinum Member
Sounds good to me.
Woodchuck2000
Golden Member
It was the use of the word framerate that had me confused...
So it should have read:
Minimum framerate is x/t where x (frames rendered) is as low as possible and t (time) is as high as possible.
Now the lowest possible x is 1
So minimum framerate = 1/t where t is the longest time to render any given frame.
😀.
Woodchuck, no one is suggesting that the average FPS be removed from reviews. How could it possibly hurt things to add this in conjunction with the average framerate? I say that a standard deviation should also be part of the results. This would really give a lot of meaning to the average.
TRENDING THREADS
-
Discussion Zen 5 Speculation (EPYC Turin and Strix Point/Granite Ridge - Ryzen 9000)- Started by DisEnchantment
- Replies: 25K
-
Discussion Intel Meteor, Arrow, Lunar & Panther Lakes + WCL Discussion Threads
- Started by Tigerick
- Replies: 25K
-
Discussion Intel current and future Lakes & Rapids thread- Started by TheF34RChannel
- Replies: 24K
-
-
