I'd like to make a petition for minimum framerate measurmenets in videocard comparisons. For credit, i'll credit Intim83 for the idea.
I'd also like to reccomend methods outlined in My thread.
Time demos in conjunction with fraps wouold provite very good measurements, provided that some degree of acuracy in starting and stopping of the benchmark could be maintained. The data could also be extrapolated into graphs (Right now FRAPS has 1 second one measurement resolution textfile dumps that could be used for making graphs.) and stuff, while it might be a bit more work, I'd say that using fraps you could get acurate average, min, max, and graph results for the upcomming GeforceFX versus R300/R350 articles.
Everyone, voice your opinion here. Maybe i'll E-mail anand to keep an eye on this thread, and maybe we'll get some results.
Edit:Anand has been alerted of this thread since it's creation. Your votes most likely *will* count.
Edit:Stuff below
RAW framerate dump
Detailed and cleaned framerate dump with markers
What everyone has been waiting for. GRAPHS!
Now, to give you a quick analaysis of the data within
System specs:
Williamette 1.7
Baracuda 4 40GB
Alot of windows open!! (6) so the data isn't meant to be absolutley acurate
256MB of RAM
Radeon8500LE 64MB
Graphs are at the right of the excel document.
Time is in seconds. Measurements of average framerate are taken every second.
Romp was just that, a quick deathmatch in UT2003. It's about 10 minutes long. The low resolution of fraps (1 second) hurts the graph by making it look chaotic. Long demos are unacceptable for this testing methodology.
Flyby's:They look very rough, as they're only about 10 seconds, and thus, 10 samples long and alot of the graph is interpolation if you'll notice. With low resolution, extremley short demos
Botmatches:These are much better looking, and show the rather chaotic nature of bot matches. Framerate jumps wildly. It seems that demos 30 or 40 seconds long are the sweet spot for this testing methodology.
Conclusion: Demos of short and long nature are unsuitable for this tetsing methodology. Unless you're just looking for min/max/average/mean statistics and not graphs.
But demos of middle nature (40 seconds less or more) are perfect and could provide very good analysis for not only framerate but also driver problems as driver problems will cause framerate to jump around erratically. As BFG10K said, recent ATi drivers had problems with erratic framerate because of texture upload anamolies.
Testing methodology:Everything besides ROMP was run with fraps running in the background dumping data to frapssec.txt while running the UT demo bench. After some work in excel and realizing which break was which demo, I was able to create spreadsheets.
Edit 2:Allright. I'd like to present an example of a useful metric of minimum framerate.
I'd like to present this metric as a possible average of the lowest 10% (framerate wise) of the sequence.
Thus, you'd take the lowest 1/10th of the total (or the 10% margin) and average it. There you go. A minimum framerate average. While this is still an average, yes, it shows the worst case situation in a given benchmark. Just the slowest frame [to draw] itself might be deceptive, so you must have some sort of an average.
This would be a pretty useful metric, given it's much more important to know how a card will perform in the most hectic of situations, than to know the average of the whole situation. Honestly, if you can push that number over 50 in the most intense situations, you'll be allright in the rest of the game. Because honestly I believe the most intense situations in benchmarks are far worse than what you would encounter in any form of real gameplay.
I'd also like to reccomend methods outlined in My thread.
Time demos in conjunction with fraps wouold provite very good measurements, provided that some degree of acuracy in starting and stopping of the benchmark could be maintained. The data could also be extrapolated into graphs (Right now FRAPS has 1 second one measurement resolution textfile dumps that could be used for making graphs.) and stuff, while it might be a bit more work, I'd say that using fraps you could get acurate average, min, max, and graph results for the upcomming GeforceFX versus R300/R350 articles.
Everyone, voice your opinion here. Maybe i'll E-mail anand to keep an eye on this thread, and maybe we'll get some results.
Edit:Anand has been alerted of this thread since it's creation. Your votes most likely *will* count.
Edit:Stuff below
RAW framerate dump
Detailed and cleaned framerate dump with markers
What everyone has been waiting for. GRAPHS!
Now, to give you a quick analaysis of the data within
System specs:
Williamette 1.7
Baracuda 4 40GB
Alot of windows open!! (6) so the data isn't meant to be absolutley acurate
256MB of RAM
Radeon8500LE 64MB
Graphs are at the right of the excel document.
Time is in seconds. Measurements of average framerate are taken every second.
Romp was just that, a quick deathmatch in UT2003. It's about 10 minutes long. The low resolution of fraps (1 second) hurts the graph by making it look chaotic. Long demos are unacceptable for this testing methodology.
Flyby's:They look very rough, as they're only about 10 seconds, and thus, 10 samples long and alot of the graph is interpolation if you'll notice. With low resolution, extremley short demos
Botmatches:These are much better looking, and show the rather chaotic nature of bot matches. Framerate jumps wildly. It seems that demos 30 or 40 seconds long are the sweet spot for this testing methodology.
Conclusion: Demos of short and long nature are unsuitable for this tetsing methodology. Unless you're just looking for min/max/average/mean statistics and not graphs.
But demos of middle nature (40 seconds less or more) are perfect and could provide very good analysis for not only framerate but also driver problems as driver problems will cause framerate to jump around erratically. As BFG10K said, recent ATi drivers had problems with erratic framerate because of texture upload anamolies.
Testing methodology:Everything besides ROMP was run with fraps running in the background dumping data to frapssec.txt while running the UT demo bench. After some work in excel and realizing which break was which demo, I was able to create spreadsheets.
Edit 2:Allright. I'd like to present an example of a useful metric of minimum framerate.
I'd like to present this metric as a possible average of the lowest 10% (framerate wise) of the sequence.
Thus, you'd take the lowest 1/10th of the total (or the 10% margin) and average it. There you go. A minimum framerate average. While this is still an average, yes, it shows the worst case situation in a given benchmark. Just the slowest frame [to draw] itself might be deceptive, so you must have some sort of an average.
This would be a pretty useful metric, given it's much more important to know how a card will perform in the most hectic of situations, than to know the average of the whole situation. Honestly, if you can push that number over 50 in the most intense situations, you'll be allright in the rest of the game. Because honestly I believe the most intense situations in benchmarks are far worse than what you would encounter in any form of real gameplay.
Facebook
Twitter