MultiGPU Update: Does 3-way Make Sense?

[taltamir]

on MultiGPU Update: Does 3-way Make Sense? on page 3:
http://www.anandtech.com/video/showdoc.aspx?i=3518&p=3

the following statement is made:

It's very difficult to really collect high quality quantitative data that shows microstutter, as the only way to really get a good idea of what's going on is to analyze raw frame data on a per frame basis (which you can't get with FRAPS)

This is incorrect, FRAPS can capture those, its in the settings, it gives exact MS "time" of each frame, and it gives them in order, in a csv file. You can create a function showing the difference between each pair, giving you the ms per frame, and you can analyze that data.

 

[alcoholbob]

Anandtech is so inconsistent, in a previous review they had minimum framerates. Now in this review they talk about microstutter, which they didn't sample at all, but don't even make a token attempt to sample minimum framerates either.

What he theck.

 

[exar333]

Good point, they should really track min, average, and max framerates. Different people care for different things, and it also allows you to compare one review to another and make a better decison on what suits you best.

 

[chizow]

Ya not sure what he meant there, I haven't read through the whole thing yet, but you should post it in the comments. Derek is pretty good about reading and replying to those comments.

I'm not sure if he was taking a jab at FRAPs itself though as I'm pretty sure I've seen him question the validity/accuracy of it. Meaning he didn't trust the accuracy of the frame time dumps from FRAPs, rather than it was incapable of generating frame time dumps.

Also Astralite, the reviews with minimums are done by other reviewers, I know Gary Key puts them in his mobo reviews now for sure. I'm sure Derek knows people wants to see them but he resists for some reason. I think part of it is due to inconsistency when it comes to getting a minimum number. Many sites use median low but it can be a pain to parse all that data to get medians, especially when you're talking something like 10-15 results per game with multiple runs per set-up.

 

[kevindorsey]

I'll have to read up on this myself

 

[apoppin]

Originally posted by: chizow
Ya not sure what he meant there, I haven't read through the whole thing yet, but you should post it in the comments. Derek is pretty good about reading and replying to those comments.

I'm not sure if he was taking a jab at FRAPs itself though as I'm pretty sure I've seen him question the validity/accuracy of it. Meaning he didn't trust the accuracy of the frame time dumps from FRAPs, rather than it was incapable of generating frame time dumps.

Also Astralite, the reviews with minimums are done by other reviewers, I know Gary Key puts them in his mobo reviews now for sure. I'm sure Derek knows people wants to see them but he resists for some reason. I think part of it is due to inconsistency when it comes to getting a minimum number. Many sites use median low but it can be a pain to parse all that data to get medians, especially when you're talking something like 10-15 results per game with multiple runs per set-up.

it can take forever to get all your ducks in a row; a set of 15 games can take a week to run and chart . . . especially with Vista

now i just set FarCry2 to run 50 runs; and walk by often
.. and the averages are so damn close now it completely evens out

but i have no more issues with minimums than i do with averages or maximums .. so i have no idea why he does not use minimums .. they make the MOST difference to me

i believe FRAPS is accurate overall .. but then you do see inconsistencies; we have to use the tools that we have. i wish they would rework and update FRAPS; it looks like another programmer could come in and do a better job perhaps; but you never see it

he is actually right about the first part of it:

It's very difficult to really collect high quality quantitative data that shows microstutter

it is hard to show it .. i am waiting a bit myself before i tackle it; and then i want to have SLi also - to make it meaningful

 

[alcoholbob]

Not including minimums is for < 2004 reviews.

If you don't give minimums you don't give viewers even a general idea of what is going on. If you are running a timedemo, there's nothing inconsistent between minimum and average. Both will be within a small window of error but should be very close. If you are talking about a game run, there's nothing consistent about average framerates either. There's actually more or less no argument you can give to NOT include minimums.

Now the time argument makes no sense. If anandtech doesn't do it, then some other review site will. That gives other sites more traffic and more revenue.

 

[nitromullet]

Originally posted by: apoppin
he is actually right about the first part of it:

It's very difficult to really collect high quality quantitative data that shows microstutter

it is hard to show it .. i am waiting a bit myself before i tackle it; and then i want to have SLi also - to make it meaningful

Good point. Ideally, you want to compare NVIDIA and ATI with both single and multi-gpu setups.

 

[taltamir]

this is not meant to be a bashing of anadtech or demands that they start doing something. So please don't make this the issue.

The only point I wanted to make is that the review claims that fraps is incapable of giving a per frame ms to render dump, where it clearly CAN. And also that microstutter cannot be empirically tested for, when it clearly can (using those frame dumps..)

 

[chizow]

Originally posted by: apoppin
but i have no more issues with minimums than i do with averages or maximums .. so i have no idea why he does not use minimums .. they make the MOST difference to me

Well it certainly depends, personally I find FRAPs tends to lurch or drop FPS momentarily when you start or stop logging FPS, so if you're marking minimums, you have to decide whether to keep those lows or manually filter them. Or you can just take the median or whatever sampling method you think is fair. Either way, those are all significantly more involved than just taking the average (which will also have those minimums) or maximum.

How that translates into actualy gamplay...well if you never log frames and the frame rates never drop that low otherwise, then manually filtering those drops out of both the minimums and average would be more accurate, but would also be exponentially more time consuming as well.

i believe FRAPS is accurate overall .. but then you do see inconsistencies; we have to use the tools that we have. i wish they would rework and update FRAPS; it looks like another programmer could come in and do a better job perhaps; but you never see it

Ya its great for what it costs, but it would be amazing if they were able to do some of the things seen in the FC2 or even Crysis benchmark tools.

it is hard to show it .. i am waiting a bit myself before i tackle it; and then i want to have SLi also - to make it meaningful

Again, not sure what you mean there, its easy enough to show microstutter provided you trust the FRAPs dumps. If you mean its hard to demonstrate it in terms of actual visual difference and gameplay then ya I'd agree (although some of the youtube videos do a decent job).

 

[chizow]

Originally posted by: taltamir
this is not meant to be a bashing of anadtech or demands that they start doing something. So please don't make this the issue.

The only point I wanted to make is that the review claims that fraps is incapable of giving a per frame ms to render dump, where it clearly CAN. And also that microstutter cannot be empirically tested for, when it clearly can (using those frame dumps..)

I don't think any of the comments were meant to bash AT. You have a legitimate question, if you want a straight answer, the best way to get it would be to post it in the article comments, as Derek is really good at reading and responding to them. That's the benefit of feedback. No one else here knows what Derek meant when he wrote that, which is why we're all left speculating. I'd be personally interested in finding out his response as well, so if you do get an answer please update here, thanks.

 

[apoppin]

Originally posted by: chizow

Originally posted by: apoppin
but i have no more issues with minimums than i do with averages or maximums .. so i have no idea why he does not use minimums .. they make the MOST difference to me

Well it certainly depends, personally I find FRAPs tends to lurch or drop FPS momentarily when you start or stop logging FPS, so if you're marking minimums, you have to decide whether to keep those lows or manually filter them. Or you can just take the median or whatever sampling method you think is fair. Either way, those are all significantly more involved than just taking the average (which will also have those minimums) or maximum.

How that translates into actualy gamplay...well if you never log frames and the frame rates never drop that low otherwise, then manually filtering those drops out of both the minimums and average would be more accurate, but would also be exponentially more time consuming as well.

i believe FRAPS is accurate overall .. but then you do see inconsistencies; we have to use the tools that we have. i wish they would rework and update FRAPS; it looks like another programmer could come in and do a better job perhaps; but you never see it

Ya its great for what it costs, but it would be amazing if they were able to do some of the things seen in the FC2 or even Crysis benchmark tools.

it is hard to show it .. i am waiting a bit myself before i tackle it; and then i want to have SLi also - to make it meaningful

Again, not sure what you mean there, its easy enough to show microstutter provided you trust the FRAPs dumps. If you mean its hard to demonstrate it in terms of actual visual difference and gameplay then ya I'd agree (although some of the youtube videos do a decent job).

that is what i mean .. you have FRAPS working fine with little glitches you have to filter; or else you have to repeat them over and over again to see the "pattern"

that IS again what i mean .. a programmer who brings us the detail of the Clear Sky or FC2 benchmarks should be able to design a better tool for most games

and again ... i can SEE microstutter and relate it easily to the FRAPS dumps

it is hard to show it [consistently/easily] to others. Even at home where i can point it out; some people just don't seem to even see it

.. just as some "audiophiles" that like big inaccurate speakers with "bass" that is really doubled distortion at 100hz; they lack the "golden ears" but they sure have an opinion
:roll:

 

[alcoholbob]

Bass hump at 100hz range is typical of headphones and "East Coast Sound." For headphones its understandable...you can't recreate real bass so you just create the illusion of it. The latter...no comment.

 

[apoppin]

Originally posted by: Astrallite
Bass hump at 100hz range is typical of headphones and "East Coast Sound." For headphones its understandable...you can't recreate real bass so you just create the illusion of it. The latter...no comment.

come on .. i was exaggerating to make a point

Years ago i worked in high end audio. i used to waste time trying to educate my customers .. some of whom are really deaf and cannot even discern between Cerwin Vega and Magnaplanar :Q

or "house brand" and Advent

[whoa listen to that bass ]

"oh, yeah .. Mudd sounds good .. let's wrap it up"
:laugh:

well, i did personally have stacked Dahlquest DQ10s, 1000w a channel GAS, Grado Signature, Thorens and Mark Levinson modified tube preamp
- i could tell the difference and hated CDs on my system