• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Why benchmark?

R

RoboTECH

Platinum Member
Why do people benchmark? It seems that reviews these days are almost entirely benchmark-centric. Very little thought goes into the reviews. Just slap the card in and run a few benchmarks. Much easier to "review" a card that way, rather than actually use it for gaming.

So why bother benchmarking?

<devil's advocate>

I'd say benchmarking is a tool used by someone who doesn't have much of their own tool. <g>

In other words, it's just bragging. Peeps that are always trumpteting this benchmark and that without much thought behind it aren't interested in buying computer hardware to enhance their computers, they're interested in enhancing their egos, so they can brag about how many 3dmark's they get.

</devil's advocate>

so if you're comfortable with your penis size, why such an emphasis on benchmarking?

<eagerly awaiting responses>
 
The only thing I ever benchmark is in the actual game. And the only reason I do it is to see if what I did improved performance. Like is it faster in 98 or win2000? Is the AThlon 600 THAT better than a k6-2 450?

If it looks good and plays better, I'm happy. I guess the penis size people also have to post in their sigs their system specs. It also makes you more manly the higher % you can overclock too 🙂

My favorite is the people who spout off what 3dmark, CPUMark or whatever BS benchmark scores they get. Wow your CPU is a 14528. Cool. Mine's only a 13924 and I feel so inadequate.
 
while benchmarking may be a 'bragging' tool, it is useful on quite a few levels. first off, it serves as a comparison point between two products. to me, it's the same as having a faster processor. and as we all know, 800mhz is better than 700mhz. a faster video card will result in more fps and therefore make the game more 'playable.' secondly, most of us are tweakers, and we enjoy getting that little extra from what we own. make slight alterations and you have a 'better' product. third, and MAYBE most importantly, it's competition. all of our lives we compete with one another. for love, money, food, sports, etc... the desire to be and have the best is what makes us 'better' and keeps us from being complacent.

at least that's the way i look at it.
 
LOL........ROTFLMAO

&quot;In other words, it's just bragging&quot;

Robo
P3 @ 966 1.65v retail HSF[b/]
MSI BX Master
64MB GTS
2x128 Mushkin Mosel CAS2 @ 138[b/b]
SB Live
SuperMicro 750a w/5000 case fans
2x30GB[/b} Quantum LM's

 
Why do people benchmark?

To find out how products compare to each other. Benchmarks are a standard and widely accepted way of gauging relative performance between products.

It seems that reviews these days are almost entirely benchmark-centric. Very little thought goes into the reviews. Just slap the card in and run a few benchmarks. Much easier to &quot;review&quot; a card that way, rather than actually use it for gaming.

Well it depends what reviews you read. If you read gaming sites' reviews all they care about how the game runs on certain hardware and possibly image quality and stability.

If you read reviews done by good PC hardware sites like Tom's Hardware, you will find a lot of other in depth information along with the benchmarks.

I'd say benchmarking is a tool used by someone who doesn't have much of their own tool.

Well if you want to play 3D games and you want to see how well a card performs, isn't benchmarking your &quot;tool&quot;?

All this talk of doing away with benchmarking is bordering on crazy talk. If you don't like the benchmarks skip over them. I and the millions of other people who want to see benchmarks will read them as normal. Problem solved.
 
I don't give a rat's @ss what the benchmark says. I don't play benchmarks, I play games.

I do care how the game looks and plays. Period. It's really quite simple.

Imho, basing choices soley on benchmarks is quite irrational, since nobody plays benchmarks. Well, I suppose if you'd rather watch 3dMark for an hour, instead of playing your favorite game... more power to ya.

BFG10K, until you have experience with a particular piece of hardware, you are not qualified to make such definite statements on the quality of it.
 
can u see any different if u dont benchmark your system? for example if u have p3-800 and p3-1000 can u see any different if u just play the os application?? that is for processor!

if i am not like to benchmark my system, may be i am not interesting in overlock my system 🙂




 
I could write a book on this subject, there are hundreds if not thousands of reasons to benchmark, particular performance in every given application for example😉 Broader reasons would be-

Countering PR- If we eliminated benchmarks from reviews what would we have? Just eyeballing the specs and games reviewers could well state that the V5 5500 for instance is about equal to a GF DDR and not comparable to the Radeon or GF2. Even with running benches this still seems to be the overall performance consensus, but the problem is that there are not enough benches run, perhaps some flight sims should be thrown in(Falcon4, IIRC, has a benchmarking tool).

New Features- This one is several issues, how well does something work within a given implementation and if the feature works at all when under extrodinary circumstances(for instance none of the current boards can handle 4x FSAA@1600x1200). How well a new feature works is extremely important, will this board allow this feature to be practical when games/applications start using it(though that is more of a future proof issue)? What kind of performance hit will I be taking if I enable feature X(FSAA, EMBM, Dot3 etc)?

Performance- Performance matters, a great deal to the majority of people, be it for whatever reason it may. Games simply look better at 60FPS then they do at 10FPS, don't think to many people will argue with that, and what could be argued to be a &quot;trivial&quot; boost of 160FPS over 80FPS now, could be the gap between 25FPS and 50FPS in future games. This also plays into bang for the buck. If you could chose either a GF2U or a G400 both for $150, which would you go for? Why? People do care about performance, without benchmarks we have to trust someone we don't know has excellent and very sensitive vision to tell us which card performs better.

Tweaking- I like to get my money's worth out of my hardware, tweaking things to achieve the best possible performance out of all of my hardware is something I do to ensure that I'm getting everything I paid for. Can I honestly tell the difference between 3%-4% here and there? No, but add them all up and it results in a much smoother system. By utilizing benchmarks I can tweak each setting one at a time until I find the optimal ones. This includes OCing, drivers, Window's settings etc., if they are all set to optimal, you get a much smoother and more stable(knowing when to back that CPU/vid card down) machine.

Future Proof?- I don't care what anyone says, would you buy a board that you knew would not play any game that was upcoming? No, you would be a fool to, how futureproof a video card is is important to a great deal of people. Take hardware T&amp;L, we know how it works, how it performs in games, and it is becoming increasingly important. Without benchmarks, we wouldn't know that this would have worked at all, but it does quite well. How useful it is for the average user is another matter entirely, the matter of how well the hardware works is what is the most important thing. EMBM, Dot3 are other examples, how well does hardware handle these features which are becoming increasingly popular(judging by previews anyway)? FSAA is one but the other way around, how much longer will the current boards be able to push new games with this feature on? Perhaps another six months, maybe less(with the exception of the V5 6K and the GF2U). That is based on benchmark performance in current games and average fillrate requirement increases.

The list goes on and on, not benchmarking is the foolish way to go IMHO. You want to trust every single thing hardware companies, or reviewers we don't know, tell us? That would lead to some truly nasty behind the scene going ons🙂
 
I ran my first benchmark last week had a PC for over 6 years anyway the reason why I wanted to do it is to see if it was stable &amp; get a rough benchmark of performance.The results were good &amp; it was rock stable during testing on my Via board so am happy with the results ,I would agree some people bragg too much, but it can give you a idea how your 3D card &amp; PC is doing ,most people like to tweak there PC(like I do which in itself is fun) &amp; benchmarks can help you with tweaking.

🙂
 
dammit Ben, you stole my thunder.

blah.

for some of you, you apparently missed my &quot;<devil's advocate> </devils' advocate>&quot; re: the &quot;tool&quot; comment. oy vey...and I thought I spelled it out.

BFG, i'm not saying to stop benchmarking, that would be silly. I'm saying to use it wisely (as ben mentioned)

Another point is that it seems that people are nitpicking in benchmarks. &quot;Card 1 does 120 fps in Q3, Card 2 does 105 in Q3, Card 1 is better&quot;. While that may be the case, that's not *always* the case.

Ben, you hit upon a HUGE point that I was trying to make: not all games have benchmarks. BFG likes to say &quot;Unreal is a horrible benchmark&quot;. Well, yes it is, if you don't play UT. If you do, then you want to know how well your system will run with UT. It may only extend to Unreal-engine based games, but if you play Unreal engine games (3 very popular ones with a 4th coming out &quot;soon&quot😉, then this is an important measure.

Another example are games like NASCAR Heat (which has a benchmark), NFS-PU and various flight sims. These are also games which are definitely &quot;performance-oriented&quot; (i.e. you don't want stutters). How often (aside from www.simhq.com) do you see ANY of these types of games benchmarked?

I have a beef with internet reviews these days. They are too benchmark-oriented, IMHO. The GTS is a few fps faster than a Radeon in 32-bit color. Wonderful. How is image quality with the same settings? How playable is the Radeon? How many fps do you need to play Quake3 without stutters? What about the 5500? Can you play Q3 on a 5500 with great graphics without sacrificing smooth gameplay?

3dMark2000 is such an OUSTANDING benchmark, but it is totally misused. Very rarely do you see anyone actually break the benchmark down. It has about 20 different &quot;things&quot; that get checked, but 99.9% of all reviewers just state the final score. Lame. There is much to be determined by that benchmark, not the least of which is how much T&amp;L will matter in the future.

Okay. So we know &quot;100 is better than 85&quot;, but how much of a difference does it make in the end?

Perhaps the title of the thread should be &quot;Why benchmark mindlessly&quot;. 🙂



 
I agree that many reviews skip over too much important info. 2d quality on Geforce2 cards is an example where people were dumbfounded by after getting them. Then there were all the people who based their decision on Q3 benches, but they only played UT(that problem seems to be fixed for Geforces now). Then of course we have the reviews where weighted benches are used, such as what happens often with cpu reviews where SSE enhanced software is used when non-enhanced applications could be used.

Q3 and 3dmark are used too much as decisive benches, IMO, but they should still be used, just not exclusively.
 
I would NEVEr listen to any site that only used one benchmark.

That'd be like a site running ONLY Deus Ex showing a Voodoo 3 kicking the crap out of a Geforce 2.

Or a K6-2 competing hand in hand with a PII in CPU mark, etc. You need the whole picture if you don't want to get burned.
 
BFG10K, until you have experience with a particular piece of hardware, you are not qualified to make such definite statements on the quality of it.

I am not qualified. That is why I read the reviews: to make myself qualified. I don't have time or money to review each board or compare it to every other board. That is what the review sites are for.

BFG likes to say &quot;Unreal is a horrible benchmark&quot;.

Well it you are playing Unreal/UT of course you can use it as a benchmark. There is nothing wrong with that. My problem is when some zombie says &quot;look nVidia sucks because the Voodoos beat them in Unreal/UT&quot;. Nope, not even close.

For a general benchmark of how well your card does overall, Unreal/UT is not suitable for this.

I mean if you were some 3D artist and all you ever ran was some 3D rendering program which just happend to work best on say, a Voodoo 1 or something, by all means get the Voodoo 1. Just don't say the Voodoo 1 kills everything else based on this benchmark, because it isn't a general benchmark.
 


<< BFG10K, until you have experience with a particular piece of hardware, you are not qualified to make such definite statements on the quality of it.
I am not qualified. That is why I read the reviews: to make myself qualified. I don't have time or money to review each board or compare it to every other board. That is what the review sites are for. >>

understood. so what happens when you get websites with &quot;an agenda&quot;? What if the peeps at the website are just plain uninformed, or lacking in knowledge? What if they get a free review card from a company, so long as it gets a good review? Don't be so blind as to think this doesn't happen. I can understand you wanting to trust a review site, and that's fine, but don't toss me lines from this website or that website when I know what I've seen with my own eyes. To borrow a saying 'you know what the media wants you to know&quot;. Comprende?



<< For a general benchmark of how well your card does overall, Unreal/UT is not suitable for this. >>



Unreal/UT is not a very good &quot;general&quot; benchmark, but in it's present incarnation, neither is 3dMark2000. 3dMark2k certainly has *potential*, were a website to actually view it in depth and not just parrot it's &quot;top scores&quot; with the default settings. Riva3d did a very good job of this in their 5500 review. Kudos to the fella.

Shudder:

<< You need the whole picture if you don't want to get burned. >>

agreed. Unfortunately, it seems that so many review sites are moving away from &quot;the whole picture&quot;, and are moving to a benchmark-centric method of review. Benchmarks should be a *part* of a review, they should NOT make up the *entire* review.

How many websites are going to show us Quake3 scores on the GTS? I mean, really. We all know approximately how fast a GTS is @ 1024x768x32 in Quake3 with the Detonator 2's and Detonator 3's. ALL GTS cards are going to be like this. How many times do we have to see it?

www.tech-report.com just did a decent write-up of the 5500. Unfortunately, they fell into the trap of 3dmark2000/Quake3/MDK2 just like several other sites have done. It's easily understandable. They are *easy* to run.

They started to do quite a bit of analysis, which is certainly a welcome change, but then stopped short of what could've been done. They overclocked the 5500, then showed one single benchmark (By itself). Ugh....so close...

Anyway....my main point is that benchmarks are a means to an end, not the end itself.


 
I think I see your main issue Robo, you want a 30-50 page review for each video card, so do I🙂

The problem is that sites get lots of complaints about overly lengthy reviews. JohnQTechie wants it quick and to the point. I'm with you, I want the overly long every aspect drawn out ten pages of benchmarks of ten to twenty different tests with analysis on every feature, but that will only come from sites that are willing to give up hits to publish the most thorough review(if anyone wants to send me boards, I'll gladly do it🙂).

I don't know how long you have been around, but a while back Anand posted what I thought to be his best single review ever, the &quot;Desktop CPU Comparison&quot;. The review was a monster and left very little for speculation on pretty much every reasonably current CPU on the market. I loved it, most people complained endlessly about it being to long. Couple that with the enormous effort that goes into doing a review of that size and you can see why it is very simple, and in the interest of *most* of the readers to simplify and shorten the review.

Take the PIII vs Athlon comparisons. For the rendering operation I talked about in the other thread(8Kx8K 16x😉) a like clocked Athlon outperforms a PIII by roughly 30%, that is *major*, but how many people care? Photoshop performance is one that the PIII handily bests the Athlon on when comparing like clocked offerings, but how many people care? Too often I see people ask for advice and people say buy processor xx because they perform almost exactly the same. The limited scope of benches run gives that impression, though one can be significantly better for a particular task.

Same goes for video cards. CPU limited games show very little in the way of alterations in FPS, but that extra 5FPS could be the same as a difference in an extra 100MHZ for the CPU(and a decent amount of added cost) by using the lower overhead Glide API particularly in flight sims.
 
understood. so what happens when you get websites with &quot;an agenda&quot;? What if the peeps at the website are just plain uninformed, or lacking in knowledge? What if they get a free review card from a company, so long as it gets a good review? Don't be so blind as to think this doesn't happen.

That is certainly possible (in fact I saw one the other day which had clearly fudged the GF2 benchmarks), so that is why I visit official, large, well-known sites, and I visit many different ones to get a wide variety of results. I seriously doubt that 10 of the biggest PC hardware sites (Tom's, Firing Squad, Sharky, Anandtech, Extreme3D etc) would have the problems you mentioned, especially since all their reviews tend to have the same results as each others.

I can understand you wanting to trust a review site, and that's fine, but don't toss me lines from this website or that website when I know what I've seen with my own eyes. To borrow a saying 'you know what the media wants you to know&quot;. Comprende?

I take your point but think about this: what happens if all the problems you mentioned above are with you? You said I shouldn't trust a website so why should I trust you?

I have no idea who you are or what you do. I only know you as an online personality. Those other sites have proper contact details and are official. Why should I trust you any more than I trust them?

I'm not saying you are a liar and that I don't trust you or anything, I am just making a point.
 


<< I take your point but think about this: what happens if all the problems you mentioned above are with you? You said I shouldn't trust a website so why should I trust you?

I have no idea who you are or what you do. I only know you as an online personality. Those other sites have proper contact details and are official. Why should I trust you any more than I trust them?

I'm not saying you are a liar and that I don't trust you or anything, I am just making a point. >>



very good points indeed. I do have an agenda. My agenda is stability and speed in the games I like, in that order. 🙂

and no one needs to trust me, no doubt. I guess I've just grown quite annoyed with so many of the website reviewage out there. Bah....thank God for Electronics Boutique and their return policy, or I NEVER would've even bothered *Trying* the 5500 at all, heh...
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top