New 3dmark03 patch - nVIDIA cheating ... again???

[Insomniak]

Originally posted by: Rogodin2
My philosophy is an aglomeration of many thinkers-even though you don't "give a rat's ass" about philosophy you are inundated with "life" and this demands a philosophy wether you like it or not.

If I may suggest a wonderful and easy to read philosopher I would suggest "Jaspers" or "Keirkregaard".

rogo

I'm sure it is - all philosopies (for lack of a better term) are such amalgams. Every individual is influenced by unique persons, thoughts, views, experiences, etc. during their run on this planet, and as such their philosophi (is that plural for philosophy?) are endlessly variable.

While I don't subscribe to any of the major philosophical schools of thought, I will readily agree with your point that everyone follows one philosophy or another. To be human is to do so...unless, I suppose, one is in a coma.

 

[Insomniak]

Originally posted by: spam
Hi Insomniak,

You do like to stir the pot don't you? If you want to discuss existentialist philosophy perhaps you could visit the Off Topic forum, that may be a better venue for this conversation. There must be some pragmatic approach to evaluating the value of a given video card. That is the purpose of this forum.

Well, existentialism is only part of what we're dicussing here, but aside from that....

Yes, I do love a good pot-stirring. Of course, the topic me and Rogo are discussing was in fact introduced by you...so let's not go condemning it now. That's just silly. Yes, the subject is off topic for this forum AND thread, but I'm too lazy to move it. Call someone else. I'm sure there are staff or some such which handle this kind of thing.

 

[Genx87]

The Beyond3d review was actually pretty good. They brought up some interesting points about rendering and in the end left it upto the individual users to decide if these minor rendering differences will make or break your buying decision.

I also enjoyed reading the responses from Nvidia and ATI. Nvidia has a point about chasing FM in terms of optimizations. This is what you can expect when you purchase an Nvidia card in real life games. I dont think this is all that bad provided the differences in the IQ is very small or non-existent which it appears is the case.

Overall I wouldnt use these minor rendering issues as a basis to not purchase an Nvidia card when I do my next upgrade. I will never notice the difference during gameplay.

 

[kylebisme]

but Genx87, you are overlooking the fact that nvidia often needs to make such "minor rendering differences" just to get acceptable performance from their hardware; so you can't rightly expect good framerate on anything but the most commonly benchmarked applications. i think that does deserve consideration when considering an upgrade.

 

[Genx87]

Often?

To be honest where else is this happening? And even it it was the rendering is so small a difference I will never notice. Right now there are not many released games I can think of where ATI holds a huge lead in. So it will come down to the two biggies on my next upgrade. Doom3 and HL2 provided they are both out.

 

[jiffylube1024]

Originally posted by: Genx87

Overall I wouldnt use these minor rendering issues as a basis to not purchase an Nvidia card when I do my next upgrade. I will never notice the difference during gameplay.

The problem is if you give Nvidia carte blanche to introduce "minor rendering issues" then what should stop ATI from doing the same. And, as pointed out before, if both companies are wasting their time optimizing for every game and trying to one-up eachother in the "slight image quality for more rendering speed" category then we get a universal dip in IQ for a minimum performance gain (sometimes it's more substantial).

Look at Nvidia's dropoff with this FM fix - it's in the ballpark of 15-20%. In most modern games, that's the difference between 100fps and 80 fps or even 150 fps vs 120fps. This is not needed to make games playable, and it's clearly just to make the Nvidia cards look favourable in benchmarks. With uneven testing methods no less.

Now some "optimizations" (like in the Mother Nature test) appear to have very little visual quality difference, which is good. However, for the ones that don't, Nvidia better strictly police what an optimization can and can't do, and if it does result in a dip in image quality then they better stick to the original rendering method. The whole point of 3dmark (according to Futuremark) is to benchmark video card without optimizations done for this benchmark. So, Nvidia focusing on this benchmark and optimizing for it is the reason for so much uproar. Both companies do transparent optimizations for games (ironcally, Quake > Quack by ATI was none other than an application detecting optimization which did not degrade image quality at all, and yet they were hounded for it to no end), but they should leave benchmarks as they are or at least if the developer of the benchmarks says "dont optimize for this benchmark because that's not what it's intended for" they should just leave it alone!

 

[kylebisme]

Originally posted by: Genx87
Often?

To be honest where else is this happening? And even it it was the rendering is so small a difference I will never notice. Right now there are not many released games I can think of where ATI holds a huge lead in. So it will come down to the two biggies on my next upgrade. Doom3 and HL2 provided they are both out.

tom's showed it happening in aquamark, and some rendering issues have been found in halo and the new tomb radar as well. it seems to be happening in games and benchmarks which make use of dx8.1 and dx9 shader tech, which of course is to be expected as the fx have been repeatedly shown to struggle in this area. however, that is beside the point as there really isn't much of an issue for the end user for the few games and benchmarks which nvidia does take the time to hand code workarounds and pack them in their drivers. the many games that are not so high profile and therefore not a priority for nvidia's "optimizations" are what should be cause for consern.

 

[vshah]

I really wish this thread would have stuck to a purely technical discussion instead of rambling all over and accusing everyone. I mean we're getting quotes from Mr. Einstein in a video card forum

It'd be nice to read this thread and be able to objectively answer these questions:

-What happens to Nvidia's performance in each game test?
-What happens to Ati's performance in each game test?
-What happens to Nvidia's IQ with/without AF/AA?
-What happens to Ati's IQ with/without AF/AA?

IMO thats all that should be under discussion here. All I care about in the end is IQ and FPS, either in a real world or synthetic benchmark. I'm sure everyone could do without the finger pointing and accusations.

Try and focus on the scores/changes themselves, not what these scores/changes imply about a company. Thats what I do and it makes my purchasing decisions much easier.
FYI i'm an "Nvidiot" running a 9700pro. As soon as i find an Nvidia card with consistenly higher performance i'm going to jump up and down on my 9700 :evil:

-Vivan

 

[Vonkhan]

Both D3 and HL2 are out, sorta ... if u know where to find 'em :beer:

me? naah ... I dont like alphas and chopped up hack jobs, but they did help me in making a choice while buying my XT

 

[Matthias99]

i'm pointing out that HE OWNS 2 ATI CARDS ON 2 OF HIS COMPUTERS, AND HE POSTS SOMETHING AGAINST NVIDIA, WHICH IS FANBOY-RELATED ACTIVITY. What's so hard to understand about that?? Can't get it throught (sic) your thick F%#&ING skull?

i'm not saying you don't have a right to complain, but if you don't even own the damn card, why complain.... and if you own fricken ATI cards (such as the dude that made this thread), it's obvious a stake out against nvidia, which you should go make your own forum to polute (sic) instead of something like anandtech.

So... clearly everyone that *doesn't* own an NVIDIA card should remain silent on this issue, and everyone that owns an ATI card is a fanboy? riiiiiight. Hey, I own a Ti4600 -- does that mean I can post one-sided remarks against NVIDIA and not be labeled an ATI fanboy? Oh wait -- I also have a GeForce2, so I must be an 'nVidiot' and ardently support everything they do.

Were only Enron stockholders and customers supposed to be upset when their financial dealings came to light? Not that anything going on here is anywhere near that scale, but it's the same sort of idea. Dishonesty hurts the whole industry, and if NVIDIA is trying to artificially inflate their benchmark scores (I don't know if they are -- there's not enough evidence yet), it affects *all* the other companies that make graphics cards and chipsets, as well as their customers.

I'd also just like to note that there's more philosophical BS and showboating in this thread than there was in the discussion sections of my first intro philosophy class in college. And that's saying a lot.

 

[chsh1ca]

Originally posted by: Matthias99
So... clearly everyone that *doesn't* own an NVIDIA card should remain silent on this issue, and everyone that owns an ATI card is a fanboy? riiiiiight. Hey, I own a Ti4600 -- does that mean I can post one-sided remarks against NVIDIA and not be labeled an ATI fanboy? Oh wait -- I also have a GeForce2, so I must be an 'nVidiot' and ardently support everything they do.

Damn, I own a GeForce 3, I guess I'll have to whip out the "ATI IS TEH SUCK" signs and start issuing pamphlets aobut how these 'optimizations' actually give all applications a performance boost.

Man, at least back in the 3dfx day people were happy that their cards would run things in 3d mode...

 

[NYHoustonman]

Nice to see all the college boys showing off what they can regurgitate from their classes. Very "Good Will Hunting" if ya axe me.

Quite so.

And, Rogodin, apparently your immense brain has no sense of sarcasm.

 

[spam]

Originally posted by: Insomniak

Yes, I do love a good pot-stirring. Of course, the topic me and Rogo are discussing was in fact introduced by you...so let's not go condemning it now. That's just silly. Yes, the subject is off topic for this forum AND thread, but I'm too lazy to move it. Call someone else. I'm sure there are staff or some such which handle this kind of thing.

Hi again Insomniak I set up a new thread on the Off Topic thread. Go have some fun. HERE:

 

[Matthias99]

Gah! Stupid posting errors. Hitting the "Reply" button at the bottom by mistake shouldn't post anything if the box is empty... :disgust:

Anyhow, trying to get back on topic...

For those of you who didn't notice this in the midst of the rambling philosophical discourse in the middle there, Beyond3D reviewed the drivers, the differences, and even spoke to NVIDIA and ATI:

Beyond3D Article

The most interesting part (to me) is at the end, when they're talking to NVIDIA:

At one [point] we asked [NVIDIA spokesman] Derek [Perez] how this sat with the optimisations guidelines that were given to press by NVIDIA, specifically the guideline that suggests "An optimization must accelerate more than just a benchmark" To which Derek's reply was "But 3DMark03 is only a benchmark" -- we suggested that this particular guideline should read "An optimization must accelerate more than just a benchmark unless the application is just a benchmark"!

If this is the stance they're taking at NVIDIA, then I don't really see what's changed. And I had given them a lot of credit for their new optimization guidelines... it'll be interesting to see if they backpedal frantically on this statement in the next few days.

 

[spam]

Matthias,

If you read FM's press release, you'll notice that it says only this benchmark is to be used for published results. Legal wise FM does not have to call Nv or ATI cheaters, instead I think it means that the IHV's have to submit to these guidelines or they will be legally liable.

 

[Matthias99]

I'm not quite sure what you thought I was trying to say, but I don't think I was trying to say what you thought I was.

FutureMark's patches and driver validation is their problem and their business. NVIDIA and FM disagreeing on how to approach optimization is one problem.

What's annoying the heck out of me is that after the *last* 3DMark03 debacle, NVIDIA published their *own* guidelines to optimization. Essentially, they are:

From the Beyond3D article (quoting from an NVIDIA press release, I assume):

An optimization must produce the correct image
An optimization must accelerate more than just a benchmark
An optimization must not contain pre-computed state

NVIDIA may have some wiggle room on 1 and 3 if they claim that they only have to be "very close" to or "no more than X% off" of the correct image, and if by "pre-computed state" they meant not using hard-coded precalculated values like clipping planes (but that replacement shaders are OK). However, at the end of the Beyond3D article it sounds like NVIDIA's spokesman is saying they think it's fine to optimize specifically for 3DMark03 because it's a benchmarking application (as opposed to, I guess, somehow optimizing *just* for the benchmarking part of a game but not the actual game itself?). That's not how I, or apparently Beyond3D, or any sane English-speaking person, interpreted rule #2 when NVIDIA published those guidelines. If that's their interpretation of their *own* rules (let alone Futuremark's), then those guidelines are absolutely meaningless.

 

[spam]

Derek went on to suggest that they may well end up chasing each patch release and re-optimising as Futuremark puts a patch that defeats previous detections!

At one we asked Derek how this sat with the optimisations guidelines that were given to press by NVIDIA, specifically the guideline that suggests "An optimization must accelerate more than just a benchmark" To which Derek's reply was "But 3DMark03 is only a benchmark" -- we suggested that this particular guideline should read "An optimization must accelerate more than just a benchmark unless the application is just a benchmark"!

This is from Dave Bauman at Beyond 3d talking to Nvidia's representative Derek Perez. Those are weasel words from Nvidia. I would be embarrasssed to have to defend them if I were Derek Perez. It implies that "we can inflate or massage any benchmark score as long as it is not a game!"

 

[Rogodin2]

"And, Rogodin, apparently your immense brain has no sense of sarcasm."

Sarcasm insn't an attribute of the brain-it's an attribute of "character".


However, I think the real issue here is not the 3DMark performance, but whether the optimisation guidelines they have reiterated to the press on two different occasions now are actually real or not. Upon first looks it would appear that the 52.16 violates all three of their guidelines in 3DMark. Are they serious about these guidelines or are they just paying lip service to appease the press?


rogo

 

[jasonja]

.

With the introduction of the GeForce FX - we built a sophisticated real-time compiler called the Unified Compiler technology. This compiler does real-time optimizations of code in applications to take full advantage of the GeForce FX architecture.

nVidia's reply about the their compiler is absolute BS. I proper realtime shader compiler/optimizer (read, like the one ATI and Matrox obviously have) detects the general cases of a shader instruction or group of instructions and can re-order the instructions or replace the instructions with faster (yet correct) instructions. This methodology allows for optimizations to occurr on shaders the moment they are introduced (such while a deveoper is creating a shader). This is how MS's compilers and optimizers work, right?

It's quite obvious that nVidia's ULTRA sophisticated optimizer is looking for 3dmark's specific known shaders and replacing them with hand tuned versions, because minor changes to the 3DMark shaders resulted in little to no optimizations on nVidia, but had no performance effect on ATI's driver. Replacing shaders is in no way realistic of how a card will run in REAL games and I'm sure developers don't "LOVE" this (as Derek likes to claim). With new games coming out everyweek, how does replacing known shaders become a viable option? Unless nVidia plans to drop a weekly driver to cover all new shaders then this is a crap approach at optimization. No, isntead they are targeting the key benchmarks that websites run, this is the reason why you see websites reporting that ATI cleans up in a large percentage of the off the shelfs apps they run.

Get a new job Derek, you're starting to sound like the Iraqi Info minister

 

[spam]

Careful! Josonja!!!

Get a new job Derek, you're starting to sound like the Iraqi Info minister

I posted a thread, "Nvidia PR department hiring Iraqi Information Minister?" and it got locked by the moderator and he called me Spam!!!

Although 3dMark is an artificial benchmark it still offers value if:
1. It reflects real world gaming experience.
2. It stands up to bullies.
3. It is not afraid to lose money or support from IHV's that are bullying it.

It is presently doing this.

If you follow the money in this case- it is Nvidia that has greatest vested interest in inflating benchmarks. Derek, said in his P.R. release that not only had they done it but they would continue to do so! 3DMark has gained a great deal of credibility in my eyes because they did find a way to police the use of their product without ending up on the wrong side of a litigation case.

 

[Rilescat]

Originally posted by: Jeff7181
What I think is funny is that people think any of this is "cheating."

Exactly..Jeff7181 is on the right subject. This is a synthetic benchmark anyway. If you want real performance markers, the tests should be run in game. If a game developer has a "cheat" that increases performance without seriously degrading image quality, do you REALLY think they won't include it? How is optimization exactly cheating? I can see there being controversy around a synthetic benchmark, but did you pay good $$$ for your video card to run benchmarks all day?

 

[chsh1ca]

Originally posted by: Jeff7181
Ok, go run 3DMark twice in a row, report your results to us.

Well, here it is, and your statement is a bit off based on the results. I decided to run 3 immediately consecutive runs in order to see if there was much of a difference produced.
Test Run1 | Run2 | Run3

Overall: 1013 | 1013 | 1015

GT1: 58.0 | 58.2 | 58.1
GT2: 8.0 | 7.9 | 7.9
GT3: 6.3 | 6.3 | 6.3
GT4: N/A | N/A | N/A

CPU Score: 378 | 376 | 377
CT1: 37.3 | 37.2 | 37.2
CT2: 7.6 | 7.5 | 7.5

FR (ST): 464.3MTexels/s (all three runs produced an identical number)
FT (MT): 1022.0MTexels/s (run 1 produced 1022.1MTexels/s)

VS: 2.2 | 2.2 | 2.2
PS2.0: N/A | N/A | N/A
RagTroll: 2.6 | 2.6 | 2.6

SND(0): 24.0 | 23.9 | 24.0
SND(24): 20.5 | 20.6 | 20.5
SND(60): N/A | N/A | N/A

System is:
Athlon XP 2000+
Asus A7V8X-X (onboard sound disabled, onboard BCM-4401 NIC enabled)
MSI GeForce 3 Ti200 128MB
Creative SoundBlaster Live!
3Com 3c905B-TX 10/100Mb NIC
Maxtor 7200RPM/ATA-100/20.4GB IDE HDD
Fujitsu 7200RPM/ATA-100/40GB IDE HDD
Pioneer 16x IDE DVD-ROM
Samsung 8x4x32 IDE CD-RW

Note: Windows only actually uses a 10GB partition on the Maxtor, so the Fujitsu doesn't enter the equation.

 

[Matthias99]

Originally posted by: Rilescat

Originally posted by: Jeff7181
What I think is funny is that people think any of this is "cheating."

Exactly..Jeff7181 is on the right subject. This is a synthetic benchmark anyway. If you want real performance markers, the tests should be run in game. If a game developer has a "cheat" that increases performance without seriously degrading image quality, do you REALLY think they won't include it? How is optimization exactly cheating? I can see there being controversy around a synthetic benchmark, but did you pay good $$$ for your video card to run benchmarks all day?

Optimization is not "cheating" (which is a loaded term) if it doesn't degrade image quality. Nobody is asking that the cards be forced to do something in an inefficent way if there's a better way to do it (such as by reordering rendering operations, like NVIDIA's newer drivers are supposedly doing). If it does degrade image quality, then they should be giving the user a choice to have either higher speed or higher quality -- let the customer decide instead of forcing it on them.

Synthetic benchmarks may be irrelevant to some people, but they are an important factor in many purchasing decisions. Performance in today's games doesn't necessarily say how a card will perform in tomorrow's games (or even other current games -- for instance, ATI cards are better at UT2K3 but worse at games based on Quake3 than NVIDIA). It's easier to develop a benchmark than an entire game, and so they're likely to use more cutting-edge technology (there are only a handful of DX9 games on the market, while 3DMark03 has been out for almost a year). Synthetic utilities are the *only* way to test most PS/VS2.0 functionality right now. While the best testing involves running multiple real games, not all games have reliable and easy-to-use benchmarking utilities built in, and most people would like to see just a few numbers that sum up a card's performance rather than 10 pages of framerate comparisons. And yes, some people *do* buy cards at least partly so that they can have the fastest thing out there, and they like to compare numbers.

For an analogous situation, how would you feel if you bought a car based on the manufacturer's performance numbers, and then its real-world performance didn't match up? Maybe they claimed a 0-60 time of 3.5 seconds -- but neglected to mention that they obtained that time on a downhill test track and with the seats taken out. You might be a little upset if you got it out on the road and found it was more like 8 seconds in most real driving situations. NVIDIA's going to have a lot of upset customers if they claim based on inflated benchmark results that their cards are as good or better at DX9 and next year's games prove otherwise.

It's Futuremark's prerogative to make sure their benchmark provides accurate, reliable numbers that correlate to the real world performance of the card, and they have every right to say NVIDIA's numbers are invalid if they aren't running the tests the way they intended them to be run.

 

[spam]

Hi Rilescat,

Originally posted by: Jeff7181
What I think is funny is that people think any of this is "cheating."

Exactly..Jeff7181 is on the right subject. This is a synthetic benchmark anyway. If you want real performance markers, the tests should be run in game. If a game developer has a "cheat" that increases performance without seriously degrading image quality, do you REALLY think they won't include it? How is optimization exactly cheating? I can see there being controversy around a synthetic benchmark, but did you pay good $$$ for your video card to run benchmarks all day?

-If it is not important or does not matter then why does NVidia bother to inflate the benchmark numbers? - Nvidia thinks its important-why? Because consumers think it is important and make their buying decisions based upon this benchmark result. Nvidia wants to make money. That is why it is important.
:sun:

 

[Genx87]

nVidia's reply about the their compiler is absolute BS. I proper realtime shader compiler/optimizer (read, like the one ATI and Matrox obviously have) detects the general cases of a shader instruction or group of instructions and can re-order the instructions or replace the instructions with faster (yet correct) instructions. This methodology allows for optimizations to occurr on shaders the moment they are introduced (such while a deveoper is creating a shader). This is how MS's compilers and optimizers work, right?


This is what the unified compiler does...............see Anands article on the 5700 Ultra. It will reorder the instructions to better fit Nvidia's arch.