Futuremark Responds to Nvidia 3DMark03 Criticism

[Spicedaddy]

Originally posted by: Adul
http://www.hardocp.com/article.html?art=NDMx

ATI's take on the whole orderdea,

From ATI:

ATI fully supports this new graphics benchmark, 3Dmark03.


Kyle:

While they do not come out and exactly say it, I am going to guess they find some value in the overall scoring methodology...


huh, Uncle Leo!?! HELLLLOOOOO!!

 

[NFS4]

Originally posted by: Spicedaddy

Originally posted by: Adul
http://www.hardocp.com/article.html?art=NDMx

ATI's take on the whole orderdea,

From ATI:

ATI fully supports this new graphics benchmark, 3Dmark03.


Kyle:

While they do not come out and exactly say it, I am going to guess they find some value in the overall scoring methodology...


huh, Uncle Leo!?! HELLLLOOOOO!!

LMFAO!! I was thinking the same thing

 

[Pariah]

But I disagree about the CPU comment. If you want to see how your CPU performs, look at the CPU Marks...

Take a look at this chart at Rage3D:

Rage3D

Notice something odd about the CPU marks for these identically configured systems except the video card? Still think I should look at CPU Marks to see how my CPU is performing? This whole test suite is bogus.

 

[magomago]

I don't put any emphasis on these...when I first built my PC I downloaded 3dmark 2000 instead of 2001 because it was smaller and I ended up scoring a 6000 on the freaking test.

Naturally it basically said my machine sucks.


But I just bought Starwars Jedi Knight 2 and can play it at 1024*768 with max details (save for pixel shaders b/c my videocard won't do tem)


Once my games start skipping and I have to turn down to 640*480 with NO detail (ie: 640*480 or 800*600 with max detail is good for me) THEN i'll upgrade my videocard.

But I find it funny how people love to put a huge emphasis on these things.

 

[human2k]

Originally posted by: SeekingTao

Originally posted by: Trader05
Yes, this bench is just a way of saying your stuff is outdated and go buy new stuff...if it works to your liking, no need to upgrade in my own opinion... i'm not going to upgrade my gf4 4200 cause it sucks in a benchmark... whatever happened to benching your system of video games, like quake 3/unreal 2

Agreed. My GF4 Ti 4200 128MB card has lots of life left in it. Especially since it now thinks it's a Ti 4400.

Double agreed. My Sapphire 9500NP that I got for $160 still has tons of life in it even tho 3dmark03 says its slow (4584). It was hard for me to convice myself to spend $160 in the first place. O well, im a happy camper.

 

[Pocatello]

I like 3dmark for what it is, a pretty little demo, no so little anymore, I guess.

 

[NOX]

Why even care about DX9 at the present time? If no titles are to take advantage of it till atleast 6 months from now I don't see what the big deal is with running DX9 benchmarks. 3dMark is used for marketing, plain and simple. I hope sites like Anandtech, Hothardware etc., never use it again to benchmark cards, because it means nothing in terms of true 3d performance.

 

[Codename49]

Originally posted by: Pariah

Can you name one announced DirectX 9 game?

Impossible Creatures. And it's not only announced, it's already availible.

 

[Leon]

The fact that NVIDIA released new drivers specifically to increase the score in 3DMark03 (and not in any other program) gives me the impression that they are full oh crap...or at least hypocritical.

Completely opposite. Nvidia released these drivers to show that they can double their scores in 3DMark, but the game performance will remain the same. Which proves their point (use the real games) precisely.

 

[THUGSROOK]

i remember a few years back when 3dm2k1 was released ~
at 1st noone cared cause it was too difficult to run and wasnt representative to current games ~
it took about 6 months for ppl to catchup and find it usefull.

with that being said...
ive already uninstalled 3dm03 ~ its useless at the moment but im sure it wont be forever.

 

[NFS4]

Originally posted by: Leon

The fact that NVIDIA released new drivers specifically to increase the score in 3DMark03 (and not in any other program) gives me the impression that they are full oh crap...or at least hypocritical.

Completely opposite. Nvidia released these drivers to show that they can double their scores in 3DMark, but the game performance will remain the same. Which proves their point (use the real games) precisely.

Is that your deduction, or does NVIDIA state that somewhere? Leaked drivers come from NVIDIA just about every week and they never acknowledge them.

 

[Leon]

Is that your deduction, or does NVIDIA state that somewhere? Leaked drivers come from NVIDIA just about every week and they never acknowledge them.

No, it was Brian Burke interview on Extremetech (or Tom, I don't remeber). He showed new drivers increase 3DMark score two fold, but didn't do squat for UT2003 and other games.

 

[Gstanfor]

Do the new drivers that nVidia released increase performance on all nVidia cards or only on GeForceFX based cards? (I currently have neither 3dmark03 or the new drivers)?

If the performance only increases on GeForceFX, then I would suggest that nVidia has simply made the driver report to applications that the card is Pixel Shader V1.4 compatible (which GeforceFX is, since Pixel Shader 2.0 includes all of PS1.4's inctructions). 3DMark03 specifically looks for PS V1.4 and only for V1.4 in certain tests. If it doesn't find V1.4 it will revert to V1.1 regardless of whether V2.0 is supported - a fairly stupid way of doing things I would suggest.

If the performance increase applies to GF4/GF3 also, then they must be optimizing the driver in other ways (either that or there are additional optimizations for earlier cards AND the drivers do what I described above when GF-FX is detected).

 

[magomago]

the nforce2's issues still aren't ironed out yet....



But I know it sounds stupid (and most of you won't care) but I love the attitude here - "I'm a benchmark junkie, but if my system does horrible at 3dmark but still plays all my games at a level that I enjoy...screw the benchmark"

Other places I have been places too much emphasis on 3dmark and people are mad and angry because their system can't push another 400-450 while another near-identical system can.

Its the AT effect

 

[Tarmax]

3DMark03 specifically looks for PS V1.4 and only for V1.4 in certain tests. If it doesn't find V1.4 it will revert to V1.1 regardless of whether V2.0 is supported - a fairly stupid way of doing things I would suggest.

Say hello to the lazy programmers.

Also, I have a few thoughts on this: In Futuremark's "response" to nVidia's lab report, they mention that Pixel Shader 1.4 is the most efficient way of doing things for the given scenes. Now, I'm no programming junkie nor do I know everything about DirectX, but from what I think I know, isn't Pixel Shader 2.0 supposed to make life a hell of a lot easier compared to v.1.4.. as far as commands and such (talking about with the aid of Cg and RenderMonkey)?

Another question/thought/opinion: Futuremark also states that there is no difference between PS versions 1.1 and 1.3. Now, I'm sorry, but that sounds totally wrong. If there's no difference between the two versions what so ever, then why is there even a v.1.3?? ...why doesn't it just go from v.1.1 to v.1.2 (1.4)??? It just doesn't make any sense to me any more =/ lol

..probably because it's way past my bed time =S ..ugh.. got work tomorrow =/

 

[NOX]

Originally posted by: Codename49

Originally posted by: Pariah

Can you name one announced DirectX 9 game?

Impossible Creatures. And it's not only announced, it's already availible.

Well, if someone is going to justify DX9 at this time it might as well be Microsoft.

Microsoft Games Impossible Creatures

 

[Nemesis77]

Originally posted by: Pariah

But 3DMark03 is no more "bogus" or "irrelevant" than 3DMark01/3DMark01SE & THAT is my main beef with NVIDIA and their "run to mama" antics now.

Yes it is.

2001SE Results

Rankings in this order: 9700Pro,9700,Ti4600,9500Pro,Ti4200,9000

Most people would agree that is they way they would be ordered in real games as well with even the percentage difference pretty close. Now you look at 2003 and you find the 9700Pro way out it in front followed by the 9500Pro way in front of everything else. No games available in any way duplicate these results at 1024 with no AA or AF which is what this benchmark defaults to. The almost complete irrelevance of the CPU speed is also not indicative of any real world gaming situation.

(this is more or less what I wrote about this on nvnews's forums)

Let me educate you what 3DMark is:

3Dmark is suppsed to showcase how the vid-card perform using upcoming tech. In case of 3DMark03, it's focused on DX9 and PS1.4. R9700 and 9500 fully support both, GF4 does not. THAT is the reason why GF gets beaten! And it should: it doesn't properly support the tech 3DMark03 benchmarks! It's a benchmark focused on advances shaders and DX9, NV chose not to support those in their mainstream-models. And that really is NV's problem, not Futuremarks! Or should Futuremark modify their benchmark so NV's crippled vid-cards would look better on it?

THis is just NV whining even though they have nobody but themselves to blame! 03 tests PS1.4 and DX9. Ati's entire lineup support both (with exception to 9100 that doesn't have DX9), NV's does not (large part of their lineup is still DX7 (GF4 MX)!). NV has been holding the industry back by not supporting superior shaders (1.4 as opposed to 1.1), whereas Ati supports the latest tech from their low-end models to their hi end models! NV has no-one else but themselves to blame! If they choose to keep on pushing old tech, they have no right to whine if their products don't do so well in benchmarks that test the latest tech! If NV wants to fix this situation, maybe they should stop pushing their crippled DX7 vid-cards in to the market!

Can you name one announced DirectX 9 game?

That's not the point. 3Dmark does not test performance in current games, it tests performance in upcoming games! It gives us a glimpse what the performance will be in games using DX9. That's the point of this benchmark and you obviously have not realized that!

 

[Nemesis77]

Originally posted by: Pariah

But I disagree about the CPU comment. If you want to see how your CPU performs, look at the CPU Marks...

Take a look at this chart at Rage3D:

Rage3D

Notice something odd about the CPU marks for these identically configured systems except the video card? Still think I should look at CPU Marks to see how my CPU is performing? This whole test suite is bogus.

Hell, With 3DMark01 people whined that it was too CPU-bound. So they made 03 in such way that it's limited by vid-card. And again, people whine! I guess some people are never happy...

You just don't understand what 3DMark is supposed to do. Are you REALLY surpsised that GF4 (a DX8 vid-card) doesn't do well in DX9 benchmark?!?!?! 3DMark is meant to test performance in upcoming games that take advantage of DX9 and advanced shaders (PS1.4). NV doesn't support those. So of course they do badly in this benchmark!

It would be same if people whined because Athlon XP does poorly in a benchmark that tests performance of SSE2! It would perform poorly because it does not support that tech!

 

[Nemesis77]

Ati said it best:

ATI fully supports this new graphics benchmark, 3Dmark03. We feel it is a valuable diagnostic tool for estimating the performance of current and future graphics products when running future game titles (to be released over the next 1-2 years). We believe that using synthetic benchmarks such as 3DMark03 in combination with real applications and game benchmarks provides the most complete picture of the overall performance and value of graphics hardware.

3DMark tests future tech, not todays tech! It tests tech that is not yet widely used in order to give you some idea how your vid-card will perform when that tech is used. Ati's lineup is alot more future-proof (with support for PS1.4 and DX9) than NV's current lineup is (which is still for large parts, DX7!!!), and 3DMark showh this! NV has got lazy. They have been pushing yesterdays tech for far too long, and now they pay the price.

The one thing that annoys me here is that many people who whine about 3DMark just parrot the party-line spoon-fed to them by NV!

 

[Czar]

Originally posted by: NOX

Originally posted by: Codename49

Originally posted by: Pariah

Can you name one announced DirectX 9 game?

Impossible Creatures. And it's not only announced, it's already availible.

Well, if someone is going to justify DX9 at this time it might as well be Microsoft.

Microsoft Games Impossible Creatures

I know of another dx9 game, currently I'm beta testing it and it I dont think I can name it yet

 

[Pariah]

Originally posted by: Codename49

Originally posted by: Pariah

Can you name one announced DirectX 9 game?

Impossible Creatures. And it's not only announced, it's already availible.

After reading a few reviews it didn't seem like anyone knew whether it was DX9 or not, just because it has DX9 on the CD doesn't mean it is. For the sake of arguement lets say it is. Can anyone here produce data that shows a 9700Pro is 3x faster than a Ti4600 in this game, or that a 9500Pro is more than twice as fast w/out FSAA of AF enabled?

Hell, With 3DMark01 people whined that it was too CPU-bound. So they made 03 in such way that it's limited by vid-card. And again, people whine! I guess some people are never happy...

When something is labeled CPU Mark, I don't know, but for some reason that leads me to believe the CPU should be the bottleneck. Maybe you can tell me why changing the video card can affect the score at least up to 25%, probably more, since no one else seems to be able to.

 

[Spicedaddy]

When something is labeled CPU Mark, I don't know, but for some reason that leads me to believe the CPU should be the bottleneck. Maybe you can tell me why changing the video card can affect the score at least up to 25%, probably more, since no one else seems to be able to.

Go read the White Paper at futuremark.com...

CPU TEST

The 3DMark03 CPU test allows users to evaluate CPU performance for 3D graphics workloads. It is important to note that this test is intended to only measure CPU performance for 3D graphics usage and not for general PC usage. For the latter, a benchmark such as PCMark, available also from Futuremark, is more appropriate.

This test runs game test 1 at a resolution of 640x480 using software vertex shaders. Game test 1 is very amenable to CPU measurement as the materials used are simple and most if the screen consists of single-textured, low polygon background objects. We also run game test 3 at a resolution of 640x480 with the use of dynamic shadows disabled. The optimized code path for 1.4 pixel shaders is disabled, since that feature should not be credited in this test. Note that this test still depends somewhat on the graphics hardare, but these settings reduce the dependency.

 

[kazeakuma]

For all those saying how 3dmark03 test future tech etc. Can anyone explain why it only has one DX9 test and not a fully DX9 test at that?
3dmark has never been more than eyecandy IMO, pretty to look at but means nothing. As people have said, 3dmark isn't representative of a real game (future or current). If you were a programmer would you use PS1.1 as a fallback for PS1.4/PS2.0 when PS1.3 is easier and more representative of the market out there? (GF4 Ti4200s out there by the droves) No you wouldn't, so why did Futuremark? Either they got lazy or Nvidia pissed them off (dropping their subscription) and this is mostly political.

The one thing that annoys me here is that many people who whine about 3DMark just parrot the party-line spoon-fed to them by NV!

Of course, now that Nvidia has discredited 3dmark (for whatever reasons), anyone who says the same is spouting the party line despite the fact that alot of us never put any stock in 3dmark in the first place.

 

[NOX]

Originally posted by: Czar

Originally posted by: NOX

Originally posted by: Codename49

Originally posted by: Pariah

Can you name one announced DirectX 9 game?

Impossible Creatures. And it's not only announced, it's already availible.

Well, if someone is going to justify DX9 at this time it might as well be Microsoft.

Microsoft Games Impossible Creatures

I know of another dx9 game, currently I'm beta testing it and it I dont think I can name it yet

Yeah, I know a few (and I mean few) DX9 titles that are in beta, but they wont see the day of light till at least 5+ months from now.

Originally posted by: kazeakuma
For all those saying how 3dmark03 test future tech etc. Can anyone explain why it only has one DX9 test and not a fully DX9 test at that?

Good question. Anybody?

 

[Czar]

Originally posted by: NOXYeah, I know a few (and I mean few) DX9 titles that are in beta, but they wont see the day of light till at least 5+ months from now.

this one is supposed to be out next month I think