NVIDIA Says Built-In Crysis GPU Benchmark Not Good To Use

[geokilla]

link

Pretty interesting e-mail from NVIDIA.

I've been testing Crysis quite extensively over the past few weeks. It turns out that the benchmark_GPU isn't quite good at measuring graphics due the speed of the camera moving across the map. It causes a lot of data streaming and ends up reducing SLI performance by 15-20%.

 

[AzN]

Nvidia blames a benchmark.

 

[IL2SturmovikPilot]

I never use it much anyways I just start up the game,activate "r_displayinfo 1" (Without the quotes of course) and see how many FPS i get while throwing KPA soldiers into each other and blowing houses up

 

[Munky]

Originally posted by: IL2SturmovikPilot
I never use it much anyways I just start up the game,activate "r_displayinfo 1" (Without the quotes of course) and see how many FPS i get while throwing KPA soldiers into each other and blowing houses up

An even better benchmark would be activating "r_displayinfo 1" and seeing how many fps you get while blowing up aliens in the carrier level.

 

[IL2SturmovikPilot]

Originally posted by: munky

Originally posted by: IL2SturmovikPilot
I never use it much anyways I just start up the game,activate "r_displayinfo 1" (Without the quotes of course) and see how many FPS i get while throwing KPA soldiers into each other and blowing houses up

An even better benchmark would be activating "r_displayinfo 1" and seeing how many fps you get while blowing up aliens in the carrier level.

Alright,i admit defeat THIS time

 

[apoppin]

it's true .. *i* have been saying that since it came out

it is a DEV tool .. *made over* into a "pretty tech" demo that's being used as a benchmark in place of "something better"

Let's make a GOOD one that everyone will use .. 3 or 4 at least!
*US*

... ATF ... Video
-together we can do anything

 

[chizow]

The built-in demo is fine, and is actually an excellent benchmark in terms of consistency. All he has to do to eliminate any "streaming data" concerns is to use "e_precachelevel 1" and run the demo more than once (which most reviewers do anyways). After that any slowdowns caused by large data transfers is a limitation of SLI or current PC specs.

 

[apoppin]

Originally posted by: chizow
The built-in demo is fine, and is actually an excellent benchmark in terms of consistency. All he has to do to eliminate any "streaming data" concerns is to use "e_precachelevel 1" and run the demo more than once (which most reviewers do anyways). After that any slowdowns caused by large data transfers is a limitation of SLI or current PC specs.

i have always run the demo at least 4 times - and i toss the first run out
- especially for Vista

.. but still, that addition is awesome info .. thank-you!!
--i will add that to my own benchmarking!

However, just as with FarCry, i think there needs to be a lot of updateable benchmarks created that will stress specific parts of the Graphics systems. Crysis is going to be an important benchmark for at least 3 or 4 more years

 

[Piuc2020]

Maybe we should create our very own Anandtech suite of Crysis benchmarks

 

[apoppin]

.. all recent relevant benchmarks .. updated regularly with each driver release
-posted for anyone to use and compare with on his own rig .. to Blow away 3DMark's DB!




sounds good .. beyond good but .. lotssa work

 

[nRollo]

I got this email from NVIDIA as well, and will be posting my findings on it soon with a single 9800GX2, and two.

 

[bryanW1995]

drat! I was betting that you were getting 3x9800gtx in tri-sli...I'll put my "amd rulz" t-shirt up against your quad sli 9800gx2's any day!

 

[nRollo]

Originally posted by: bryanW1995
drat! I was betting that you were getting 3x9800gtx in tri-sli...I'll put my "amd rulz" t-shirt up against your quad sli 9800gx2's any day!

LOL

I could wear the "AMD rulz" shirt to cloak my true motives from the nOObs, and increase productivity 21%.

:beer:

 

[bryanW1995]

that's the problem with marketing. I mean, how the heck do you measure productivity? number of posts? number of words typed? do you give people a double super secret order code so that they get a free pack of rollo's when you influence their purchase? My evil amd masters don't even return my calls...or emails...or text messages...or faxes...I did get a "hector blows" shirt the other day, but I think that it was from a disgruntled am2+ buyer. anyway, a 21% increase in productivity, if it could actually be measured, would definitely be a worthwhile goal. that would give nvidia a 98.97611 % market share instead of 98.97610 % share.

 

[geokilla]

What are all these commands that you guys are talking about?

Thought it'd be a good idea to post this, but too much OFF TOPIC comments here. Don't screw this thread up guys. Please??

 

[bryanW1995]

sorry. I find it very interesting that nvidia is complaining about the crysis benchmark since they've looked so good in it with the 8800 series. I guess that a 2 pcb 2 gpu one slot solution isn't exactly the same as sli after all...

 

[BFG10K]

It's quite interesting nVidia chose to send this email now, with the impending release of the 9800 GX2, and one can't help but wonder if said card won't have great results in the benchmark.

It also lends credibility to [K]yle's claims about performance gains in the benchmark not necessarily translating to actual game-play.

 

[v8envy]

Live by the synthetic benchmark, die by the synthetic benchmark.

Crysis is a CPU pig. Is anyone shocked that a multi-GPU solution isn't going to scale well with that title? I'm looking forward to the single digit frame rates reviewers post at 25x16 -- the only reason to have multi-GPU.

OTOH having lights dim when you fire up your 6-gpu box would be its own reward, even if the final peformance < single 8800 Ultra.

 

[Foxery]

Classic "timedemos" used to be a recording of one of the devs playing through an actual level, behaving normally. The Crysis benchmark doesn't look like actual gameplay. The camera floats 30 feet off the ground, flies at breakneck speed, and doesn't approach detailed objects closely.

Does Crysis include an ability to make & play back recordings? If not, this is an unforunate oversight...

 

[nemesismk2]

Nvidia probably saw this Crysis benchmark result and wondered why anyone would buy the 9800 GX2 for this kind of performance increase over the 8800 GT OC and HD 3870 X2?

Still it's good to see the Sapphire HD 3870 X2 performing so well with UT3

In all honesty the 9800 GX2 is suffering badly from poor driver, it should? be much faster than the 8800 GT OC IMO

 

[apoppin]

Originally posted by: Foxery
Classic "timedemos" used to be a recording of one of the devs playing through an actual level, behaving normally. The Crysis benchmark doesn't look like actual gameplay. The camera floats 30 feet off the ground, flies at breakneck speed, and doesn't approach detailed objects closely.

Does Crysis include an ability to make & play back recordings? If not, this is an unforunate oversight...

FRAPS does .. would you like me to make one for you?

ANd many benchmarks are like this .. STALKER is *extreme* as the camera actually moves in and out of the cells - looks up at the sky, burrows "underground". . . . goes thru walls ...
:roll:

STALKER benches also leave much to be desired .. BUT IF you can run one OK, you will breeze thru actually playing it - even in the firefights.
Did you notice that they *ALMOST NEVER* have a benchmark demo of a "firefight" ... it is a "run"
:Q

However, playing ingame, when you get stuck in the middle of a particularly intense battle - with effects going off in all 4 corners of your screen at once, did you notice that your rig may tend to occasionally *bog* and even *chug* ... even though you ran the benchmark OK?
. . . sometimes it is the AI and/or just massive Graphics overlaod ... and the CPU-GPU-Memory stop working together smoothly and then it goes to HD and everything tends to come to a grinding halt.

 

[taltamir]

yes, most time demos are crap.
It will be interesting when 3dmark comes out with their new benchmark suites. Those will be actual games with very detailed benchmarking built into the game. I am expecting a time-demo done right...

 

[Sylvanas]

Yeah I have always thought the built in GPU bench was kinda useless but people still insist on using it... :/. The best way to gauge your performance is get in there and PLAY the game, for instance with driver updates- if there is an increase you'll know it. But for the purposes of benching games....I guess it comes down to fraps.

 

[chizow]

Originally posted by: geokilla
What are all these commands that you guys are talking about?

Thought it'd be a good idea to post this, but too much OFF TOPIC comments here. Don't screw this thread up guys. Please??

Its a command to load all level textures to memory to the extent of your OS/physical memory limitations. You will notice a longer initial load, but much less HDD accesses while playing. Save game loads should be near instant. Most reviews specifically toss out 1st run benches because of this "streaming" where textures are accessed from the HDD and loaded to memory, any other complaints about "streaming" simply comes down to limitations with current memory controllers, memory bandwidth, frame buffer etc. and as such cannot be avoided without better platform hardware.

Originally posted by: Sylvanas
Yeah I have always thought the built in GPU bench was kinda useless but people still insist on using it... :/. The best way to gauge your performance is get in there and PLAY the game, for instance with driver updates- if there is an increase you'll know it. But for the purposes of benching games....I guess it comes down to fraps.

The demo is actually much better at precisely that....as its repeatable and accurate which allows accurate comparisons to be made between different configurations and drivers. A subjective "game play" demo, like the ones HOCP advocate are much more prone to error. Personally I prefer a mix of both, in-game demo to benchmark and compare with other results out there and then game play demo/benches to get an idea of actual relative performance. From that I can draw parallels between changes in the demo to what I can expect from actual game play performance.