AT Posts Revised benchmarks Ryzen/Intel, much close to other sites

[frozentundra123456]

Gaming results here link

Results are much more in line with other testing sites. Anomalous results were due mainly to HPET timings, not spectre/meltdown patches as some in these forums claimed.

8700k at stock is equal or faster than Ryzen in every game tested, with overclocking headroom that would only widen the gap.

 

[Malogeek]

Wow that's some huge differences for the Intel

 

[tamz_msc]

Bleh, still needs FCAT analysis with external capture to ensure that in-game metrics aren't misreporting the numbers.

 

[moonbogg]

LOL wow. As if everyone didn't already know something was way off there.

 

[IEC]

Bleh, still needs FCAT analysis with external capture to ensure that in-game metrics aren't misreporting the numbers.

HWBOT requires HPET being enabled for Windows 8+ for some of their benchmarks for this very reason. From the Ryzen launch:
https://hwbot.org/newsflash/4335_ry...bias_w88.110_not_allowed_on_select_benchmarks

For reliable performance measurements with at run-time overclocking, we recommend enabling the High Performance Event Timer (HPET). Alternatively you can opt to use a Windows 7 based operating system.

 

[CHADBOGA]

LOL wow. As if everyone didn't already know something was way off there.

Think of those people who made fools of themselves INSISTING that Ryzen was equal to the i7 8700K in gaming.

They have been embarrassing themselves for months now, but they really outdid themselves with this effort.



This is both trolling and inflammatory posting.


esquared
Anandtech Forum Director

 

[Billy Tallis]

Anomalous results were due mainly to HPET timings, not spectre/meltdown patches as some in these forums claimed.

Note that while it's true this wasn't simply due to whether or not the Spectre/Meltdown patches were installed, Meltdown still plays a major role in this mess. Without the Meltdown workaround, the HPET overhead is reasonably small and about the same on Intel vs AMD. With the Meltdown workaround on Intel, the HPET overhead skyrockets and configuring the system to use the HPET instead of the TSC suddenly has a huge extra effect that only shows up on one CPU brand.

 

[tamz_msc]

HWBOT requires HPET being enabled for Windows 8+ for some of their benchmarks for this very reason. From the Ryzen launch:
https://hwbot.org/newsflash/4335_ry...bias_w88.110_not_allowed_on_select_benchmarks

The concern of this thread seems to be gaming, however the more interesting situation is regarding the compile benchmark, since the article claims the following:

• On Intel:
  

  The biggest gains here were in the web tests, a couple of the renderers, WinRAR (memory bound), and PCMark 10. Everything else was pretty much identical. Our compile tests gave us three very odd consecutive numbers, so we are looking at those results separately.

• On AMD:
  

  This is a lower gain, with the biggest rise coming from PCMark10’s video conference test to the tune of +16%. The compile test results were identical, and a lot of tests were with 1-2%.

I'm more interested if they can independently verify The Stilt's findings.

 

[Zucker2k]

Bleh, still needs FCAT analysis with external capture to ensure that in-game metrics aren't misreporting the numbers.

LOL! Here we go!

 

[Zucker2k]

HWBOT requires HPET being enabled for Windows 8+ for some of their benchmarks for this very reason. From the Ryzen launch:
https://hwbot.org/newsflash/4335_ry...bias_w88.110_not_allowed_on_select_benchmarks

First of all, it states clearly AM4; the bias only affects Ryzen. Secondly, there are no game benchmarks in the 1.6 guide.
http://hwbot.org/article/general_rules

 

[epsilon84]

Why am I not surprised?

I said it from launch day that Anandtechs results were an outlier, when every other reviewer shows the 2700X about 10% behind at 1080P gaming and Anandtech shows the 2700X as 17% faster...

 

[IEC]

First of all, it states clearly AM4; the bias only affects Ryzen. Secondly, there are no game benchmarks in the 1.6 guide.
http://hwbot.org/article/general_rules

Aside from pointedly ignoring my overall point, those rules you cite specifically deal with the fact that Windows 8/8.1/10 ALL have real-time clock issues:

1.6 – Windows 8 Restrictions
Due to severe validity problems with the Windows 8/8.1/10 real time clock (“RTC”), not all benchmarks results achieved with Windows 8/8.1/10 can be trusted. The main problem lies with the RTC being affected when over- or underclocking under the operating system. The operating system uses the RTC as reference clock, and benchmarks use it to reference (benchmark) time. Refer to our original coverage article for more detailed information. The restrictions for Windows 8/8.1/10 due to the RTC bug can be found below.

• All new Windows 8/8.1/10 Benchmark Results will be blocked, including for on-going competitions
• Existing Windows 8/8.1/10 based benchmark submissions will be blocked if seemingly out of line or world record or top score.
• Existing Windows 8/8.1/10 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above
• Existing Windows 8/8.1/10 based benchmark submissions will not be blocked or removed if they were made on an AMD system (New ones will be clocked)

Just because games aren't listed as HWBOT benchmarks, doesn't mean that there isn't an issue with how game FPS is measured in software. If one second isn't actually one second, of course that can impact results (including "Frames Per Second"). That doesn't mean that there IS an issue either, but it would be nice to independently validate the results.

 

[The Stilt]

Note that while it's true this wasn't simply due to whether or not the Spectre/Meltdown patches were installed, Meltdown still plays a major role in this mess. Without the Meltdown workaround, the HPET overhead is reasonably small and about the same on Intel vs AMD. With the Meltdown workaround on Intel, the HPET overhead skyrockets and configuring the system to use the HPET instead of the TSC suddenly has a huge extra effect that only shows up on one CPU brand.

I saw a huge HPET penalty in games already on KBL, long before GPZ were announced (or patches existed).

 

[Justinus]

Aside from pointedly ignoring my overall point, those rules you cite specifically deal with the fact that Windows 8/8.1/10 ALL have real-time clock issues:
Just because games aren't listed as HWBOT benchmarks, doesn't mean that there isn't an issue with how game FPS is measured in software. If one second isn't actually one second, of course that can impact results (including "Frames Per Second"). That doesn't mean that there IS an issue either, but it would be nice to independently validate the results.

It's my understanding (this may be flawed, we shall see) that the issue isnt that the results are randomly and without reason unreliable. HWBots requirement to enable HPET is to stave off a method of exploit where you can underclock the bus (that drives the clock) with in-OS software, causing some benchmarks or methods of measuring performance to believe time is going more slowly and thus believe more work has been done per second and give an erroneously high score.

I dont believe this is (or as significant as) an issue in a controlled environment, such as a reviewer benchmarking games, provided they did not change the bus speed after entering Windows.

 

[IEC]

It's my understanding (this may be flawed, we shall see) that the issue isnt that the results are randomly and without reason unreliable. HWBots requirement to enable HPET is to stave off a method of exploit where you can underclock the bus (that drives the clock) with in-OS software, causing some benchmarks or methods of measuring performance to believe time is going more slowly and thus believe more work has been done per second and give an erroneously high score.

I dont believe this is (or as significant as) an issue in a controlled environment, such as a reviewer benchmarking games, provided they did not change the bus speed after entering Windows.

It wouldn't seem to be, but the huge differences in results with HPET forced is eyebrow-raising.

 

[PeterScott]

Gaming results here link

Results are much more in line with other testing sites. Anomalous results were due mainly to HPET timings, not spectre/meltdown patches as some in these forums claimed.

8700k at stock is equal or faster than Ryzen in every game tested, with overclocking headroom that would only widen the gap.

I am glad they figured this out, because looking at other benchmarks, it was clear AT was messing something up, so I just stopped looking at their benchmarks.

 

[frozentundra123456]

Why am I not surprised?

I said it from launch day that Anandtechs results were an outlier, when every other reviewer shows the 2700X about 10% behind at 1080P gaming and Anandtech shows the 2700X as 17% faster...

Yea, remarkable how AT could publish such obviously anomalous data without further review, and how a few posters insisted AT was right and all the other review sites didnt do their testing properly.

 

[epsilon84]

Yea, remarkable how AT could publish such obviously anomalous data without further review, and how a few posters insisted AT was right and all the other review sites didnt do their testing properly.

I too, am surprised that they would go ahead and publish such numbers, quite frankly that is just negligence on their part. I mean, imagine the tables were reversed and Anandtech showed a 2700X losing to a 8700K in say, Cinebench, there would be a huge uproar, and rightly so, because such a result shouldn't be possible, given properly configured systems.

The same goes for gaming - there is no way a 2700X could possibly be 17% faster on average than a 8700K at 1080P gaming, yet we get all these theories about how those results could possibly be correct (Spectre/Meltdown patches reducing gaming performance by 30%? Like seriously?!) when in fact, we should have focused on finding out why they were incorrect, and so far out of line with 'expected' results.

 

[darkswordsman17]

Its good that they looked into it and figured out what was happening. Unlike some people, I don't think its the end of the world for them to get wrong results and then look into why. Funny how many have been saying for a long time now that Anandtech has no credibility, but that sure doesn't stop them from visiting here and acting like if they don't set them right by throwing tantrums then it'll cause some cataclysm.

Another thing that's always fun is seeing people that post in the forums and comment on the article and seeing a disparity (the overall tone is typically the same, but notice they kinda become unhinged in commenting on the article, but, I assume because of the active moderation leave a lot of their name calling and other vitriol commentary out of their forum posts). Reminds me of the video forum, where there's some people that try to play moderate in the forums, but you see their comments (especially elsewhere) and you almost have to laugh at how much they must hate having to restrain themselves in order to post here.

I too, am surprised that they would go ahead and publish such numbers, quite frankly that is just negligence on their part. I mean, imagine the tables were reversed and Anandtech showed a 2700X losing to a 8700K in say, Cinebench, there would be a huge uproar, and rightly so, because such a result shouldn't be possible, given properly configured systems.

The same goes for gaming - there is no way a 2700X could possibly be 17% faster on average than a 8700K at 1080P gaming, yet we get all these theories about how those results could possibly be correct (Spectre/Meltdown patches reducing gaming performance by 30%? Like seriously?!) when in fact, we should have focused on finding out why they were incorrect, and so far out of line with 'expected' results.

There are plausible situations where that could be possible. Remember when Rage came out and performance and/or quality was all over the place because of how much they streamed textures in? And didn't they release an uncompressed texture pack (that was like 100GB)? I'd be curious if the Spectre/Meltdown hit on disks would come into play there.

Another area I wonder if there might be some big hits is game streaming (where you have your PC render and then stream it to another device in your home).

Not a consumer workload but that could potentially be a big thing for network game streaming like Nvidia is developing heavily too. And companies are absolutely moving in that direction (to both further lockdown their IP and prevent exploits/cheating; plus it would enable them to sell cheap consoles that don't need to do heavy graphics processing). That has implications beyond Intel potentially as well, since ARM is making inroads in servers, so if Nvidia was looking at offering ARM cores strapped to their big GPUs, it might present problems since weren't newer ARM designs also affected by Meltdown (as well as Spectre)?

 

[frozentundra123456]

I stated when the results came out that they were outliers, and have stated repeatedly even before this, that I put no faith in AT's gaming benchmarks. Still true. To be honest, I *very* rarely actually go to AT itself, but go to the forums because a lot of data is presented from a lot of sites and different news sources. I did pretty much go to the home page every day after these initial results were linked, in order to see how long it would take them to print a retraction. Which is basically what the new data is, although they of course tried to spin it as some sort of interesting new discovery.

Granted it is not "the end of the world" to publish erroneous data, it is only a cpu, after all, and both are competent for gaming. Publishing data like this is much more reasonable for new information, for which we have no precedent, but for data like this that was so divergent from any previous data and that which any other site was obtaining, they certainly should have more carefully examined their methods. And of course, you get the forum posters who cant wait to quote this data. Are they going to go back update every post they made with the erroneous data? I seriously doubt it.

 

[moonbogg]

I'm glad this situation is now ironed out, but it somehow reminds me of that old joke where someone sais something like "I'm driving against traffic on the freeway 'cause all y'all going the wrong dang way" or something. It's kind of funny, lol.

 

[PeterScott]

Its good that they looked into it and figured out what was happening. Unlike some people, I don't think its the end of the world for them to get wrong results and then look into why.

It is not good that it took them this long to figure it out. It gives AT a credibility black eye. It indicates they aren't sanity checking their results enough.

AT used to be my "go to" source. But the last few reviews I noticed some things were seriously out of wack in the review benchmarks and I just started discounting AT as a reliable source. In this latest one. I just did a quick check on the gaming benchmarks to confirm that they were still broken and nothing more.

This is a step in the right direction. Hopefully they can do more rigorous sanity checking going forward.

 

[snstr]

They did not publish "wrong" data, only some of the context was initially unknown. It was data gathered in specific circumstances that may happen to some users out there, even if they may be a minority. And I think that the lessons learned there are highly valuable.

So I think the anandtech team should be thanked for their thoroughness.

 

[Schmide]

Seriously. You publish data you get from your test suite. If it results in anomalies that show or bring light to mitigating factors that affect performance, it is a good thing.

Edit: effect

 

[wahdangun]

I too, am surprised that they would go ahead and publish such numbers, quite frankly that is just negligence on their part. I mean, imagine the tables were reversed and Anandtech showed a 2700X losing to a 8700K in say, Cinebench, there would be a huge uproar, and rightly so, because such a result shouldn't be possible, given properly configured systems.

The same goes for gaming - there is no way a 2700X could possibly be 17% faster on average than a 8700K at 1080P gaming, yet we get all these theories about how those results could possibly be correct (Spectre/Meltdown patches reducing gaming performance by 30%? Like seriously?!) when in fact, we should have focused on finding out why they were incorrect, and so far out of line with 'expected' results.

Technically it's still correct results it's just intel have some bug/performance hit, when using HPET, so if your program need HPET then if you use latest intel cpu, you will get big performance hit.

So what anandtech results was not outlier or incorrect, they just using different environment (in this case HPET always on)