Everyone in the enthusiast community knows that manufacturer-provided benchmarks must be taken with a grain of salt. One could write a book on the various ways that companies tend to shade the truth to paint their own products in a positive light. Some of these practices are defensible, at least to a certain degree — a company that chooses to put its best foot forward by selecting tests in which it performs well may have a perfectly defensible argument if the tests it chooses are well-known industry standards and represent workloads the component is expected to run.
Other times, however, the changes companies make when comparing their hardware to other systems aren’t defensible. And sometimes, they cross the line from “favors our own products” into “blatantly misrepresents the performance of the competition.”
Intel — or to be precise, a company Intel hired to create a whitepaper on Core i9 gaming performance — has crossed that line. According to Forbes, Intel contracted with Principled Technologies to distribute a whitepaper containing various claims about gaming performance between Intel’s upcoming Core i9-9900K and Core i7-8700K and the AMD Threadripper 2990WX, 2950X, and Ryzen 7 2700X. With AMD having surged into competitive positioning in the past 18 months and Intel taking heat from its 10nm delays, Chipzilla has every reason to push a narrative that puts it in the driving seat of gaming. But Intel is using this whitepaper to claim that it’s up to 50 percent faster than AMD in gaming based on Ashes of the Singularity in particular, and that’s where the problems start. The Intel results are somewhat higher than we’d expect, but the AMD CPUs — particularly the Ryzen 7 2700X — are crippled.
There are several problems with the AMD benchmarks as run by Principled Technologies. PT was careful to document its own configuration steps on both systems, which is why we know what, precisely, the company did wrong.
First, the Ryzen systems were tested without XMP enabled. XMP is the high-end memory timing standard that enthusiast kits use to hit maximum performance and Ryzen gaming performance is often tied directly to its RAM clock and sub-timings. Using substandard timing could lower Ryzen’s performance by 5-15 percent.
Second, all of the benchmarks in question were run using a GTX 1080 Ti and a resolution of just 1080p. If you wanted to create a report tailor-made to Ryzen’s weaknesses, that’s the resolution you’d use. Unfair? Not necessarily — it’s the most common resolution after all. But there’s a reason we include 1440p and 4K results in our resolutions comparisons for gaming, and Intel/Principled didn’t do so.
Third, Principled Technologies notes that it enabled “Game Mode” in AMD’s Ryzen Master utility. The implication is that it did this on both systems. This can have serious side effects on how well an AMD system benchmarks. On Threadripper, engaging Game Mode cuts the CPU core count in half and enables NUMA to allow the remaining CPU cores to schedule workloads on the cores closest to the memory controllers. On Ryzen 7, clicking Game Mode just cuts the core count in half. That’s why AMD’s user guide for Ryzen 7 specifically states that Game Mode is reserved principally for Threadripper and that Ryzen customers shouldn’t use it:
If Principled had consulted AMD’s documentation, it would’ve seen that it shouldn’t be using this test mode for Ryzen 7 in any case. If it didn’t consult AMD’s documentation, it had no business using Ryzen Master to adjust Ryzen 7 CPU settings. But the 50 percent performance gain that Intel claims for itself is exactly the kind of result we’d expect if the 2700X had been crippled by having its CPU neutered.
Their Core i7-8700K is actually a touch slower than ours, but our Ryzen 7 2700X is a massive 1.36x faster. While our results use different detail settings, TechSpot actually checked the exact results with AotS benchmarks of their own. In the graphs below, red bars indicate Principled Technologies results.
Their Assassin Creed Origin tests are similarly broken:
Because they’re effectively benchmarking the Ryzen 7 2700X as a quad-core CPU with lousy memory timings, it’s no particular surprise that the Ryzen 7 ends up getting its a_ss kicked. This goes beyond simply adjusting a few game settings in a way that favors your hardware but subtly disadvantages the competition. The Ryzen 7 2700X has been configured to run with half its cores disabled in a non-optimized memory configuration with sub-optimal timings while the Intel system was configured with an ideal memory subsystem and all of its cores and threads enabled.
Misrepresenting product performance by 3-5 percent is a tilt. Misrepresenting it by 1.2x (AotS) and almost 1.25x (as in ACO) is a lie. And that means these results are lies. They may be lies of ignorance or error rather than the result of a deliberate malicious intervention, but given Intel’s history, enthusiasts are unlikely to extend much benefit of the doubt. Even a casual readthrough of the document ought to have caught these mistakes — if, in fact, they were mistakes. And even in the most charitable reading, Principled had no business using an application like Ryzen Master if they weren’t going to read the documentation AMD provides to tell you how to use the damn thing. Anybody can have a test run go poorly or mistype a number, but TechSpot found evidence of manipulation in every single benchmark they checked. Either the 8700K was strangely faster than it ought to have been, the 2700X was significantly slower, or both.
What makes the entire affair that much more perplexing is that we’d expect Intel to win this comparison anyway. There was no need to resort to crippling the 2700X to pull ahead. The company could’ve done that simply by using 1080p and choosing tests where Ryzen doesn’t compete as well. The sharp-eyed would call foul, but people are used to taking vendor tests as preliminary indications at best. Instead, Principled Technologies has called into question its own expertise and raised serious questions about what, exactly, Intel was attempting to accomplish with this whitepaper.
When asked for comment by Forbes, Intel responded:
“We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world’s best gaming processor. Principled Technologies conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third party testing in the coming weeks.”
Guys, I don’t know what you think “in spec” looks like, but running the 2700X with half its cores disabled doesn’t fit the bill.
After the outcry, Principled Technologies is currently retesting AMD processors using the correct mode. They used the correct DOCP memory settings from AMD, something they didn't say before. It is unknown if they will also run the retests with the same aftermarket cooler they used with Intel's processors, presumably they will. https://www.tomshardware.com/news/intel-amd-gaming-benchmarks-controversy,37912.html
I don't have tech friends to talk about this with, but I just want to express how angry I am with Intel over this. I've never owned or built anything but Intel machines previously, but lately I was very much set on the Ryzen 2700X. With this revelation, I don't think I want to do business with Intel anymore. This opened a whole can of worms regarding that company that I never knew about, including how many times they've rigged benchmarks against AMD in the past and the lawsuit they lost over it. I hope that AMD can manage to develop much improved IPC and overclockability with Ryzen. If they can get within spitting distance of Intels single core performance, I'll be a permanent convert.
Two days ago, Intel released a whitepaper it commissioned from Principled Technologies that contained errors so significant, they rise to the level of lying about its competitor’s performance. Principled Technologies has now released a response to these claims defending its own work — and ExtremeTech has performed its own partial confirmation of the claims PT makes about its Intel CPU results.
In his response to various allegations and claims, the co-founder of Principled Technologies, Bill Catchings, has provided the following statement. His full statement can be read at WCCFTech, we’ve reprinted the relevant sections below:
Use of “Game Mode” on the AMD Ryzen™ 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD’s recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling the Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper™ and Ryzen™. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with the new results.
Memory speeds: To have complete parity across all systems, and to allow the Intel® Core™ i9 X-series and AMD Ryzen™ Threadripper™ to fully utilize memory bandwidth, we used 4 16GB DDR4 DIMMs on all configurations. We took the following memory configuration steps.
The rest of Catchings response deals with various topics we didn’t raise at ET and aren’t as concerned about with regard to their impact on the test results. The above two points are the major ones — and they’re the ones we need to discuss in a bit more detail.
First, Principled Technologies clarifies that, in fact, it made a catastrophic error when configuring Game Mode. It’s true that AMD recommends enabling Game Mode for Threadripper in certain instances. But AMD also makes it clear that Game Mode is intended for Threadripper. The following image is from Page 26 of AMD’s Ryzen Master Quick Reference Guide:
Furthermore, AMD’s Overclocking Guide for Ryzen Master makes the function of Game Mode extremely clear. Page 25: “When turned on with Ryzen or Ryzen Threadripper processors offering >4 physical processors, Legacy Compatibility Mode reduces the active logical processor resources by half.”
There’s nothing wrong with testing advanced, enthusiast features, but one typically reads the instructions first before activating them. The best explanation for PT’s work is that the company failed to perform elementary due diligence in this matter.
The company’s explanations for its memory configurations, however, do not pass elementary scrutiny. We are told that the RAM configurations used were chosen “to have complete parity across all systems.”
This is a lie.
XMP was enabled on the Intel Z390, disabled on the X299-Deluxe, and enabled on the Prime Z370-A used to test both the Core i7-8086K and the Core i7-8700K. For Threadripper, the D.O.C.P. Standard setting was used (Asus has called this “XMP by another name”). The Asus Prime X470 Pro motherboard used to test the 2700X didn’t enable XMP or DOCP at all. In other words, out of five different motherboards, three had XMP or an equivalent enabled and two did not. RAM clocks were apparently set according to maximum manufacturer specifications. But choosing to use manufacturer-specified memory clocks and leaving XMP disabled and enabled at turns means the memory subsystems of these platforms were not configured for maximum parity.
It is remarkably surprising that the mainstream AMD system that represents the Core i9-9900K’s direct competition both disabled half its CPU cores and managed to utilize neither XMP or DOCP with no explanation provided for this configuration decision. Keep in mind, XMP is an Intel-optimized standard to start with. It may improve performance on AMD chips compared with whatever garbage compatibility timings are implemented by default, but it isn’t AMD-optimized. Even enabling XMP on all systems doesn’t truly constitute a level playing field, but enabling it for some AMD platforms and not for others creates an additional tilt.
If Principled wanted to configure for maximum parity, it should have tested with XMP fully enabled or disabled across all products and all systems using the same RAM clock. If it wanted to test in a manufacturer’s approved configuration, it should have stated it was doing so and explained why XMP / DOCP was enabled or disabled in each specific circumstance relative to the best practices communicated by AMD and Intel. What we got was neither.
The situation gets even stranger once you see that even some of PT’s Intel performance claims are odd. Looking through the whitepaper, I realized Principled Technologies had used an Asus Prime Z370-A motherboard running UEFI 1406 with a GTX 1080 Ti — which happens to correspond exactly to the system I used for our recent review of the RTX 2080 and 2080 Ti. They were even kind enough to test many of the same games.
This made me curious. Could I validate the Intel results they reported for the Core i7-8086K? For the most part, yes — with some notable and odd exceptions.
This isn’t a complete validation cycle; only part of our benchmark family overlaps. I re-tested the Core i7-8086K and GeForce GTX 1080 Ti according to the game settings, Windows configuration details, and UEFI settings detailed in the PT whitepaper. I also tested them with a single setting changed. It turns out that if you change the “Power Saving and Performance Mode” UEFI option to “Performance Mode” and away from “Auto,” you also disable all C-states and lock the CPU at a flat 4.3GHz constant clock with no Turbo whatsoever. It seems odd to lock a chip at a single static frequency in a “Performance Mode,” but I wanted to see what difference it might be making to the final results.
I tested our Core i7-8086K in two different configurations: One configuration corresponds exactly to the steps described in Principled Technologies’ whitepaper — no more, no less. One configuration corresponds to those configuration steps, but with one change: I left the “Performance Mode” setting on Auto. And, of course, we’ve included Principled Technologies’ own results. Here are the results, graphed to two charts for readability:
Things mostly check out fine, with two glaring oddities and one smaller one. Both of the major problems are in Warhammer II. First of all, Warhammer II shouldn’t be tested in DirectX 12 with an Nvidia GPU, as we discussed earlier this week. But second, the results reported for the Skaven benchmark are far too low. Our own Core i7-8086K + GeForce GTX 1080 Ti scored 87fps when run through that benchmark in Ultra mode. Principled Technologies reports just 83fps at “High” detail, our re-tests of the scene at that quality level put our score 1.25x higher.
There’s a similar problem in the Warhammer II: Laboratory test. We’re told to expect a result of 41; our actual result was 48 or 49, a gain of 1.17x – 1.19x.
Finally, in the Final Fantasy XV benchmark, the score of 10,858 reported by PT was 7-8 percent higher than our own results in both configurations. There was no clear advantage or disadvantage to enabling or disabling Performance Mode in the tests we could verify, but we were unable to confirm benchmark results in three out of the 13 tests we investigated. That’s not a great finding for a project when we already know the AMD CPU was blatantly misconfigured. I don’t know why I can’t replicate PT’s results in every test — but I know I’ve done similar validation of benchmark results by using Reviewer Guides provided by Intel and AMD on many occasions. Something is wrong here.
I don’t know if Principled Technologies is responsible for this error or if Intel is — by which I mean, I don’t know if Principled Technologies simply misunderstood how AMD’s utility worked or if they were told to misunderstand how AMD’s utility worked. Ryzen Master’s UI isn’t identical for Threadripper versus Ryzen 7; the Legacy Compatibility Mode option has additional switches that don’t appear if you’re only running it on an eight-core. This should’ve clued PT in to the fact that the capability functioned differently on different microprocessors. Both of AMD’s documents make it plain that “Game Mode” will disable CPU cores. Coverage of Ryzen since launch has made this clear as well.
Companies are typically extremely careful when they talk about their competition because it can open them up to legal liability if they’re found to be making false claims. In many cases, companies prefer to simply talk about “the competition” even when they literally compete with one other company. Engaging a third-party to write a whitepaper is one way of offering additional information that isn’t seen as coming directly from Intel itself. But make no mistake — Principled Technologies is a company writing a document for a client, which means the client has oversight and input into both the test conditions and the final product. If Intel doesn’t like a setting, game, or performance test, that test isn’t in this document.
Neither Intel nor Principled Technologies have responded to our emails. Could this be the result of an honest mistake or oversight? Of course. It wouldn’t be the first time someone flipped a wrong switch and then spent 10 hours cheerfully benchmarking the wrong hardware configuration, believe me. But there are problems, plural, with this whitepaper, its data, Intel’s failure to respond to it, and the fact that Principled Technologies is still trying to slide out of from acknowledging that it published verifiably false results with a lame reference to retesting AMD CPUs that it should have known were misconfigured in the first place. The company continues advancing a false narrative about its memory configuration and how it supposedly established memory configuration parity between testbeds when it did nothing of the sort.
At a time when Intel needs to be building a narrative around the Core i9-9900K as its first mainstream eight-core desktop processor, a demonstration of forward progress, and a distraction from its own security concerns and manufacturing delays, we’re instead having an extended conversation about whether the chip giant attempted to lie to consumers by proxy. It’s not a good look.
I see so much disparity for benchmarks I have become cynical
way i see it is does the game work on the rig I own or now and I have ashes so i can try any cpu or gpu i want
They published updated results, though they did NOT use the aftermarket cooler they used on the 9900K, so still wait for the third party benchmarks, especially since they were done at 1920x1080 and a GTX 1080 TI, not something someone with $3000 in hardware would game at.