In December 2012, a blog was posted to the AnTuTu Labs website simply titled “Stop cheating.” The post explained that manufacturers, instead of optimizing overall performance, were optimizing software to assist their devices’ performance in benchmarks. Even running a dozen tests and averaging them doesn’t really make the scores any more helpful.įurther, it’s not difficult for OEMs to optimize their own devices for benchmarking. Run Quadrant Standard once, you might score a 6,000. On top of all that, there are major discrepancies between devices, reviews, different outlets, and even individual tests. ICC, for those wondering, is a proprietary compiler designed by … Intel. For ARM-based chips, AnTuTu uses the open source GCC compiler, says The Register’s Neil McAllister, while AnTuTu for Intel chips uses ICC for versions 2.9.4, forward. The Register also revealed McGregor’s suspicions were spot-on. For an Exynos chip, the improvements weren’t nearly as generous it only experienced an improvement of 59 percent, and an increase of 53 percent in the RAM test. But between AnTuTu version 2.9.3 and 3.3, results for the Intel processor increased 122 percent with a 292 percent increase for the RAM score. McGregor explains that between updates of synthetic benchmarks, it’s common to see changes in test results, which makes sense. Powered by an Intel Atom chip, the Galaxy Tab 3 10.1 performs well in benchmarks, but no so much in daily use. EE Times’ Jim McGregor reveals some discrepancies in an AnTuTu update, Intel’s claims of a superior processor with better performance and less power draw, and the constantly varying results of different benchmark tests. With Android, one of the toughest challenges for benchmark developers is maintaining support for various chipsets, software versions, and new processor technologies.