The AnTuTu Benchmark, a benchmarking tool for Android smartphones and tablets, has been revised following some discrepancy over whether the latest Intel Atom processor outperformed ARM-based chips from several vendors in key aspects of the benchmark. Under the revised benchmark, overall scores for the Atom Z2580 dropped by about 20 percent.
As I indicated in a post on EE Times earlier this week, there appeared to be some discrepancies with the AnTuTu benchmark in relation to older versions of the benchmark and other benchmarks. (See: Has Intel Really Beaten ARM?) Technical consulting firm BDTI pointed out that the compiled code for the Intel processor was not executing all instructions that were intended for the RAM test. This artificially improved the results for the Lenovo K900 smartphone and the Intel Atom processors. The problem appears to arise from the ICC compiler introduced around version 2.9.4 of the AnTuTu benchmark and used just for the Intel processors. Code for all other processors uses a different compiler, called the GCC.
To rectify the situation, AnTuTu issued revision 3.2.2 to the benchmark Wednesday evening. The revision still uses the ICC compiler, but the resulting scores are drastically different for the Intel processor. The AnTuTu CPU and overall scores dropped by approximately 20 percent, while the AnTuTu RAM score plummeted by approximately 50 percent, as shown in the figure below. All other scores for the Intel processor remain relatively the same. Similarly, scores for the Samsung S4 Oct and Qualcomm Snapdragon 600, both used in the Samsung Galaxy S4, remain relatively unchanged. (To be continued)
This'll hurt hard to AnTuTu's creditbility. High level optimized compiler will remove the dummy codes. If the benchmark test with whole lots of redundant code , people will question its validity. Does it measure the performance of activities that ever happen in the real applications?
This also wake up Intel lovers to face reality - INTC still had some room to improve in mobile processor.
yap yap yap all day long pumping and bashing on a nothing message board.
Facts about what's going down:
before nvda has gotten a subtainable toehold on the mobile market, you have a 800 lb gorilla coming in who WILL siphon off market share and $s. Most importantly, you have one more guy in line in the only high runner in tablets that is available to merchant app proc - Nexus 7.
Tegra 1 lost money. Tegra 2 lost money. Tegra 3 lost money. No doubt Tegra 4 will lose money too.
When will Tegras break even? Real investors and real shareholders are asking. Brown nosers forever hold their peace.
It's hard to resist a foot in the mouth type of comment . . 'yap yap yap all day long'. The ones who are spending the majority of their time clogging this board are the bashers with nothing better to do. The majority of the long-term investors, including ibexx and others that post a few times a day, don't post nearly as much as the bashers. So what makes this a 'nothing message board'? It's the losers that are simply talking to each other with the same tired old distortions and lies. Nvidia and the stockholders are on cruise control as the ramp up just begins. There's no desire to waste time responding to those that clog this board (except once in a while as I'm doing here) because we as stockholders of Nvidia are not worried at all.
As a result of the revised scores, the AnTuTu benchmark is no longer an outlier from the other benchmarks and paints a competitive picture similar to the other benchmarks, with the Samsung Exynos 5 Octa processor outperforming the Intel Atom Z2580 processor (see the figure below). But, are these numbers valid? It is unclear what changes were made to revision 3.2.2 the benchmark.
However, AnTuTu did indicate that new testing standards would be issued in August, presumably with the next major revision of the benchmark. At that point, it will be interesting to see the new "standards" and the resulting changes in the benchmark scores.
Suddenly, the battle between smartphone processors looks much different than all the hype that was floating around because of the AnTuTu benchmark. It's clear that the ARM processors still hold a significant advantage over the Intel processors, whether you include the new AnTuTu scores or just eliminate them completely from the evaluation.
The current and upcoming revisions to the AnTuTu benchmark will also drastically alter the scores of Intel's upcoming Bay Trail Atom processor. As with the Atom Z2580, many in the press prematurely proclaimed it the victor over the next-generation ARM processors. Now that also appears to be highly questionable.
So, will all the sensationalistic bloggers retract their stories about Intel beating ARM? Likely not, but this definitely questions their credibility just as it does benchmarks. The moral of the story is that you have to question all benchmark data and use many data points from different sources before drawing a conclusion. It's clear that Intel still has an uphill climb to catch the ARM camp, much less surpass it. It is also clear that we need better benchmarks for mobile devices that test for platform efficiency and usage models.
— Jim McGregor is founder and principal analyst at Tirias Research.