Huawei caught cheating performance test for new phones

[axaxs]

This comes as no surprise. This is a company who has also been caught claiming their phone took a photo - but left in DSLR exif data, claiming a phone had UFS but shipped EMMC instead, boasting to work with dev community - only to lock them out months later, among other shady practices. They make nice phones at a good price, but is not a company to be trusted.

[sschueller]

I am not surprised as others have been cought doing the same. Anyone remember the DSLR in the reflection in a Nokia ad that was supposedly shot a Nokia? Have you seen the rig Apple uses? doesn't look much like a phone anymore but technically is still an iPhone.

Sadly the P20 pro has an incredible camera and would stand on its own without forgery.

All I really care about is that they unlock their bootloader. The marketing crap is always half truths anyway. Retina, you mean 4k?

[dspillett]

> we have decided to delist the affected models and remove them from our performance rankings

I would be inclined to be more aggressive than that. Leave them listed, but add support in the site or other publication system to display "deliberately broke the rules of the test" as a negative, perhaps displaying in a negative looking colour and in any graphs show as a if they scored, say, 20% worse then the worst other score.

If presenting some arithmetically derived overall rating of benchmarks and other properties, balloon this further. Then if the rest of the phone is demonstrably brilliant they still have a hope of not looking terrible, but the cheating hurts the result otherwise.

Unfortunately in these litigious times such an inclination would probably cause me some financial trouble, so it is probably for the best that I don't run a device comparison site! This may be why simply delisting was the chosen way of handling the situation here.

[foobaw]

I used to work at an OEM that did this. When the VW Emissions Scandal was publicized, we removed almost every piece of code that did things like these from all our phones.

Not sure if it was ever reverted eventually but performance test might not matter to HN users but it is something that typical consumers do look at when making purchasing decisions, based on our research.

[maxerickson]

Your second paragraph is completely ethically bankrupt.

That a lie increases sales is at the top of the list of bad reasons to lie to customers (and they are all bad reasons).

[EZ-E]

I'm gonna get down voted but that's why I buy Apple phones. Comparing processors, ram, test scores means nothing compared to the actual experience

[beerlord]

I switched to Apple after my Nexus 5 stopped getting updates about a year after I bought it.

Bought an iPhone 6S second hand for $320. Still getting updates now, and looks like I will for the foreseeable future.

The core feature iPhones are missing is USB Type C- I won't upgrade to a new iPhone if I cant use the same headphones on my laptop and phone (either 3.5mm or TypeC)

[drampelt]

And that's why I buy Google phones. The processor being slightly slower doesn't matter to me if I prefer the overall experience

[jackconnor]

Yea I kind of agree. I think most people, myself included, just don't really recognize minor performance differences, only relatively large ones. And I've always thought that these things improve so fast year-to-year, that it's kind of pointless to try and keep up. So, i shoot for a phone I like that performs pretty well, and just leave it at that (hello iPhone SE).

[ali_af]

  People who are really serious about software should make their own hardware

I never get tired of this quote, and I will always point to it when people compare iOS/Android or macOS/Windows.

[Someone1234]

> Comparing processors, ram, test scores means nothing compared to the actual experience

But Apple brags endlessly about their A11 "Bionic" chip and similar[0].

Arguing that Apple values experience over specs might have been a valid argument back with Steven Jobs, but Apple's presentations spend significant amounts of time talking about hardware specs these days.

I don't specifically mind that Apple does, but people need to stop pretending like Apple hasn't changed as a company with the shift in leadership.

[0] https://youtu.be/M4Pc1eQtWmQ?t=11m43s

[yason]

This is just what is expected: the vendors do this because they need to shine in these meaningless benchmarks because that's merely the name of the game. The tech magazines are in between: they publish results in reviews and phone comparisons but it seems that not many people really buy phones based on benchmark scores. They might give some overall idea of whether the phone is middle-end or high-end, but differences within one category tend to be small enough to not matter for the general buyer.

To fix this, there should be different runs for these benchmarks. A max-clocks run with the phone set in an actively cooled cabin to see what the hardware can theoretically do at its very best, and another run in non-actively cooled room temperature with the test running for about an hour to rule out any benefits from temporary boosts or ignoring thermal limits to get an idea of continuously supported performance.

Even those results wouldn't really tell much to the end-user. I always suggest my friends and relatives to buy a phone with excessively large memory and storage space because lack of memory and storage really is what turns phones slow after a few rounds of application and system updates.

The performance edge on mobile is thinning out pretty much what happened with PCs. Ditto for memory and storage. Early on, vendors competed on who has the highest MHz cpus but somewhere between 1-2GHz the performance got high enough in nearly any case. 95% of people could buy about just any PC or laptop and it would be "fast enough". There's gigabytes of memory in even the sloppiest laptop these days, and enough SSD to make things fly. The same will happen with phones which makes it impossible to buy a phone that is too slow. At that point components with lower performance and less capacity will become more expensive due to the lack of volumes, that no vendor will bother any longer.

[komaromy]

I don't see why an earnest benchmarking firm would let manufacturers know details like the name of the testing app.

[halflings]

Transparency. Everybody can run the benchmark on their own device [1] and check the results, that way they can't deny the results saying that it's malicious, a conflict of interest or a configuration error.

[1] https://play.google.com/store/apps/details?id=com.glbenchmar...

[shard]

Have you ever thought about what the business model is for benchmark companies?

It probably won't come as a surprise to many of you, but there's a major conflict of interest when it comes to benchmark companies. They have "benchmark development programs" which companies pay tens or hundreds of thousands of dollars to join, and in turn the member companies get early access to beta versions of the benchmarks, and get to propose optimizations to the benchmarks.

Now the benchmark companies insist that they make impartial decisions when it comes to what to include or improve in the benchmarks, but you have to wonder just how much influence these member companies have over the benchmark companies, especially when it comes to new features that are in one but not all devices, or features that are not often used by developers but have different performance on different devices, what the decision process is to include tests for those features.

[duxup]

Fans who is read about the benchmarks want to know.

[thisisit]

My first reaction was reaction isn't this a dupe. But then the earlier one was about Huwaei's camera cheat:

https://news.ycombinator.com/item?id=17805027

[wenbert]

Can these be grounds for refund?

[craftyguy]

Class action lawsuit, you get $10 off your next Huawei purchase.

[Joakal]

In Australia, can return for a full refund that it's not working as advertised.

[k_sze]

How ironic that Huawei uses the word "Honor" in some of its models.

[baybal2]

Well, they sure missed the dis* prefix

[jaclaz]

>however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.

>In other words, the phones aren’t so smart after all.

If I may, also the fact that benchmarks are normally/usually "recognizable" through some "label" doesn't seem "smart" to me.

[da_murvel]

If something is cheap in terms of the price you pay for it, you're going to pay by something else. Be it poor quality components, information collection or unethical manufacturing processes. Yes, of course I am generalizing. There are good cheap(er) products and bad expensive ones.

[Bayart]

AFAIK Chinese companies make money (in China) with the software they ship on their phones, and the associated Internet services. They all have a slew of what we'd call bloatware, but apparently Chinese people expect it and enjoy it for it being "free stuff".

I don't really know what's their roadmap for Western markets considering nobody wants to let off their Google world. Even though the cynics among us will placate it as the Party bankrolling it for spying.

[chrisper]

That's a shame. I got a P20 Pro and I really like it (even the software).

But I don't think I'll get another Huawei next time, because the company itself is really weird.

[yjftsjthsd-h]

Using "weird" as a euphemism for unethical, or some other problem?

[on_and_off]

Huawei caught cheating at a meaningless 'test' nobody should care about.

I spoke a fair bit about performances with a Googler in charger of it. Apparently he is appalled that OEMs have been optimizing their phones in order to score high on these benchmarks.

They don't reflect real use at all and are not what should be optimized against.

In the end, he documented the work he did on the various terminals he worked on so OEMs could do what they should have been doing for years..

[ezoe]

It reminds me of the old GPU vendors when we were still using AGP bus(or pre-AGP era I forgot). Good evil days.

[taneq]

Are those days gone? Benchmark-specific hacks in graphics drivers, I mean, not AGP. :P

Also all the outrage when people discovered that a Radeon 9600 was just a Radeon 9800 running in crippled mode, and that you could bridge the tracks and re-enable it in 9800 mode. (Or something similar, anyway, that was a long time ago... :P )

[lysp]

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.

Source code for their AI engine:

if (app == benchmark) { increasePerformance(); }

[vvanders]

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this.

They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps.

We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.

[remlov]

Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.

[userbinator]

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

[samsonradu]

Looks a lot like the Volkswagen’s emission software /s

[lamarpye]

This is a change from their normal business practices, usually they would steal performance results and publish them as their own.