top | item 26103533

Tesla Recalls Cars with EMMC Failures, Calls Part a ‘Wear Item’

184 points| nickthegreek | 5 years ago |hackaday.com

281 comments

order
[+] walrus01|5 years ago|reply
I see four failures here on the part of tesla:

1) choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years.

2) linux/operating system/software engineering implementation, for a certain volume of MB of writes per day, that would be known to wear out the flash write endurance over a short number of years. possibly a result of the software team not communicating properly with the hardware team. the software team very well may have been operating in a vacuum of assuming that they could do whatever they wanted within the CPU, I/O, RAM, and disk space limits of the architecture, and weren't even thinking at all of the consequences of their logging setup.

3) not making that a socketed/removable part.

4) asking the owners to pay for the replacement due to tesla's own screwup.

[+] ping_pong|5 years ago|reply
I have a Model 3 that is less than 2 years old. The build quality is terrible.

The side of my Model 3 trim on the inside just fell off for no goddamn reason. I don't even know how to fix it, it looks like the glue wasn't applied properly? This is on top of countless other problems that I've faced over the last 2 years.

It's impossible that Tesla is 3X the market cap of Toyota Why they didn't stop buying of Tesla the same way they stopped GME is criminal.

And they keep failing external quality tests and no one seems to care:

https://finance.yahoo.com/news/teslas-failed-every-edmunds-e...

[+] alfalfasprout|5 years ago|reply
Frankly people give Tesla a pass for things that would be unacceptable from any other automaker. The reality is that I don't see Tesla coming anywhere close to the build quality of established high end brands anytime soon.

You compare eg; a Mercedes or Porsche interior with a comparably priced Model S and it's no contest.

[+] smmnyc|5 years ago|reply
For what it's worth, my Model 3 from December 2018 with 35K miles on it has only had one issue (with squeaking upper control arms) and they replaced it under warranty, coordinated through the mobile app. I haven't had any other issues that would indicate build quality is terrible.
[+] bluesquared|5 years ago|reply
I just recently got a Chevy Bolt. 240 mile range for 1/2 of the cheapest Model 3 (used, paid in cash). It has the typical GM fit-and-finish, nowhere near what I'd demand out of a car with an MSRP at ~$43k with the trim level I have. GM took so many shortcuts with materials and design choices in what I assume was an attempt to not take a loss on each Bolt sold. Aside from a pretty nice big infotainment screen, it lacks the features, fit, and finish that I'd demand from a $20k car. I still love it, very peppy electric motor, drives well, plenty of space for a baby seat and passengers, decent cargo room.

It's still a bargain GM car at luxury prices, you'd be nuts to pay full MSRP for this vehicle. I got a great deal on a used one; tons of great deals were to be had on cars coming off of the first batch of 3-year leases until the battery recall put a stop-sale on everything at GM dealerships.

I know they don't stack up good against other brands comparing MSRP to MSRP, but I have to wonder how bad Tesla's "initial quality" really compares against all of the electric options out now.

[+] kryogen1c|5 years ago|reply
> It's impossible that Tesla is 3X the market cap of Toyota Why they didn't stop buying of Tesla the same way they stopped GME is criminal.

...what?

- GME trading wasn't halted. some high profile platforms like robinhood went literally insolvent and had to put a stop to some volatile stock trades and borrow billions in cash to stay legal. other well-capitalized platforms never restricted GME.

- related to above, GME wasn't halted on some platforms because of some market-cap vs actual value mismatch. this is not a thing.

- as for the market cap, well... prevailing wisdom is we're in some kind of bubble. something bad will happen eventually.

[+] gruez|5 years ago|reply
>It's impossible that Tesla is 3X the market cap of Toyota Why they didn't stop buying of Tesla the same way they stopped GME is criminal.

GME wasn't stopped because it was too overpriced. A few brokerages stopped accepting new GME positions because they couldn't afford the deposit requirements. That's it. If you went with another broker (eg. fidelity), you could buy all the GME you want.

[+] canada_dry|5 years ago|reply
> no one seems to care

Shareholders only care about increasing the volume of cars rolling off the assembly line.

Tesla was on my short list of next car purchases... but now that the old guard are getting fully into the game it's very unlikely (unless Tesla is able to pull a Jaguar like quality rebound or they actually deliver on "full" self driving).

[+] szczys|5 years ago|reply
I don't have a Tesla, but I had upholstery on the backs of the rear seats (the part that becomes the floor of the hatchback when you fold them down) all off on my Ford C-Max. It looks like an adhesive failure there.

Not making excuses for the big T, just saying that there's a lot places for things to work out poorly in automotive.

[+] wilhil|5 years ago|reply
Can't agree enough.

I'm not wealthy, but, got enough to be happy... I really wanted to treat myself to a Tesla and went to the local shop a while ago and I went out thinking - I'm not buying this.

The glovebox required such force to close. I put my arm on the armrest and the rear drinks holder opened The centre console was VERY loose.

and so many other problems.

Compare this to a family member's ~£13k Hyundai i10 where the gear selector was SOLID, I just didn't feel like the Tesla (Model S) was good value for money in the slightest.

I don't think anyone can touch them technically (yet), but, they have A LOT to learn about building cars.

[+] caycep|5 years ago|reply
I kind of wonder - how much of the valuation is actually in cars/car sales vs. the charger/battery infrastructure and expertise that they are building?
[+] draw_down|5 years ago|reply
I mean, they got your money... and people are still falling all over themselves to give Tesla their money... their stock price has gone extremely high... so I guess I wonder how much of a problem this really is for them. Doesn't seem to affect their brand, even if one thinks it should.
[+] nickthegreek|5 years ago|reply
Tesla's response is ridiculous.

"Now they are claiming that the eMMC chip, ball-grid soldered to the motherboard, inaccessible without disassembling the dash, and not specifically mentioned in the owner’s manual, should be considered a “wear item”, and thus should not be subject to such scrutiny."

[+] foepys|5 years ago|reply
I didn't expect other from a company that is immediately blaming the victim when their self-driving car once again drives somebody into a stationary object or lane divider instead of showing compassion first and launching an investigation later.

As a parent/child the first thing that comes from Tesla is a blog post and public statement on how your loved one allegedly was too stupid to use the car while Tesla goes into technical telemetry to grant themselves absolution.

[+] toomuchtodo|5 years ago|reply
Tesla's response is of course self serving. It should be expected from their legal department as a cost control measure. You cannot expect corporations to act benevolently or morally, hence the need for legislation (right to repair, warranty periods set by statute as is done in the EU) and regulation (NHTSA forced Tesla's hand to issue a recall, which they were strongly attempting to avoid).

Usual disclosures: We own Teslas and TSLA.

[+] Shivetya|5 years ago|reply
As an owner there are times when you just shake your head at some of their decisions. I like my TM3, I won the quality raffle and have no issues, but I do worry when Tesla takes a stance about a part like they did here.

Plus combined with the fact Musk was forced to acknowledge that Tesla did not offer anything for FSD on trade ins and "will look into it" makes me wonder just how much they expect customers to put up with.

Still it beats out my previous Ford and even my Infiniti in both reliability and oddly interior quality. Paint wise the Infiniti was far better and paint on my Ford was far worse. Both my Ford and Infiniti also had more than one real recall and a few visits to fix stupid things.

[+] qazxcvbnmlp|5 years ago|reply
Tesla: “People keep buying our cars, must not be an issue” Fan People: “Tesla is the future” Silent Majority: “Yeah, I’m just gonna keep waiting for a normal car maker to release an electric car”
[+] walrus01|5 years ago|reply
Nissan, a normal car maker, released a huge number of Leafs with improperly implemented battery cooling and battery charge/storage voltage management systems, which resulted in terrible cycle life and drops in range capacity in the 1st and 2nd generation Leafs.
[+] hinkley|5 years ago|reply
By the time people stop buying your stuff, things have gotten very bad.

Also the people who are most capable of fixing the problem? Probably gave up and work for a competitor.

[+] throwaway0a5e|5 years ago|reply
I present the following as a counter point

Toyota every year since 1980: "Don't worry guys the new truck frames won't rust out, also sorry last year's truck frames rusted out it was out supplier's fault"

People on the internet: "Warlords in Africa like the Hilux, the only truck they have a supply chain for, therefore every Toyota truck and SUV must be great"

[+] soganess|5 years ago|reply
The silly thing about all this is eMMC is a wear part.

Tesla made a really serious mistake and didn't take that into account and now the user is "holding it wrong". That needs to be in a place where the user or a technician can service it easily.

They were just trying to save a penny or a second of engineering time and now pretending they knew all along.

[+] FireBeyond|5 years ago|reply
It is. But its wear is almost exclusively for Tesla's benefit, not the customers. That's what's galling to me. Capturing extensive logging data so Tesla can do product improvement is NOT something I should be hammered hundreds of dollars for service on because it impacts the ability of me to use my vehicle.
[+] bdowling|5 years ago|reply
Tesla isn't blaming the user or pretending it knew all along, it's trying to avoid a mandatory recall. The letter here [0] is a response to a letter from the National Highway Transport Safety Administration, which has the power to force a mandatory recall of all affected Tesla vehicles. Tesla argues that a mandatory recall is unnecessary because the eMMC wearout is not a safety concern and is not the type of defect that should trigger a mandatory recall. Part of that argument is that eMMC is an expected wear item, that even without the over-the-air firmware updates already deployed the eMMC is expected to last 5-6 years, and that a failing eMMC does not cause dangerous conditions. Tesla has also promised to voluntarily recall and replace any affected hardware for free. Tesla argues that what it is doing makes a mandatory recall unnecessary.

[0] https://static.nhtsa.gov/odi/inv/2020/INRL-EA20003-82493P.pd...

[+] 34679|5 years ago|reply
What percentage of writes are coming from the consumer's normal use of the vehicle and what percentage comes from Tesla's telemetry collection?

It amazes me that people are OK with paying $60k+ only to be constantly tracked everywhere they go by a process that destroys the product they purchased. I don't even like being tracked in a free web browser.

[+] averynicepen|5 years ago|reply
Look, most people are probably smacking that "Accept all cookies" button on websites faster than the speed of light, so let's be real that the average consumer doesn't really understand how a Tesla really works.
[+] sschueller|5 years ago|reply
I don't know how this ever went into production. At least someone should have known that writing logs files to eMMC is a disaster. Anyone who has worked with any flash for a bit knows about this issue.

How many other items like that made it into the car and how many of those are a safety risk?

[+] elihu|5 years ago|reply
It's at least plausible that there was some good engineering analysis up front to decide whether the flash endurance was good enough for what they were doing and it was, and then some time later after the hardware designs were in production the software was changed by some other engineer who wasn't aware of the earlier analysis or the fundamental limitations of the hardware.

Abstraction is one of those things that makes software development tolerable; we can (usually) treat anything Turing complete as "just another computer". But once in awhile that isn't true, and you have to deal with low-level details.

In retrospect we can say this ought to have been a foreseeable problem, but people are fallible and make mistakes. If not this, then something else. What I find more worrying about Telsa is their level of secrecy around car repair information in general, and their occasional reluctance to own their mistakes and fix them for free.

[+] delecti|5 years ago|reply
Tesla strikes me solidly as a "software company" that happens to make cars. And having worked in software for software companies, and in software for car companies, I would definitely never get a car made by a software company.
[+] lmilcin|5 years ago|reply
This is certainly not a "wear item" but incompetence.

I had on two separate occasions designed an embedded POS application that worked on a dumb flash (dumb == no wear leveling).

There is nothing special about it. You just need to control where your writes go. You don't resize files (because this updates inodes and directories). You don't create files for stupid things like locks. If you have a database, you create a file and treat is as circular buffer or append only log. If you need OTA, you can split the memory in two and have boot loader boot alternately. And so on.

With wear leveling this is much, much simpler. You run the OS for some time and you capture all your writes at low level to see how much many blocks are being updated. Then you can predict how long the memory is going to last and you just size it accordingly.

[+] mrfusion|5 years ago|reply
The vehicle Should still be able To operate without this at some level.

I once hit a curb and the whole computer unit fell out of the socket in my 96 ford. There were flames and smoke coming out of the exhaust but it still started up and I was able to drive it to the shop.

[+] ouid|5 years ago|reply
kinda sounds like you shouln't have driven it to the shop though.
[+] therealbilly|5 years ago|reply
Oh man, Tesla is not covering themselves in glory on this issue. C'mon Elon, step up and do the right thing. Don't burn your goodwill.
[+] vkou|5 years ago|reply
Why would he worry about concrete things, like customer service, when he can add three billion dollars to his on-paper wealth by sending two tweets about bitcoin?

Responding to this sort of thing would mean acknowledging that TSLA's future is defined by something rooted in reality, as opposed to purely sentiment. As long as it doesn't actually have to stop and deal with these kinds of problems, its valuation is like that of dot-com startup - unbounded.

[+] naebother|5 years ago|reply
Just write to the blockchain, problem solved.
[+] etcet|5 years ago|reply
I've been burned by bad uses of eMMC twice this year.

First, Polycom Studio destroys itself: https://knowledgebase-iframe.polycom.com/kb/viewContent.do;j...

Secondly, all the Edgecore top o' rack switches we have are backed by eMMC and will all eventually go read-only. This is an especially fun surprise while working remotely.

Shit's junk and should never store anything that gets written to regularly (i.e. a log file).

[+] superkuh|5 years ago|reply
I've had a couple eMMC go bad on single board computers. I initially only noticed the off-by-1 read errors when the radio data that was being recorded by the computers (8kB/s for 2+ years) as comma separated values started sometimes getting commas turned into the next character up in ascii which broke my visualization python script.

It's definitely something you learn the first time you use eMMC (or any low end storage tech).

[+] walrus01|5 years ago|reply
what volume of writes per day, in MB, should you be expecting on a top of rack switch, though?

does it have some logging features you can't turn off? using as an example a cisco nexus 3064 I have sitting around here, all of the logging is done to an off-system destination over the network.

I can think of very few legitimate or sound network engineering practices that would require 1U, mostly layer-2 feature set switches to be writing anything more than a few dozen KB per day to their internal storage.

I can't say that I'm surprised to hear of something failing in a new and unique way in an edgecore switch, since the company (Accton) has also released a real dumpster fire of a series of outdoor radios in their Ignitenet brand, which have a spectacular failure rate.

[+] yawaworht1978|5 years ago|reply
Tesla will get away with terrible manufacturing practices and quality until the established or new car makers come up with an equally viable electric car and charging network solution, not just the compliance cars they produce at negative margins(almost makes you think it's a reverse marketing thing to make people buy petrol cars). Also, how can anyone think Tesla or Elon are good people with those self serving contracts and terms and conditions is unexplainable to me.
[+] throwaway0a5e|5 years ago|reply
I'm a big believer in "everything is a wear item on a long enough timeline". You won't catch me hurring and durring and getting my panties in a bunch because somsone's gonna wear out the ball joints on their lifted truck in 80k instead of 100k. But spending 50k to have the display on your Tesla or the frame on your Toyota crap out in under a decade is unreasonable.
[+] walrus01|5 years ago|reply
I thought Toyota replaced those frames at no cost, some people even got entirely new trucks.
[+] powerbroker|5 years ago|reply
For well over 12 months, the Model S Owners Manual (p. 40) and the Model X owners Manual recite, "When you finish speaking the command, tap the voice button again or simply wait."

Neither of my Teslas operate in the manner described in the manual since late 2019. If Tesla's manual writers can't communicate with the software developers -- is there anything in the manual you can trust? Can we always count on Tesla to arbitrarily loose functionality without anyone telling their manual authors?

Anyway, I gave up telling Tesla about this problem (in multiple iterations of the manuals), and reported it to the NHTSA https://www-odi.nhtsa.dot.gov/VehicleComplaint/

[+] localhost|5 years ago|reply
I wound up paying ~$1600 to replace the entire MCU1 in my 2015 S last year. I’m waiting for the refund of that amount in March and will likely just apply the proceeds to upgrade to MCU2. No way I want an ancient Tegra3 computer that isn’t made anymore in my car.
[+] Reason077|5 years ago|reply
> "This leads to widespread failures in the car, typically putting it into limp mode and disabling many features controlled via the touchscreen."

eMMC failure does not put the car in "limp mode". It does disable the touchscreen, making things like the air conditioning controls and backup camera unavailable (in older software versions).

"Limp mode" is when you have severe power restrictions due to very low battery charge, or some other major powertrain fault in an EV. Sometimes called "turtle mode" because a turtle icon will appear on the instrument cluster.

[+] samstave|5 years ago|reply
Elon's latest appearance on Joe Rogan was really interesting - and he specifically says that the manufacture and production is the hardest part - and that they are still learning.

They are building the CyberTruck factory in Austin TX next to the Airport.

He also says that the X is the best model - however, the S is his personal car and his favorite.

https://open.spotify.com/show/4rOoJ6Egrf8K2IrywzwOMk