> In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices.
and then it goes into canary testing, "pre flight tests" and rollback
To be fair, Volvo did the same thing recently, so it's not just weird American startups that do this - Volvo never released the numbers on how many cars were affected, but I'm a member of several facebook groups for Volvo owners and it was just like an onslaught, people were posting daily warning others about not applying the latest software patch or it had a good chance of bricking your car. Absolutely no idea how that got released into the wild.
I hadn't heard about the Volvo one! I had a 2022 Volvo C40 before I got my Rivian R1T.
When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian.
Was Volvo able to fix it with another OTA or did people have to go in for service?
> With the botched 2023.42 update Rivian explained that they pushed the wrong build with the wrong certificates to customer vehicles. This made me immediately think they probably don’t have a canary fleet of vehicles that they roll out to first.
Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.
Repeating what you said in my own words to ensure I understand it.
I think you're suggesting they might have a canary fleet, but there wasn't anything/enough that preventing a mistake from bypassing the carry fleet before going to production?
Software is such a powerful tool that I understand motor vehicles having as much code in them as they do. What I don't want is for that software to be shoddy or for it to spy on me. I also want complete control over whether or when it changes, and I want to understand the nature of and reason for the updates, just as I do for my Linux laptop on which I use apt-listchanges before accepting upgrades.
* New upstream version 9.16.48
- CVE-2023-4408: Parsing large DNS messages may cause excessive CPU
load
- CVE-2023-5517: Querying RFC 1918 reverse zones may cause an assertion
failure when "nxdomain-redirect" is enabled
...
glibc (2.31-13+deb11u8) bullseye; urgency=medium
* debian/patches/any/local-qsort-memory-corruption.patch: Fix a memory
corruption in qsort() when using nontransitive comparison functions.
* Fix CVE-2021-3574: memory leak was found in TIFF coder
* Fix CVE-2021-4219: a special crafted file could lead to a DOS.
* Fix CVE-2021-20241 / CVE-2021-20243: divide by zero in
some coders (Closes: #1013282)
And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too?
For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.
I've also often thought about what an open source car software stack might look like, but with different motivations. I'd love to be able to see more diagnostics about what the car is actually doing and to add 3rd party extensions.
For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you.
That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.
Imo the current continuous update while letting customers beta test new updates starts to fall apart as the cost of the hardware increases.
Bricking am expensive smart phone is infuriating, but bricking an expensive household appliance or even more expensive automobile is a non starter.
The signed image on USB seemed to be the norm from maybe 2010-2020 but it seems cellular connectivity has gotten too cheap and telemetry too valuable...
Software in cars and OTA is the stupidest thing in recent years. Like the damn laptop riddled with mediocre software wasn't already frustrating enough, let's fuck up cars too.
To be fair cars as currently designed are a pretty stupid idea to begin with. Let’s just waste energy carrying around 5000lbs of car at ridiculous speeds to move around a 200lb person.
froh|2 years ago
and then it goes into canary testing, "pre flight tests" and rollback
MattGrommes|2 years ago
gambiting|2 years ago
kelp|2 years ago
When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian.
Was Volvo able to fix it with another OTA or did people have to go in for service?
aranchelk|2 years ago
Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.
kelp|2 years ago
I think you're suggesting they might have a canary fleet, but there wasn't anything/enough that preventing a mistake from bypassing the carry fleet before going to production?
Could be!
steelframe|2 years ago
For example:
apt-listchanges: Changelogs
---------------------------
bind9 (1:9.16.48-1) bullseye-security; urgency=high
...glibc (2.31-13+deb11u8) bullseye; urgency=medium
...imagemagick (8:6.9.11.60+dfsg-1.3+deb11u2) bullseye; urgency=medium
And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too?For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.
kelp|2 years ago
For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you.
That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.
nijave|2 years ago
Bricking am expensive smart phone is infuriating, but bricking an expensive household appliance or even more expensive automobile is a non starter.
The signed image on USB seemed to be the norm from maybe 2010-2020 but it seems cellular connectivity has gotten too cheap and telemetry too valuable...
yazzku|2 years ago
dopylitty|2 years ago
csours|2 years ago
giantg2|2 years ago
[deleted]