Here's the low-level controller for Comma AI's self-driving software. Lateral and longitudinal control, as well as actuator command are contained within.
Their low level control is done in Python, on top of Android, on a smartphone. No further comment, your honor.
No safety relevant code is written in Python. All the safety relevant code runs real-time on a STM32 micro (inside the Panda), it's written in C and it's placed at the interface between the car and the Eon. This code ensures the satisfaction of the 2 main safety principles that a Level 2 driver assistance system must have: 1- the driver needs to be able to easily disengage the system at any time; 2- the vehicle must not alter its trajectory too quickly for the driver to safely react. See https://github.com/commaai/openpilot/blob/devel/SAFETY.md
Among the processes that runs on the Eon, you can find algorithms for perception, planning and controls. Most of it is actually autogenerated code in C++ (see model predictive controls). Python code is used mainly as a wrapper and for non-computational expensive parts.
To use functional safety terminology, the Eon functionality is considered QM (Quality Management). This means that any failure in delivering the desired output at the right time is perceived as bad quality and has no safety implications. So, how often those algorithms deliver the wrong output because some parts are written in Python? How often because RT isn’t enforced? Negligible. Pretty much all the mistakes of a level 2 driver assistance system are due to the quality of the algorithms, the models, the policies etc… There is a long way to go before changing the coding language will be the lowest hanging fruit to improve the system. Until then, using the simplest and most agile coding language (given performance constraints) is probably the best way to maximize quality.
As someone who has had a bit to do with safety critical software in agricultural machinery, and I am sure to any one in any other regulated industry this sounds crazy. The regulation in many industries for software that can kill is onerous, python on Android for any safety related code in other industries would be the punch line of a joke.
Ummm... If you look in the actual lib/ directory where the work happens, you will see a bunch of high-performance C written in and generated by Acado, which is a very efficient optimization framework. It does look like there is a bunch of python logic for managing that controller, but a significant amount of work seems to have gone into performance optimization.
I am curious how they prevent the Android GC from causing problematically long pauses, though.
As much as I love Python, you need realtime processing for this kind of things. You can't have a slow language with gc pauses and no way to guaranty execution time for a given operation.
In the 1980's the backup flight control software for landing the space shuttle ran on a HP-41 calculator, though they never had to use it AFAIK, with the multiple redundant onboard computers.
It sounds to me like you don't understand how little performance you actually need for this kind of stuff. People have been writing effective autopilot software for decades using hardware that wasn't even particularly fast at the time and may as well be a pile of sand today. What exactly do you think that that code needs to do? I promise that Python on a new smartphone can do way more now than we could do 20 years ago on a Pentium in C or C++. You're lucky if any of your sensors emit at even 100Hz.
When George demonstrated his first model I showed a friend who designs safety critical systems. To paraphrase he said, that of course you can hack something together in a weekend that does this.
However a analysing a million edge cases, making sure all failures fail in a safe way takes a lot of time and knowledge.
It made me think that this is a great demonstration of the difference between hobbyist hacking and professional engineering.
> that of course you can hack something together in a weekend that does this. However a analysing a million edge cases, making sure all failures fail in a safe way takes a lot of time and knowledge.
Thats true for a lot of things. Most developers can whip up a Slack clone in a weekend, create an uber clone in a weekend or create a Netflix clone in a weekend. But dealing with all the edge cases, the details, making it scalable, etc. that's were the pain is.
> California’s DMV requires all companies testing self-driving cars to report how many times safety operators were forced to take control of its vehicles.
You make what you measure. This encourages companies to have their safety operators take over less often: if you tell the operators to err on the side of safety and take over more often, your car will look less safe.
I don't have a solution here, just pointing out how this actually can benefit companies that care less about safety.
Sorry if this is a stupid question but what does California do with these numbers? If Wayne has 10 engagements per mile driven with 10M miles driven and Tesla has 2 per mile with 500k miles driven and Uber has 1 per mile with 1M miles driven (all made up numbers), will there be some sort of penalty for below average scorers? Is it possible to categorically say who is the best? How does this work?
If you are interested in learning more about George, you should listen to him talking at Mapbox's Locate conference with the Mapbox CEO, Eric Gundersen [1]. Its another example of George being George. I just happened to be called out during the talk, but overall rather enjoyed it.
> If the driver looks down or away from the road for two seconds, a visual warning will appear on the Eon’s screen. After four seconds, the driver will hear an audible alert. And after six seconds, the system will disengage and begin to slow down.
The biggest use of some kind of driver assist system is something that can keep me going straight and at a constant speed on a relatively open road while I twist open a bottle cap and take a sip of a drink. Making me have a death stare out of the windshield to avoid disengagement is about as useful as standard cruise control. Except standard cruise control isn't going to be prone to making a mistake about whether or not I'm paying attention, and then stopping my car because it isn't satisfied.
Right. Keeping in lane for 2 secs when you spill your coffee or the kid screams or you do something else that makes you lose your concentration for a short while is great.
I don’t need to sleep or work for an hour during a commute. Not now not tomorrow. I can drive the damn thing or at least watch it drive itself. And like mr Hotz I believe that’s what it will be like for decades to come. My prediction is flying cars before completely self driving ones.
And not some novelty one person quadcopter. I mean the proper Blade Runner ones.
What would the point of current self-driving car systems be? If you can't take your attention off the road anyway, it's basically a glorified, expensive cruise control.
What’s wrong with good cruise control? Any modern car is already loaded with expensive cruise control, often with distance sensing, some times with lane assist etc.
There is a huge market for this and a few players (such as Bosch Mobility) own the market. I don’t see why it’s a bad idea to try to enter this market as a startup. You don’t have to revolutionize or disrupt a market to have a viable product.
I think (like Hotz) that very good assist tech is the future, and dodgy self driving tech is a temporary fad.
Complete autonomy on geofenced highways may well be good enough soon that we can sleep or work for 90% of the commute because the autonomous system can give us 20 sec to take over when confused, rather than 1. And that can probably be solved with very cheap hardware (1-2 cameras, a front radar) compared to the huge expensive lidar and camera arrays you need for driving in complex scenarios.
So if someone can make a system that offers fantastic assist and limited but safe autonomy for 10% of the roads but 90% of the driving time - that’s not a bad thing.
Ever drive 8-12 hour days for 2 days straight? Used to do it 4-5, but sometimes up to 10, times per year. I'd pay many thousands for lane-keeping on rural interstates in a heartbeat.
That's the hurdle that will or will not kill this wave of SDV.
You can't give bits and piece of automation for a vehicle. Mainstream tech is all about making it fool proof. Not giving warnings and calling about weighted death stats.
> What would the point of current self-driving car systems be?
You’re teaching the AI in Tesla’s case. Theoretically, your drive is safer with driver augmentation safety systems in other manufacturers’ vehicles, even if they’re not contributing data back for algo refinement.
The point would be to save lives. With most accidents coming from drivers being drunk/distracted, a vastly improved cruise control could prevent many deaths.
I think that this being marketed as anything other than the next generation of cruise control is a lie. Don’t get me wrong, for that it’s absolutely great. I would pay for this kit (if my car was new enough to actually have software for it) just for the ability to be able to eat a sandwich on the way to work or cruise down the boring parts of E6 or I-10 in a more relaxed, reclined position.
I think Hotz is wrong about the predictions of self driving cars, but he doesn’t have to be right to make this business succeed in the medium term because he is filling an entirely different market niche here.
Truly self driving car is incredibly hard. He’s right: waymo, cruise, Uber, Tesla are over-selling it. People have died in Teslas because drivers were over confident in their technology. Uber has run over people because their software couldn’t see humans in dark.
A reliable and smarter cruise control can save so many lives. I think we’re trying to jump the gun sometimes.
The number of highway deaths is insane. If software just augmented humans better and saved those people/animals we’re coming out ahead as a civilization.
Sour grapes? I recall when push button elevators were installed. The union insisted an operator was still needed - who asked you 'what floor?' and then pushed that button.
Without operators, western, and indeed all global civilization would come to an end...
I personally do not want a smartphone with a 3D printed case and heatsink controlling my car. Do you hear that? It's the sound of a critical flaw in your product, no matter what you do technically you are never going to have a mass appeal product that becomes more than the nerds who hang out in your Slack channel.
But it's fine for a lifestyle business I suppose. You can market to them and sell them over-priced odb2 fit-bit-for-your-car dongles. Until Honda or whoever comes out with a "good enough" consumer grade "auto-pilot" (whatever that means these days because you still have to watch the car like a teenage driver), you'll loose a huge swatch of users right there who are only spending time on it to be on the cutting edge of tech and to look cool for their friends. Once it becomes a blender, those users will leave and go on to the next thing that grabs their attention in a banner ad. Or else people will get bored, loose interest after the first few joy rides with their friends. I'd imagine retention would be difficult in a business like that which takes a lot of sustained effort and engagement from the users.
The only possible path forward in this market is to partner with an OEM (car manufacture vroom vroom), have access to users, or make some kind of must-have tech that you can then con one of the real big money backed self-driving companies into buying. I've seen a bunch of web apps popping up lately that offer to annotate your training data, those seem like a lot better pump and dump tech companies than this dashcam that plays Pandora and runs some cool python PID controller.
Driver assist is a good way to get people comfortable with automated driving. The only way autonomous vehicles will ever really show any benefit is if communities, municipalities and eventually, governments start rolling them out en masse and incentivising their purchase; SDVs driving together is the only way to finally eliminate traffic related deaths and make travel smooth and fast. As long as you have computers trying to predict and/or respond to erratic human behavior, you're fighting a losing battle; it doesn't matter how many times you rehearse it you'll hit a certain plateau where the correct decisions simply can't be determined.
It seems likely that autonomy will not be first adopted by the US but instead in the 'wild west' of BRICs - Brazil, Russia, India or China.
Also maybe in a country small, rich, authoritarian, and organized enough to build infrastructure around it and limit liability like Singapore or Luxembourg.
And oh look, a cheap and open source solution that gets you started in CommaAI. I think they (or a fork of them) are quite good candidates for the first system to reach one trillion miles driven. There will no doubt be carnage. But the benefits of self driving are going to be first on display outside of US markets.
This guy is all talk and represents one problem with Silicon Valley. There are a lot of people now who can talk the talk and get the attention needed to get money but with no ability to deliver. Stop giving him attention until he actually shows something that matches even a little bit of what he is promising.
People who know what they're doing technically are deeply aware of the issues with their technology. Anyone in this area knows how hard it is to make these vehicles work. He is finding out the reality of what he promised two years late and is spinning it to get more attention.
You are aware we are giving away open source software that will control your car, right? It's online today! And it's on par with the best systems money can buy, Tesla Autopilot and GM Super Cruise.
So now we'll have weekend "hackers" downloading this open source software, tweaking it, and hurling their cars down the road at high speed. Great.
The system also seems to have a single front facing camera. How will this detect for example a cyclist riding along the car and prevent the car from turning into him?
You seem to have pretty strong opinions without even actually taking the time to look into comma.ai and how it works.
All it takes is 5 minutes to know that it actually uses the built-in safety systems in cars like ADAS, LKAS, ACC. Hence, they only support only specific make/models. It's not just the "single front facing camera".
Brilliant! Pure AI is a dead end street. Turing machines cannot increase their minimal Kolmogorov sufficient statistic, so will always be unable to learn more information than is put into them originally. Humans are the only known information creators, and so only hybrid human/AI systems can work in the real world.
[+] [-] asteli|7 years ago|reply
Their low level control is done in Python, on top of Android, on a smartphone. No further comment, your honor.
https://github.com/commaai/openpilot/tree/devel/selfdrive/co...
[+] [-] lordricky|7 years ago|reply
Among the processes that runs on the Eon, you can find algorithms for perception, planning and controls. Most of it is actually autogenerated code in C++ (see model predictive controls). Python code is used mainly as a wrapper and for non-computational expensive parts. To use functional safety terminology, the Eon functionality is considered QM (Quality Management). This means that any failure in delivering the desired output at the right time is perceived as bad quality and has no safety implications. So, how often those algorithms deliver the wrong output because some parts are written in Python? How often because RT isn’t enforced? Negligible. Pretty much all the mistakes of a level 2 driver assistance system are due to the quality of the algorithms, the models, the policies etc… There is a long way to go before changing the coding language will be the lowest hanging fruit to improve the system. Until then, using the simplest and most agile coding language (given performance constraints) is probably the best way to maximize quality.
[+] [-] nodnyl|7 years ago|reply
[+] [-] HALtheWise|7 years ago|reply
[+] [-] sametmax|7 years ago|reply
[+] [-] stevenwoo|7 years ago|reply
[+] [-] sebastianavina|7 years ago|reply
[+] [-] BugsJustFindMe|7 years ago|reply
[+] [-] jimnotgym|7 years ago|reply
However a analysing a million edge cases, making sure all failures fail in a safe way takes a lot of time and knowledge.
It made me think that this is a great demonstration of the difference between hobbyist hacking and professional engineering.
[+] [-] jsiepkes|7 years ago|reply
Thats true for a lot of things. Most developers can whip up a Slack clone in a weekend, create an uber clone in a weekend or create a Netflix clone in a weekend. But dealing with all the edge cases, the details, making it scalable, etc. that's were the pain is.
[+] [-] zck|7 years ago|reply
You make what you measure. This encourages companies to have their safety operators take over less often: if you tell the operators to err on the side of safety and take over more often, your car will look less safe.
I don't have a solution here, just pointing out how this actually can benefit companies that care less about safety.
[+] [-] mcny|7 years ago|reply
[+] [-] masonic|7 years ago|reply
[+] [-] flippmoke|7 years ago|reply
[1] https://www.youtube.com/watch?v=N9WxlR1ZTZc
[+] [-] SOLAR_FIELDS|7 years ago|reply
[+] [-] aresant|7 years ago|reply
The original rap diss of Sony who was suing him, actually badass :)
[+] [-] Shank|7 years ago|reply
The biggest use of some kind of driver assist system is something that can keep me going straight and at a constant speed on a relatively open road while I twist open a bottle cap and take a sip of a drink. Making me have a death stare out of the windshield to avoid disengagement is about as useful as standard cruise control. Except standard cruise control isn't going to be prone to making a mistake about whether or not I'm paying attention, and then stopping my car because it isn't satisfied.
[+] [-] alkonaut|7 years ago|reply
I don’t need to sleep or work for an hour during a commute. Not now not tomorrow. I can drive the damn thing or at least watch it drive itself. And like mr Hotz I believe that’s what it will be like for decades to come. My prediction is flying cars before completely self driving ones. And not some novelty one person quadcopter. I mean the proper Blade Runner ones.
[+] [-] ianlevesque|7 years ago|reply
[+] [-] raslah|7 years ago|reply
[+] [-] juliangoldsmith|7 years ago|reply
[+] [-] alkonaut|7 years ago|reply
I think (like Hotz) that very good assist tech is the future, and dodgy self driving tech is a temporary fad.
Complete autonomy on geofenced highways may well be good enough soon that we can sleep or work for 90% of the commute because the autonomous system can give us 20 sec to take over when confused, rather than 1. And that can probably be solved with very cheap hardware (1-2 cameras, a front radar) compared to the huge expensive lidar and camera arrays you need for driving in complex scenarios.
So if someone can make a system that offers fantastic assist and limited but safe autonomy for 10% of the roads but 90% of the driving time - that’s not a bad thing.
[+] [-] throwawayjava|7 years ago|reply
[+] [-] oldgradstudent|7 years ago|reply
Real self driving cars, on the other hand, only exist in inspirational videos.
[+] [-] agumonkey|7 years ago|reply
You can't give bits and piece of automation for a vehicle. Mainstream tech is all about making it fool proof. Not giving warnings and calling about weighted death stats.
[+] [-] toomuchtodo|7 years ago|reply
You’re teaching the AI in Tesla’s case. Theoretically, your drive is safer with driver augmentation safety systems in other manufacturers’ vehicles, even if they’re not contributing data back for algo refinement.
[+] [-] agildehaus|7 years ago|reply
No level 1-3 system can get you from A to B unassisted.
[+] [-] leesec|7 years ago|reply
[+] [-] airstrike|7 years ago|reply
Baby steps, man... baby steps
[+] [-] sebastianavina|7 years ago|reply
[+] [-] SOLAR_FIELDS|7 years ago|reply
I think Hotz is wrong about the predictions of self driving cars, but he doesn’t have to be right to make this business succeed in the medium term because he is filling an entirely different market niche here.
[+] [-] nojvek|7 years ago|reply
A reliable and smarter cruise control can save so many lives. I think we’re trying to jump the gun sometimes.
The number of highway deaths is insane. If software just augmented humans better and saved those people/animals we’re coming out ahead as a civilization.
[+] [-] brokenmachine|7 years ago|reply
[+] [-] amelius|7 years ago|reply
Curious, what kind of regulations does your country have in place?
[+] [-] aurizon|7 years ago|reply
[+] [-] iamleppert|7 years ago|reply
But it's fine for a lifestyle business I suppose. You can market to them and sell them over-priced odb2 fit-bit-for-your-car dongles. Until Honda or whoever comes out with a "good enough" consumer grade "auto-pilot" (whatever that means these days because you still have to watch the car like a teenage driver), you'll loose a huge swatch of users right there who are only spending time on it to be on the cutting edge of tech and to look cool for their friends. Once it becomes a blender, those users will leave and go on to the next thing that grabs their attention in a banner ad. Or else people will get bored, loose interest after the first few joy rides with their friends. I'd imagine retention would be difficult in a business like that which takes a lot of sustained effort and engagement from the users.
The only possible path forward in this market is to partner with an OEM (car manufacture vroom vroom), have access to users, or make some kind of must-have tech that you can then con one of the real big money backed self-driving companies into buying. I've seen a bunch of web apps popping up lately that offer to annotate your training data, those seem like a lot better pump and dump tech companies than this dashcam that plays Pandora and runs some cool python PID controller.
[+] [-] lowtolerance|7 years ago|reply
[+] [-] raslah|7 years ago|reply
[+] [-] jacksmith21006|7 years ago|reply
[deleted]
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] stillsut|7 years ago|reply
Also maybe in a country small, rich, authoritarian, and organized enough to build infrastructure around it and limit liability like Singapore or Luxembourg.
And oh look, a cheap and open source solution that gets you started in CommaAI. I think they (or a fork of them) are quite good candidates for the first system to reach one trillion miles driven. There will no doubt be carnage. But the benefits of self driving are going to be first on display outside of US markets.
[+] [-] anfractuosity|7 years ago|reply
[+] [-] maym86|7 years ago|reply
People who know what they're doing technically are deeply aware of the issues with their technology. Anyone in this area knows how hard it is to make these vehicles work. He is finding out the reality of what he promised two years late and is spinning it to get more attention.
[+] [-] commaai|7 years ago|reply
https://github.com/commaai/openpilot
No ability to deliver?
[+] [-] 21|7 years ago|reply
The system also seems to have a single front facing camera. How will this detect for example a cyclist riding along the car and prevent the car from turning into him?
[+] [-] xedeon|7 years ago|reply
All it takes is 5 minutes to know that it actually uses the built-in safety systems in cars like ADAS, LKAS, ACC. Hence, they only support only specific make/models. It's not just the "single front facing camera".
[+] [-] stephengillie|7 years ago|reply
[+] [-] yters|7 years ago|reply
[+] [-] mirceal|7 years ago|reply
I actually don’t believe this guy. He may be brilliant, but there are quite a few brilliant people working on this problem - so no.
[+] [-] mieseratte|7 years ago|reply
You've piqued my inner language geek, but cursory Googling / Google Translate hasn't given me any luck. What language(s) would that be?