Karpathy's presentation is really good. Watch it later if you get some time. The key points are
- Telsa is using fleet for learning (Fleet learning). This has 3 components,
a. Trigger infrasture that collects the kind of data telsa is looking for training. The rational here is we don't need massive amounts of same kind of data but needs right kind of data for training neural networks correctly.
b. Data Engine which learns from these examples.
c. Shadow mode where they deploy the learned model and test how its doing in real world and iterate this process.
- Drivers are themselves acting as labelers and tesla is in a unique position to take advantage of this.
- People drive vision only. Visual recognition is essential for autonomy. Lidar has much less information than vision, for example to identify the thing on the road is a plastic bag, lidar provides few points. But vision gives more data for telling this.
Karpathy is great at presenting executive-101. However self-driving is much much more than image classification. You have a time component, you need to integrate information as it comes in to meaningful state (aka SLAM) and make sequential decisions that has potential to kill people. This is still unsolved research problem unlike punny ImageNet classification. Karpathy could have easily spent entire day talking about current state of the art and how they have improved on it but instead we got Deep Learning 101. I hope he didn't intentionally used this tactic to avoid talking about the elephant in the room. It seems they are hell bent on vision-based end-to-end learning which vast majority of experts would agree that is far away in the future. So this whole thing is quite bold and if they really make this happen in 1yr3mo then I'd say you should dump your entire 401K in buying Tesla stock. Collecting data is actually more trivial part of this problem.
>Karpathy's presentation is really good. Watch it later if you get some time.
I would second this recommendation. He did a really good job not only explaining Tesla's approach, but also breaking down the concepts in an easy to understand way for laypeople. That is a rare skill to have especially among someone who is obviously so technically accomplished.
It was significant that the brief bits of video showed the system identifying "driveable" areas, shown in blue. That's important. The first step is to conservatively mark driveable areas, where you can physically drive. Then you get to plan where to go.
Amusingly, if you wanted to train a vision system to recognize driveable areas, the straightforward approach would be to use a LIDAR system to measure road flatness. Watching the road go by on video as a human driver drives is not very useful for this. That just tells you about the part of the road a human driver chose to use, not about the road geometry itself. The driver probably won't use the road shoulders, but the drivable area system needs to evaluate them.
Recognizing drivable areas is what keeps you from hitting solid objects. You don't have to classify them, just note that they're not flat road.
> The rational here is we don't need massive amounts of same kind of data but needs right kind of data for training neural networks correctly.
This is really important. At a certain point additional miles on similar roads in nice conditions with fair traffic won’t help anymore. The data variety is really important.
What I found interesting as well is that they (him and Elon) hated the idea of using high definition maps, saying that if you need them you don't do self-driving cars really. For example, when driving condition change (roadblock or anything) you're done.
Does anyone know how Lidar would work in the future when many other vehicles are also using lidar?
From working with some different types of photo sensors (CCD, avalanche photodiodes, photomultiplier tubes), you might imagine that incident light from another car's emitter would be much brighter than the reflected light the lidar sensors are reading and cause saturation.
Can you elaborate on what you mean here? Do you mean that Tesla will only be using Computer Vision for autonomy? Doesn't or wouldn't Lidar compliment CV? Or is there a practical design consideration that makes them mutual exclusive?
No we don't. We have vision which we then interpret with our _minds_. We have the ability to _understand_ what we are seeing, reason from it, and make decisions accordingly. "Deep Learning", no matter how big your dataset, fundamentally cannot possess this capability. Which goes a long way into explaining why Tesla's "self driving" tech keeps on killing people in situations that are trivially simple for humans. There's no way to rigorously test for edge cases in opaque deep learning algorithms, nor is there any way to implement redundancy as there is in ordinary deterministic systems.
This is a mad attempt to replace human cognition with "AI".
I'm sorry but I'm finding it really difficult to watch this and match up what the engineer is saying to what Elon Musk is saying.
For example, the engineer says the custom ASIC does 144 TOps for 2 chips vs the NVidia drive Xavier - does 21 TOps. Okay, well yeah I expect your custom ASIC does have a nice performance advantage over the equivalent GPU. at 3.5x advantage probably seems reasonable. Cue Elon Musk:
"At first seems improbable, how could it be that Tesla, who has never designed a chip before would design the best chip in the world but that is objectively what has occured. Not the best by a small margin, the best by a huge margin".
Mate, it's a dot product with some memory attached, and not a single detail your half hour deep dive has gone into suggests anything other than a bog standard ASIC.
"All the cars being produced right now have all the hardware necessary for full self-driving"
And this is where I'm totally lost. I want to believe! But he's lied so many times now. This man is sucking the credibility out of every engineer in the room. Don't repeat the same lie twice.
I watch Tesla fairly closely from both the bull and bear sides. In short, I don't believe Tesla is anywhere close to Waymo. They won't achieve FSD in 2020.
It's important to see this event in the context of their significant demand and cash problems. Even enthusiastic Tesla investors like Galileo Russell suggested that Tesla is in a cash crunch and should raise money. Which hints at the main mystery about Tesla, why haven't they raised money yet?
Take a moment and think about why are they doing this event now. Elon is setting a stage for a capital raise, he's pitching the autonomy narrative after the Model 3 cash cow narrative failed. They're trying to convince investors (and customers) to give them money because money-printing autonomous taxi service is coming next year.
Also, think about why basically everyone except Tesla uses Lidars. Is it because they're stupid or because Tesla cannot use Lidars even if they wanted to?
If I were a billionaire and I wanted to change the world, I would put pressure on existing companies by creating a competing company and pretending extraordinary achievements.
Truth or not, failure or not, Musk moves forward the hopes and expectations, and I'm thankful just for that.
I don't see how we can blame him for taking position
the fact that suppliers don't like them has been a red flag for a while now
bulls and dreamers don't want to pay any attention to the very imminent problems with tesla's financials and furthermore tesla's executive record
there's a very good reason bears refer to him as fraudboy
but watching him scam taxpayers/unfortunately gullible people/govt (my fav ex. battery swap zev credits) while people fan over him and his company is pretty amazing
Tesla needs to use Lidars too and frequently crashes into things Lidars would save them from (white trucks that loook like the sky, etc.). Unfortunately, Tesla promised that the cars they were shipping had all the equipment needed for self driving, so they can't say that publicly. Lidars would also bump up their cost by thousands, resulting in fewer sales.
...out of all the things I expected to see from a Tesla stream, I definitely did not expect a 25 minute discussion on the cost of 32bit additions, dot products, sram bandwidth, and chip design, at the level of a third year college hardware course.
Holy cow Elon said a lot of things that will drastically change the world in the next 2-3-5 years. I think it's very, very clear they've been doing a lot of stuff behind that scenes that nobody has given them credit for over these last 5 years. Everyone thought they were just throwing everything at the Model 3 ramp and doing nothing else, but it turns out the opposite is true.
At the end Elon said autonomy is basically their entire expense sheet!
Whether or not Tesla can pull it off is obviously going to be an enormous topic of debate with haters and lovers on each side.
This is super, super exciting. I'm going to grab the popcorn and enjoy watching Tesla try. Whether they succeed or fail I admire them aiming so high, and planning so far ahead.
So far, half an hour from the head of IC design. Nice special purpose IC and board. Dual everything for redundancy. Wide special purpose neural net evaluation. 100 watts for the compute system. Code signing. Shipping in new Model 3 cars since last 10 days.
"All we need to do is improve the software" - Musk [12:07 PDT]
LIDAR "unnecessary" - Musk [12:13 PDT]
Computer vision guy is now speaking.
Recognizes "driveable space", not just obstacles. Video shown, but just for a freeway. This is crucial to safety. Need to see this is a cluttered environment.
> "All we need to do is improve the software" - Musk [12:07 PDT]
That's an old claim, see this 2016 press release [1]:
> as of today, all Tesla vehicles produced in our factory – including Model 3 – will have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.
At this point I believe the claim once I see the self-driving functionality having been rolled out to the public and the accident number been reduced.
This talk of a "cut-in" detector is scaring me. It's like they lack any sort of higher level planning and decision making (which notoriously is not a neural network of any sort).
the narrative in this conference seems, minus the exaggeration, to follow exactly the flow for the presentations given by the MobilEye CTO for the last six years.
The thing I don't get about Tesla is why they keep heaping more on their plate rather than just (for large values of just) re-doing the motive power bit. That alone is a massive challenge and it is not something they've licked at economies of scale that allow a $10K vehicle to be brought to market, which is what it will take to really make this a success.
Every extra they tack on to the base product is one they will be expected to deliver on as well diverting attention from the main problem they are dealing with, which may in the longer term leave open enough room for the competition to wiggle through.
Sure, autonomy is a big deal, but it is also something that will once cracked be an instant commodity and there is a lot of money and talent focused on that particular problem, which is surprisingly hard to do well.
If Tesla ends up going under because of one of the side shows (Solar City, autonomy, Power Wall etc) that would be a serious loss.
- First principles hardware design of focused self driving computer (many times better than any competing existing hardware). Already shipping in all newly produced cars. Currently working on next gen that will be 3x better to ship in a couple years.
- Lidar is an unnecessary mistake that competitors are making that won't succeed (too expensive, need too many, unnecessary).
- Real world fleet testing is critical to success, simulations are not good enough since there are too many unknown unknowns in the real world. Tesla uses simulations too, but nobody else comes close on real world fleet testing.
At the tail end of Karpathy's presentation, he said something that reminded me of why Peter Norvig decided to join Google: because that's where the data is. In Tesla's case, they have a unique and likely accelerating advantage in having the best source of data up on which to train their models. I think this forms the basis of an enduring moat relative to their other competitors, none of which are collecting data at the same scale.
While you're waiting for the main event to start, here are some recent interviews with Elon about self-driving cars. He's very confident.
"To me right now, this seems 'game, set, and match,'" Musk said. "I could be wrong, but it appears to be the case that Tesla is vastly ahead of everyone."
And Elon has a long history of making false claims about Tesla’s progress. For example in 2015 and 2016 he claimed that Teslas would be fully self-driving by 2018.
My guess is he means "on the highway". The scary bits of self-driving is person detection, crossing detection, roadwork detection, cyclist detection (e.g. coming up on the right when you are trying to make a right turn).
The Waymo end-game that I heard was "able to go through a drive-thru". I highly doubt Tesla is anywhere near that point.
Well they’re vastly ahead in one area: data collection. No other company is even close. You could argue about the quality of data but the platform is there and ever growing, and they can upgrade their hardware in the future and augment existing data.
I really really don't want Tesla to die. I think it's an important company for a sustainable future. Failures in autonomous driving could easily turn into the straw that breaks the camels back.
If you call something "autopilot" and promote it as if it will drive for you and then it ends up killing dozens of people ... that's where successful class actions come from.
Musk: "You're only going real fast in the forward direction."
Dozens of times in my life, I have been driving down the freeway at 70 MPH in a 60 MPH zone, and I've noticed someone weaving through traffic behind me, going closer to 140 MPH.
I need to know whether to change lanes, stay in my lane, stop changing lanes, pump my breaks to indicate there is slowdown ahead of me that car might not see, etc.
Just food for thought.
EDIT: I'd also like my vehicle to be good at avoiding a car that's about to T-Bone me, at night, with no headlights on. I may not be very good at avoiding that kind of accident today, but if LIDAR is necessary to protect me from that kind of accident, then I might think it's a wonderful idea.
Those are some very bold claims. Level 5 by the end of this year? I find the software approach intriguing and Karpathy's segment was enlightening and did a lot to convince me of Tesla's advantages.
On the other hand, I've been sensing that Tesla is finding it harder and harder to raise cash and has been getting increasingly desperate. Are we on the cusp of a new transformative technology or the peak of the mother of all bubbles? Time will tell.
Option 1: They have solved a problem so hard that no one else, even Waymo, is claiming to be close even with $100k+ worth of sensors and a lot more compute and they have also quietly solved multiple hard unsolved research problems.
Here's the new self-driving demo video from today.[1] From Tesla's HQ in Palo Alto, out to I-280, down one exit, use interchange at Sand Hill to turn around, come back. No visible conflicting traffic on non-freeway streets.
Compare the 2016 demo video.[2] That's a tougher route. That's the one where we now know it took a lot of tries to get a clean video.
Waymo and Cruise have put up videos of their cars in city traffic. They get criticized for things like getting stuck behind double-parked cars, and being a bit shy of parked cars that project into a traffic lane. But they get where they are going. Tesla is not showing anything near that level.
Supposedly the analysts at the meeting got to ride in a self-driving car. Anyone seen reports from them?
Feels like they're just throwing a bunch of hardware specs at investors to distract from any discussions about the current state of the software beyond "just need to improve it now".
You cannot access the stream from Tesla anymore, I found it here - https://www.youtube.com/watch?v=tbgtGQIygZQ
Karpathy did a great job explaining this, I took Neural Networks a long time back and his back to basics approach refreshed a lot of concepts I'd forgotten!
[+] [-] sairahul82|6 years ago|reply
- Telsa is using fleet for learning (Fleet learning). This has 3 components, a. Trigger infrasture that collects the kind of data telsa is looking for training. The rational here is we don't need massive amounts of same kind of data but needs right kind of data for training neural networks correctly. b. Data Engine which learns from these examples. c. Shadow mode where they deploy the learned model and test how its doing in real world and iterate this process.
- Drivers are themselves acting as labelers and tesla is in a unique position to take advantage of this.
- People drive vision only. Visual recognition is essential for autonomy. Lidar has much less information than vision, for example to identify the thing on the road is a plastic bag, lidar provides few points. But vision gives more data for telling this.
[+] [-] sytelus|6 years ago|reply
[+] [-] slg|6 years ago|reply
I would second this recommendation. He did a really good job not only explaining Tesla's approach, but also breaking down the concepts in an easy to understand way for laypeople. That is a rare skill to have especially among someone who is obviously so technically accomplished.
[+] [-] Animats|6 years ago|reply
Amusingly, if you wanted to train a vision system to recognize driveable areas, the straightforward approach would be to use a LIDAR system to measure road flatness. Watching the road go by on video as a human driver drives is not very useful for this. That just tells you about the part of the road a human driver chose to use, not about the road geometry itself. The driver probably won't use the road shoulders, but the drivable area system needs to evaluate them.
Recognizing drivable areas is what keeps you from hitting solid objects. You don't have to classify them, just note that they're not flat road.
[+] [-] b_tterc_p|6 years ago|reply
This is really important. At a certain point additional miles on similar roads in nice conditions with fair traffic won’t help anymore. The data variety is really important.
[+] [-] krzkaczor|6 years ago|reply
[+] [-] torpfactory|6 years ago|reply
From working with some different types of photo sensors (CCD, avalanche photodiodes, photomultiplier tubes), you might imagine that incident light from another car's emitter would be much brighter than the reflected light the lidar sensors are reading and cause saturation.
[+] [-] tjoff|6 years ago|reply
[+] [-] mzs|6 years ago|reply
[+] [-] bogomipz|6 years ago|reply
Can you elaborate on what you mean here? Do you mean that Tesla will only be using Computer Vision for autonomy? Doesn't or wouldn't Lidar compliment CV? Or is there a practical design consideration that makes them mutual exclusive?
[+] [-] DeonPenny|6 years ago|reply
[+] [-] ibeckermayer|6 years ago|reply
No we don't. We have vision which we then interpret with our _minds_. We have the ability to _understand_ what we are seeing, reason from it, and make decisions accordingly. "Deep Learning", no matter how big your dataset, fundamentally cannot possess this capability. Which goes a long way into explaining why Tesla's "self driving" tech keeps on killing people in situations that are trivially simple for humans. There's no way to rigorously test for edge cases in opaque deep learning algorithms, nor is there any way to implement redundancy as there is in ordinary deterministic systems.
This is a mad attempt to replace human cognition with "AI".
[+] [-] Traster|6 years ago|reply
For example, the engineer says the custom ASIC does 144 TOps for 2 chips vs the NVidia drive Xavier - does 21 TOps. Okay, well yeah I expect your custom ASIC does have a nice performance advantage over the equivalent GPU. at 3.5x advantage probably seems reasonable. Cue Elon Musk:
"At first seems improbable, how could it be that Tesla, who has never designed a chip before would design the best chip in the world but that is objectively what has occured. Not the best by a small margin, the best by a huge margin".
Mate, it's a dot product with some memory attached, and not a single detail your half hour deep dive has gone into suggests anything other than a bog standard ASIC.
"All the cars being produced right now have all the hardware necessary for full self-driving"
And this is where I'm totally lost. I want to believe! But he's lied so many times now. This man is sucking the credibility out of every engineer in the room. Don't repeat the same lie twice.
[+] [-] RivieraKid|6 years ago|reply
It's important to see this event in the context of their significant demand and cash problems. Even enthusiastic Tesla investors like Galileo Russell suggested that Tesla is in a cash crunch and should raise money. Which hints at the main mystery about Tesla, why haven't they raised money yet?
Take a moment and think about why are they doing this event now. Elon is setting a stage for a capital raise, he's pitching the autonomy narrative after the Model 3 cash cow narrative failed. They're trying to convince investors (and customers) to give them money because money-printing autonomous taxi service is coming next year.
Also, think about why basically everyone except Tesla uses Lidars. Is it because they're stupid or because Tesla cannot use Lidars even if they wanted to?
P.S.: Nvidia issued a statement saying that Tesla's claims about their chip are incorrect: https://www.marketwatch.com/story/nvidia-says-tesla-inaccura...
Edit: In 2012, Waymo reached the milestone of handling 8 100-mile routes, specifically chosen to capture the full complexity of driving. I doubt Tesla is currently at that level. Source: https://events.technologyreview.com/video/watch/dmitri-dolgo...
[+] [-] antpls|6 years ago|reply
Truth or not, failure or not, Musk moves forward the hopes and expectations, and I'm thankful just for that.
I don't see how we can blame him for taking position
[+] [-] hi5eyes|6 years ago|reply
the fact that suppliers don't like them has been a red flag for a while now
bulls and dreamers don't want to pay any attention to the very imminent problems with tesla's financials and furthermore tesla's executive record
there's a very good reason bears refer to him as fraudboy
but watching him scam taxpayers/unfortunately gullible people/govt (my fav ex. battery swap zev credits) while people fan over him and his company is pretty amazing
also the list of lawsuits...
[+] [-] lnanek2|6 years ago|reply
[+] [-] dblotsky|6 years ago|reply
[+] [-] eanzenberg|6 years ago|reply
[+] [-] 2bitencryption|6 years ago|reply
[+] [-] grecy|6 years ago|reply
At the end Elon said autonomy is basically their entire expense sheet!
Whether or not Tesla can pull it off is obviously going to be an enormous topic of debate with haters and lovers on each side.
This is super, super exciting. I'm going to grab the popcorn and enjoy watching Tesla try. Whether they succeed or fail I admire them aiming so high, and planning so far ahead.
[+] [-] Animats|6 years ago|reply
"All we need to do is improve the software" - Musk [12:07 PDT]
LIDAR "unnecessary" - Musk [12:13 PDT]
Computer vision guy is now speaking.
Recognizes "driveable space", not just obstacles. Video shown, but just for a freeway. This is crucial to safety. Need to see this is a cluttered environment.
[+] [-] est31|6 years ago|reply
That's an old claim, see this 2016 press release [1]:
> as of today, all Tesla vehicles produced in our factory – including Model 3 – will have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.
At this point I believe the claim once I see the self-driving functionality having been rolled out to the public and the accident number been reduced.
[1]: https://www.tesla.com/blog/all-tesla-cars-being-produced-now...
[+] [-] isoprophlex|6 years ago|reply
[+] [-] stefan_|6 years ago|reply
[+] [-] Animats|6 years ago|reply
[+] [-] foobiekr|6 years ago|reply
[+] [-] 6d6b73|6 years ago|reply
[+] [-] jacquesm|6 years ago|reply
Every extra they tack on to the base product is one they will be expected to deliver on as well diverting attention from the main problem they are dealing with, which may in the longer term leave open enough room for the competition to wiggle through.
Sure, autonomy is a big deal, but it is also something that will once cracked be an instant commodity and there is a lot of money and talent focused on that particular problem, which is surprisingly hard to do well.
If Tesla ends up going under because of one of the side shows (Solar City, autonomy, Power Wall etc) that would be a serious loss.
[+] [-] gonehome|6 years ago|reply
- First principles hardware design of focused self driving computer (many times better than any competing existing hardware). Already shipping in all newly produced cars. Currently working on next gen that will be 3x better to ship in a couple years.
- Lidar is an unnecessary mistake that competitors are making that won't succeed (too expensive, need too many, unnecessary).
- Real world fleet testing is critical to success, simulations are not good enough since there are too many unknown unknowns in the real world. Tesla uses simulations too, but nobody else comes close on real world fleet testing.
[+] [-] iandanforth|6 years ago|reply
Every law enforcement entity on the planet: drool
[+] [-] bluthru|6 years ago|reply
[+] [-] xedeon|6 years ago|reply
[+] [-] localhost|6 years ago|reply
[+] [-] netinstructions|6 years ago|reply
"To me right now, this seems 'game, set, and match,'" Musk said. "I could be wrong, but it appears to be the case that Tesla is vastly ahead of everyone."
I am eager to see what they unveil today.
https://www.youtube.com/watch?v=dEv99vxKjVI
https://ark-invest.com/research/podcast/elon-musk-podcast
[+] [-] dforrestwilson|6 years ago|reply
https://www.google.com/url?sa=i&source=web&cd=&ved=2ahUKEwiR...
And Elon has a long history of making false claims about Tesla’s progress. For example in 2015 and 2016 he claimed that Teslas would be fully self-driving by 2018.
So why shouldn’t we be skeptical?
https://arstechnica.com/cars/2019/03/teslas-self-driving-str...
[+] [-] cflewis|6 years ago|reply
The Waymo end-game that I heard was "able to go through a drive-thru". I highly doubt Tesla is anywhere near that point.
[+] [-] eaurouge|6 years ago|reply
[+] [-] tigershark|6 years ago|reply
[+] [-] Hamuko|6 years ago|reply
[+] [-] iandanforth|6 years ago|reply
If you call something "autopilot" and promote it as if it will drive for you and then it ends up killing dozens of people ... that's where successful class actions come from.
(I obviously don't want people to die either.)
[+] [-] cmsonger|6 years ago|reply
That does not mean that their cars can self drive today.
That does not mean that their cars can self drive three years from now.
It's 100% not proven or obvious how car self driving skill and car self driving error rates scale with compute -- but it's surely not linear.
[+] [-] VikingCoder|6 years ago|reply
Dozens of times in my life, I have been driving down the freeway at 70 MPH in a 60 MPH zone, and I've noticed someone weaving through traffic behind me, going closer to 140 MPH.
I need to know whether to change lanes, stay in my lane, stop changing lanes, pump my breaks to indicate there is slowdown ahead of me that car might not see, etc.
Just food for thought.
EDIT: I'd also like my vehicle to be good at avoiding a car that's about to T-Bone me, at night, with no headlights on. I may not be very good at avoiding that kind of accident today, but if LIDAR is necessary to protect me from that kind of accident, then I might think it's a wonderful idea.
[+] [-] dougmwne|6 years ago|reply
On the other hand, I've been sensing that Tesla is finding it harder and harder to raise cash and has been getting increasingly desperate. Are we on the cusp of a new transformative technology or the peak of the mother of all bubbles? Time will tell.
[+] [-] hellllllllooo|6 years ago|reply
Option 2: He's lying to get money.
[+] [-] Animats|6 years ago|reply
Compare the 2016 demo video.[2] That's a tougher route. That's the one where we now know it took a lot of tries to get a clean video.
Waymo and Cruise have put up videos of their cars in city traffic. They get criticized for things like getting stuck behind double-parked cars, and being a bit shy of parked cars that project into a traffic lane. But they get where they are going. Tesla is not showing anything near that level.
Supposedly the analysts at the meeting got to ride in a self-driving car. Anyone seen reports from them?
[1] https://www.youtube.com/watch?v=tlThdr3O5Qo [2] https://www.youtube.com/watch?v=eAal0juXXzU
[+] [-] Hamuko|6 years ago|reply
[+] [-] sharadov|6 years ago|reply
[+] [-] pepijndevos|6 years ago|reply
[+] [-] grey-area|6 years ago|reply
Strong words from Musk about sticking with video only.
Later on in the software talk:
“Lidar is really a shortcut which sidesteps the fundamental problems...and gives us a false sense of progress”