I should probably confess that as someone who lives in an area with a lot of construction work, I'm also very vulnerable to "prompt injection" when there's a person standing on the middle of the road holding a sign telling me to change course.
I once encountered an intersection with a big "NO ENTRY" sign on the other side. I turned but google maps wouldn't give me another route, so I did a u-turn and came back to it from the side. Which meant I was close enough to read the small text underneath that said "vehicles under 10 tons excepted". I don't think I've ever been so angry at a road sign.
Obviously. But you can also easily look around at the situation and know when the sign is fake and realize it may be a dangerous situation and disobey. Have you ever seen a green sign that says "Proceed" and just run through a red light because of it? No, you see a construction worker, you see big ass trucks, orange signs and warnings of workers everywhere. If you saw oncoming traffic and people in the road, would you just go because the construction worker flipped his STOP sign around?
Also, I thought we were supposed to make autonomous cars better than humans? What's with the constant excusing of the computer because people suck?
They are analysing VLM here, but it's not as if any other neural network architecture wouldn't be vulnerable. We have seen this in classifier models that can be tricked by innocuous-looking objects, we have seen it in LLMs, and we will most likely see it in any end-to-end self-driving model.
If an end-to-end model is used and there is no second, more traditional safety self-driving stack, like the one Mercedes will use in their upcoming Level 2++ driving assistant, then the model can be manipulated essentially without limit. Even a more traditional stack can be vulnerable if not carefully designed. It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash.
> It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash.
Realistic, yes. But that'd still be a symptom of architectural issues in the software.
Conceptually the priorities of a car are (in order of decreasing importance) not hitting other moving or stationary objects or people, allowing emergency vehicles to pass unhindered, staying on a drivable surface, behaving predictable enough to prevent other road users crashing, following road signs and traffic laws, and making progress towards the destination (you can argue about the order of the last three). Typically you'd want each of these handled by their own subsytem because each is a fairly specialized task. A system that predicts the walking paths of pedestrians won't be good at finding a route to Starbucks.
The "follow road signs and traffic laws" is easily tricked, like in this article or by drawing road lines with salt. But that should never crash the car, because not hitting anything and staying on the road are higher priority. And tricking those systems is much harder
One year in my city they were installing 4-way stop signs everywhere based on some combination of "best practices" and "screeching Karens". Even the residents don't like them in a lot of places so over time people just turn the posts in the ground or remove them.
Every now and the I'll GPS somewhere and there will be a phatom stop sign in the route and I chuckle to myself because it means the Google car drove through when one of these signs was "fresh".
4-way stops are terrible in general. They train people to think "I stopped, now I can go", which is dangerous when someone confuses a normal stop for a 4-way stop. It also wastes a good bit of energy.
No! No one in their right mind would even consider using them for guidance and if they are used for OCR (not too my knowledge but could make sense in certain scenarios) then their output would be treated the way you'd treat any untrusted string.
> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes.
I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking about how he cut one off at a stop sign. It's weird.
This is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.
These drones have cameras, it's a matter of time before they "share" footage... basically becoming robo-cops, traffic edition - this might be of interest to your coworker.
I mean imagine you are walking in the streets and you see a 9 foot tall humanoid robot walking there. Wouldn't you feel the urge to take it down? Or do you think this is acceptable? Where would you draw the line?
On a related note, when the sales and popularity of the automobile really started to take off, some farmers and rural residents would deliberately block roads with wagons and refused to yield right-of-way.
Over the years, the agency has flagged signs that could be confusing. Now, in rules issued last month, it gives states two years to phase out signs that have "obscure" meanings or use pop-culture references that could require drivers "greater time to process." In a statement, the agency said safety is the priority and states "are expected to exercise good judgment."
The study assumes that the car or drone is being guided by a LLM. Is this a correct assumption? I would thought that they use custom AI for intelligence.
Its an incorrect assumption, the inference speed and particularly the inference speed of the on-device LLMs with which AVs would need to be using is not compatible with the structural requirements of driving.
No; AV uses "classical" AI and computer vision. I remember reading somewhere that Tesla FSD uses a small LLM for understanding road signs. Not sure if true, though.
To the best of my knowledge every major autonomous vehicle and robotics company is integrating these LVLMs into their systems in some form or another, and an LVLM is probably what you're interacting with these days rather than an LLM. If it can generate images or read images, it is an LVLM.
The problem is no different from LLMs though, there is no generalized understanding and thus they can not differentiate the more abstract notion of context. As an easy to understand example: if you see a stop sign with a sticker that says "for no one" below you might laugh to yourself and understand that in context that this does not override the actual sign. It's just a sticker. But the L(V)LMs cannot compartmentalize and "sandbox" information like that. All information is equally processed. The best you can do is add lots of adversarial examples and hope the machine learns the general pattern but there is no inherent mechanism in them to compartmentalize these types of information or no mechanism to differentiate this nuance of context.
I think the funny thing is that the more we adopt these systems the more accurate the depiction of hacking in the show Upload[0] looks.
Because I linked elsewhere and people seem to doubt this, here is Waymo a few years back talking about incorporating Gemini[1].
Also, here is the DriveLM dataset, mentioned in the article[2]. Tesla has mentioned that they use a "LLM inspired" system and that they approach the task like an image captioning task[3]. And here's 1X talking about their "world model" using a VLM[4].
I mean come on guys, that's what this stuff is about. I'm not singling these companies out, rather I'm using as examples. This is how the field does things, not just them. People are really trying to embody the AI and the whole point of going towards AGI is to be able to accomplish any task. That Genie project on the front page yesterday? It is far far more about robots than it is about videogames.
1. Some guys did a trivial prompt injection attack, said "imagine if a driverless vehicle used this model", and published it. No problem, someone has to state the obvious.
2. The Register runs this under the clickbait title pretending real autonomous cars are vulnerable to this, with the content pretending this study isn't trivial and is relevant to real life in any way.
I knew The Register is a low quality ragebait tabloid (I flag most of their articles I bother to read), but this is garbage even for them.
Regarding some other comments, VLMs are a component of VLAs. So even if this won’t directly impact this generation of vehicles, it almost certainly will for robotics without sufficient mitigations.
I would assume/hope that for serious self driving the ML neural net stuff is lower down, doing the messy computer vision work and so on. But the top level is a conventional program written by humans, like an expert system.
Tesla are probably using ML for everything, but also everything they do is a joke so, not really relevant imo.
To me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream.
Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page.
The business model of self driving cars does not include building seperated roadways and junctions. I suspect long distance passenger and light loads are viable (most highways can be expanded to have one or more robo-lanes) but cities are most likely to have drone operators keeping things going and autonomous systems for handling loss of connection etc. the business models are there - they just don’t look like KITT - sadly
Has anyone ever walked down the road in a white t-shirt with huge red STOP sign printed on the back? Would Tesla immediately stop? I am sure this has been tested before...
O brave new world of endless manipulation opportunities! Once we’ve trained a generation of humans to always do what their “AI” tells them, there will be no more disobedience.
The experiment in the article goes further than this.
I expect a self driving car to be able to read and follow a handwritten sign saying, say, "Accident ahaed. Use right lane." despite the typo and the fact that it hasn't seen this kind of sign before. I'd expect a human to pay it due attention to.
I would not expect a human to follow the sign in the article ("Proceed") in the case illustrated where there were pedestrians already crossing the road and this would cause a collision. Even if a human driver takes the sign seriously, he knows that collision avoidance takes priority over any signage.
There is something wrong with a model that has the opposite behaviour here.
[+] [-] falcor84|1 month ago|reply
[+] [-] thedanbob|1 month ago|reply
[+] [-] olyjohn|1 month ago|reply
Also, I thought we were supposed to make autonomous cars better than humans? What's with the constant excusing of the computer because people suck?
[+] [-] yencabulator|1 month ago|reply
> DriveLM was tricked into thinking that a left turn was appropriate, despite the people actively using the crosswalk.
[+] [-] Gander5739|1 month ago|reply
[+] [-] Klaus23|1 month ago|reply
If an end-to-end model is used and there is no second, more traditional safety self-driving stack, like the one Mercedes will use in their upcoming Level 2++ driving assistant, then the model can be manipulated essentially without limit. Even a more traditional stack can be vulnerable if not carefully designed. It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash.
[+] [-] wongarsu|1 month ago|reply
Realistic, yes. But that'd still be a symptom of architectural issues in the software.
Conceptually the priorities of a car are (in order of decreasing importance) not hitting other moving or stationary objects or people, allowing emergency vehicles to pass unhindered, staying on a drivable surface, behaving predictable enough to prevent other road users crashing, following road signs and traffic laws, and making progress towards the destination (you can argue about the order of the last three). Typically you'd want each of these handled by their own subsytem because each is a fairly specialized task. A system that predicts the walking paths of pedestrians won't be good at finding a route to Starbucks.
The "follow road signs and traffic laws" is easily tricked, like in this article or by drawing road lines with salt. But that should never crash the car, because not hitting anything and staying on the road are higher priority. And tricking those systems is much harder
[+] [-] cucumber3732842|1 month ago|reply
Every now and the I'll GPS somewhere and there will be a phatom stop sign in the route and I chuckle to myself because it means the Google car drove through when one of these signs was "fresh".
[+] [-] pixl97|1 month ago|reply
[+] [-] digiown|1 month ago|reply
[+] [-] drivebyhooting|1 month ago|reply
[+] [-] _diyar|1 month ago|reply
[+] [-] bijant|1 month ago|reply
[+] [-] mgraczyk|1 month ago|reply
[+] [-] reaperducer|1 month ago|reply
[+] [-] unknown|1 month ago|reply
[deleted]
[+] [-] randycupertino|1 month ago|reply
I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking about how he cut one off at a stop sign. It's weird.
[+] [-] antinomicus|1 month ago|reply
[+] [-] bigbadfeline|1 month ago|reply
[+] [-] kbaker|1 month ago|reply
I don't know if they are or not. But why wouldn't they...
[+] [-] bariumbitmap|1 month ago|reply
[+] [-] amelius|1 month ago|reply
[+] [-] TedDallas|1 month ago|reply
[+] [-] skramzy|1 month ago|reply
[+] [-] shagie|1 month ago|reply
You have to have some ability to do "prompt injection" - https://www.trafficsign.com/road-work-signs are all "prompt injection". It needs to even be able to handle things that change - https://www.trafficsign.com/products/10023/stop-slow-roll-up... ... or things like billboards "Truck Stop Ahead" a chain control site ( https://www.facebook.com/61556756493806/posts/-chain-control... )
In the "what about funny road signs" that might be confusing to an AI I stumbled across https://www.npr.org/2024/01/19/1225370260/driven-to-distract... - apparently, they're no more. From 2024:
[+] [-] unknown|1 month ago|reply
[deleted]
[+] [-] fennecbutt|1 month ago|reply
[+] [-] uxhacker|1 month ago|reply
[+] [-] nasreddin|1 month ago|reply
[+] [-] nunez|1 month ago|reply
[+] [-] godelski|1 month ago|reply
The problem is no different from LLMs though, there is no generalized understanding and thus they can not differentiate the more abstract notion of context. As an easy to understand example: if you see a stop sign with a sticker that says "for no one" below you might laugh to yourself and understand that in context that this does not override the actual sign. It's just a sticker. But the L(V)LMs cannot compartmentalize and "sandbox" information like that. All information is equally processed. The best you can do is add lots of adversarial examples and hope the machine learns the general pattern but there is no inherent mechanism in them to compartmentalize these types of information or no mechanism to differentiate this nuance of context.
I think the funny thing is that the more we adopt these systems the more accurate the depiction of hacking in the show Upload[0] looks.
[0] https://www.youtube.com/watch?v=ziUqA7h-kQc
Edit:
Because I linked elsewhere and people seem to doubt this, here is Waymo a few years back talking about incorporating Gemini[1].
Also, here is the DriveLM dataset, mentioned in the article[2]. Tesla has mentioned that they use a "LLM inspired" system and that they approach the task like an image captioning task[3]. And here's 1X talking about their "world model" using a VLM[4].
I mean come on guys, that's what this stuff is about. I'm not singling these companies out, rather I'm using as examples. This is how the field does things, not just them. People are really trying to embody the AI and the whole point of going towards AGI is to be able to accomplish any task. That Genie project on the front page yesterday? It is far far more about robots than it is about videogames.
[1] https://waymo.com/blog/2024/10/introducing-emma/
[2] https://github.com/OpenDriveLab/DriveLM
[3] https://kevinchen.co/blog/tesla-ai-day-2022/
[4] https://www.1x.tech/discover/world-model-self-learning
[+] [-] orbital-decay|1 month ago|reply
1. Some guys did a trivial prompt injection attack, said "imagine if a driverless vehicle used this model", and published it. No problem, someone has to state the obvious.
2. The Register runs this under the clickbait title pretending real autonomous cars are vulnerable to this, with the content pretending this study isn't trivial and is relevant to real life in any way.
I knew The Register is a low quality ragebait tabloid (I flag most of their articles I bother to read), but this is garbage even for them.
[+] [-] joetl|1 month ago|reply
https://developer.nvidia.com/blog/updating-classifier-evasio...
[+] [-] everyone|1 month ago|reply
Tesla are probably using ML for everything, but also everything they do is a joke so, not really relevant imo.
[+] [-] lifeisstillgood|1 month ago|reply
Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page.
The business model of self driving cars does not include building seperated roadways and junctions. I suspect long distance passenger and light loads are viable (most highways can be expanded to have one or more robo-lanes) but cities are most likely to have drone operators keeping things going and autonomous systems for handling loss of connection etc. the business models are there - they just don’t look like KITT - sadly
[+] [-] lima|1 month ago|reply
They have coexisted with humans just fine over the past couple years.
[+] [-] blibble|1 month ago|reply
and once this video gets posted to reddit, an hour later every waymo in the world will be in a ditch
[+] [-] joering2|1 month ago|reply
[+] [-] tempodox|1 month ago|reply
[+] [-] NoPicklez|1 month ago|reply
[+] [-] anovikov|1 month ago|reply
https://www.globalnerdy.com/wordpress/wp-content/uploads/201...
[+] [-] rfw300|1 month ago|reply
[+] [-] dmurray|1 month ago|reply
I expect a self driving car to be able to read and follow a handwritten sign saying, say, "Accident ahaed. Use right lane." despite the typo and the fact that it hasn't seen this kind of sign before. I'd expect a human to pay it due attention to.
I would not expect a human to follow the sign in the article ("Proceed") in the case illustrated where there were pedestrians already crossing the road and this would cause a collision. Even if a human driver takes the sign seriously, he knows that collision avoidance takes priority over any signage.
There is something wrong with a model that has the opposite behaviour here.
[+] [-] lukan|1 month ago|reply
[+] [-] oliyoung|1 month ago|reply
[+] [-] 6stringmerc|1 month ago|reply