[rpicard]

I’ve noticed a strong negative streak in the security community around LLMs. Lots of comments about how they’ll just generate more vulnerabilities, “junk code”, etc.

It seems very short sighted.

I think of it more like self driving cars. I expect the error rate to quickly become lower than humans.

Maybe in a couple of years we’ll consider it irresponsible not to write security and safety critical code with frontier LLMs.

[xnorswap]

I've been watching a twitch streamer vibe-code a game.

Very quickly he went straight to, "Fuck it, the LLM can execute anything, anywhere, anytime, full YOLO".

Part of that is his risk-appetite, but it's also partly because anything else is just really furstrating.

Someone who doesn't themselves code isn't going to understand what they're being asked to allow or deny anyway.

To the pure vibe-coder, who doesn't just not read the code, they couldn't read the code if they tried, there's no difference between "Can I execute grep -e foo */*.ts" and "Can I execute rm -rf /".

Both are meaningless to them. How do you communicate real risk? Asking vibe-coders to understand the commands isn't going to cut it.

So people just full allow all and pray.

That's a security nightmare, it's back to a default-allow permissive environment that we haven't really seen in mass-use, general purpose internet connected devices since windows 98.

The wider PC industry has got very good at UX to the point where most people don't need to worry themselves about how their computer works at all and still successfully hide most of the security trappings and keep it secure.

Meanwhile the AI/LLM side is so rough it basically forces the layperson to open a huge hole they don't understand to make it work.

[tootubular]

I know exactly the streamer you're referring to and this is the first time I've seen an overlap between these two worlds! I bet there are quite a few of us. Anyway, agreed on all accounts, watching someone like him has been really eye opening on how some people use these tools ... and it's not pretty.

[voidUpdate]

Yeah, it does sound a lot like self-driving cars. Everyone talks about how they're amazing and will do everything for you but you actually have to constantly hold their hand because they aren't as capable as they're made out to be

[bpt3]

You're talking about a theoretical problem in the future, while I assure you vibe coding and agent based coding is causing major issues today.

Today, LLMs make development faster, not better.

And I'd be willing to bet a lot of money they won't be significantly better than a competent human in the next decade, let alone the next couple years. See self-driving cars as an example that supports my position, not yours.

[anonzzzies]

Does it matter though? Programming was already terrible. There are a few companies doing good things, the rest made garbage already for the past decades. No one cares (well; consumers don't care; companies just have insurance when it happens so they don't really care either; it's just a necessary line item) about their data being exposed etc as long as things are cheap cheap. People daily work with systems that are terrible in every way and then they get hacked (for ransom or not). Now we can just make things cheaper/faster and people will like it. Even at the current level software will be vastly easier and faster to make; sure it will suck, but I'm not sure anyone outside HN cares in any way shape or form (I know our clients don't; they are shipping garbage faster than ever and they see our service as a necessary business expense IF something breaks/messes up). Which means that it won't matter if LLMs get better; it matters that they get a lot cheaper so we can just run massive amounts of them on every device committing code 24/7 and that we keep up our tooling to find possible minefields faster and bandaid them until the next issue pops up.

[furyofantares]

> Today, LLMs make development faster, not better.

You don't have to use them this way. It's just extremely tempting and addictive.

You can choose to talk to them about code rather than features, using them to develop better code at a normal speed instead of worse code faster. But that's hard work.

[philipp-gayret]

What metric would you measure to determine whether a fully AI-based flow is better than a competent human engineer? And how much would you like to bet?

[kriops]

> I think of it more like self driving cars.

Analogous to the way I think of self-driving cars is the way I think of fusion: perpetually a few years away from a 'real' breakthrough.

There is currently no reason to believe that LLMs cannot acquire the ability to write secure code in the most prevalent use cases. However, this is contingent upon the availability of appropriate tooling, likely a Rust-like compiler. Furthermore, there's no reason to think that LLMs will become useful tools for validating the security of applications at either the model or implementation level—though they can be useful for detecting quick wins.

[lxgr]

Have you ever taken a Waymo? I wish fusion was as far along!

[rpicard]

My car can drive itself today.

[andrepd]

Let's maybe cross that bridge when (more important, if!) we come to it then? We have no idea how LLMs are gonna evolve, but clearly now they are very much not ready for the job.

[kingstnap]

For now we train LLMs on next token prediction and Fill-in-the-middle for code. This exactly reflects in the experience of using them in that over time they produce more and more garbage.

It's optimistic but maybe once we start training them on "remove the middle" instead it could help make code better.

[tptacek]

There are plenty of security people on the other side of this issue; they're just not making news, because the way you make news in security is by announcing vulnerabilities. By way of example, last I checked, Dave Aitel was at OpenAI.

[rpicard]

Fair! I’ve been surprised in some cases. I’m thinking specifically of a handful of conversations I was in or around during the Vegas cons.

I might also be hyper sensitive to the cynicism. It tends to bug me more than it probably should.

[croes]

It’s the same problem as with self driving cars.

Self driving cars maybe be better than the average driver but worse than the top drivers.

For security code it’s the same.

[lxgr]

Regardless of whether that comparison is valid: In a world where the average driver is average, that honestly doesn't sound too bad.