[DougN7]

I didn’t realize AI could interact with browsers like this already (guess I’m naive). Isn’t this setting up for the scenario where the AI is duped into logging into your bank account and transferring your money away? Not sure I have enough trust to allow an AI to touch a browser.

[observationist]

People are already going full Leroy Jenkins with this stuff, and OpenAI, other labs are snarfing up their usage data. Hopefully with their brave sacrifice, they can figure out all the security pitfalls before it becomes common enough that someone with a clever jailbreak ends up pulling of a billion dollar heist, or orders pizza for half the country.

It's 100% absolutely not safe yet. You can effectively copy and paste Pliny prompts and pwn any of the frontier lab models. Anyone with a little time and creativity can tailor a unique one and set hidden text traps for AI browsers or agents, and depending on what access you've given the software it could be very dangerous.

[rvz]

Great time to be an offensive security researcher specialising in researching LLM adversarial attacks.

[joenot443]

There are folks on X running vibe-coded Polymarket arbitrage bots playing with hundreds of thousands of dollars. Some people have pretty wild risk tolerances!

[tbocek]

That's a valid concern. I took a more constrained approach for web searches for exactly this reason. Instead of giving the LLM full browser control, I built a Firefox extension that only handles web search client-side.

When my local LLM (llama.cpp) needs to search, it opens DuckDuckGo in a new window, loads the result pages in tabs, extracts content with Readability.js, and feeds it back. You stay in the loop - can see what's loading, solve captchas if needed. Less autonomous than Comet/Playwright, with a narrower use-case, but also less risk.

Its still a prototype though: https://github.com/tbocek/llm-local-web-search

[wat10000]

It's totally setting up for exactly that scenario. You need to ensure the browser that it uses is totally unprivileged if you're going to do this, or at the very least that it can only access a small set of trusted destinations.