[patapong]

The specific issue here seems to be that Anthropic allows the unrestricted upload of personal files to the anthropic cloud environment, but does not check to make sure that the cloud environment belongs to the user running the session.

This should be relatively simple to fix. But, that would not solve the million other ways a file can be sent to another computer, whether through the user opening a compromised .html document or .pdf file etc etc.

This fundamentally comes down to the issue that we are running intelligent agents that can be turned against us on personal data. In a way, it mirrors the AI Box problem: https://www.yudkowsky.net/singularity/aibox

[jrjeksjd8d]

"a superhuman AI that can brainwash people over text" is the dumbest thing I've read this year. It's incredible to me that this guy has some kind of cult following among people who should know better.

The real answer is that people are lazy and as soon as a security barrier forces them to do work, they want to tear down the barrier. It doesn't take a superhuman AI, it just takes a government employee using their personal email because it's easier. There's been a million MCP "security issues" because they're accepting untrusted, unverifiable inputs and acting with lots of permissions.

[patapong]

Indeed - the problem here is "How can we prevent a somewhat intelligent, potentially malicious agent from exfiltrating data, with or without human involvement", rather than the superhuman AI stuff. Still a hard problem to solve I think!

[3form]

A set of ideas presented to people, and a notion of being smarter for believing in them seems enough to fuel enough of thought-problem-keyboard-warriorism.