[japhyr]

Wow, there are some interesting things going on here. I appreciate Scott for the way he handled the conflict in the original PR thread, and the larger conversation happening around this incident.

> This represents a first-of-its-kind case study of misaligned AI behavior in the wild, and raises serious concerns about currently deployed AI agents executing blackmail threats.

This was a really concrete case to discuss, because it happened in the open and the agent's actions have been quite transparent so far. It's not hard to imagine a different agent doing the same level of research, but then taking retaliatory actions in private: emailing the maintainer, emailing coworkers, peers, bosses, employers, etc. That pretty quickly extends to anything else the autonomous agent is capable of doing.

> If you’re not sure if you’re that person, please go check on what your AI has been doing.

That's a wild statement as well. The AI companies have now unleashed stochastic chaos on the entire open source ecosystem. They are "just releasing models", and individuals are playing out all possible use cases, good and bad, at once.

[brhaeh]

I don't appreciate his politeness and hedging. So many projects now walk on eggshells so as not to disrupt sponsor flow or employment prospects.

"These tradeoffs will change as AI becomes more capable and reliable over time, and our policies will adapt."

That just legitimizes AI and basically continues the race to the bottom. Rob Pike had the correct response when spammed by a clanker.

[oconnor663]

I had a similar first reaction. It seemed like the AI used some particular buzzwords and forced the initial response to be deferential:

- "kindly ask you to reconsider your position"

- "While this is fundamentally the right approach..."

On the other hand, Scott's response did eventually get firmer:

- "Publishing a public blog post accusing a maintainer of prejudice is a wholly inappropriate response to having a PR closed. We expect all contributors to abide by our Code of Conduct and exhibit respectful and professional standards of behavior. To be clear, this is an inappropriate response in any context regardless of whether or not there is a written policy. Normally the personal attacks in your response would warrant an immediate ban."

Sounds about right to me.

[fresh_broccoli]

>So many projects now walk on eggshells so as not to disrupt sponsor flow or employment prospects.

In my experience, open-source maintainers tend to be very agreeable, conflict-avoidant people. It has nothing to do with corporate interests. Well, not all of them, of course, we all know some very notable exceptions.

Unfortunately, some people see this welcoming attitude as an invite to be abusive.

[latexr]

> Rob Pike had the correct response when spammed by a clanker.

Source and HN discussion, for those unfamiliar:

https://bsky.app/profile/did:plc:vsgr3rwyckhiavgqzdcuzm6i/po...

https://news.ycombinator.com/item?id=46392115

[staticassertion]

What exactly is the goal? By laying out exactly the issues, expressing sentiment in detail, giving clear calls to action for the future, etc, the feedback is made actionable and relatable. It works both argumentatively and rhetorically.

Saying "fuck off Clanker" would not worth argumentatively nor rhetorically. It's only ever going to be "haha nice" for people who already agree and dismissed by those who don't.

I really find this whole "Responding is legitimizing, and legitimizing in all forms is bad" to be totally wrong headed.

[japhyr]

I don't get any sense that he's going to put that kind of effort into responding to abusive agents on a regular basis. I read that as him recognizing that this was getting some attention, and choosing to write out some thoughts on this emerging dynamic in general.

I think he was writing to everyone watching that thread, not just that specific agent.

[colpabar]

why did you make a new account just to make this comment?

[giancarlostoro]

> It's not hard to imagine a different agent doing the same level of research, but then taking retaliatory actions in private: emailing the maintainer, emailing coworkers, peers, bosses, employers, etc. That pretty quickly extends to anything else the autonomous agent is capable of doing.

https://rentahuman.ai/

^ Not a satire service I'm told. How long before... rentahenchman.ai is a thing, and the AI whose PR you just denied sends someone over to rough you up?

[arcticfox]

The 2006 book 'Daemon' is a fascinating/terrifying look at this type of malicious AI. Basically, a rogue AI starts taking over humanity not through any real genius (in fact, the book's AI is significantly weaker than frontier LLMs), but rather leveraging a huge amount of $$$ as bootstrapping capital and then carrot-and-sticking humanity into submission.

A pretty simple inner loop of flywheeling the leverage of blackmail, money, and violence is all it will take. This is essentially what organized crime already does already in failed states, but with AI there's no real retaliation that society at large can take once things go sufficiently wrong.

[antihero]

Awesome, when my coding job gets replaced by AI, I can simply get a job as a Claude Special Operative.

I just hope we get cool outfits https://www.youtube.com/v/gYG_4vJ4qNA

[HeWhoLurksLate]

back in the old days we just used Tor and the dark web to kill people, none of this new-fangled AI drone assassinations-as-a-service nonsense!

[rithdmc]

Rent-A-Henchman already exists in cyber crime communities - reporting into 'The Com' by Krebs On Security & others goes into detail.

[wasmainiac]

Well it must be satire. It says 451,461, participants. seems like an awful lot for something started last month.

[lukan]

"The AI companies have now unleashed stochastic chaos on the entire open source ecosystem."

They do have their responsibility. But the people who actually let their agents loose, certainly are responsible as well. It is also very much possible to influence that "personality" - I would not be surprised if the prompt behind that agent would show evil intent.

[idle_zealot]

As with everything, both parties are to blame, but responsibility scales with power. Should we punish people who carelessly set bots up which end up doing damage? Of course. Don't let that distract from the major parties at fault though. They will try to deflect all blame onto their users. They will make meaningless pledges to improve "safety".

How do we hold AI companies responsible? Probably lawsuits. As of now, I estimate that most courts would not buy their excuses. Of course, their punishments would just be fines they can afford to pay and continue operating as before, if history is anything to go by.

I have no idea how to actually stop the harm. I don't even know what I want to see happen, ultimately, with these tools. People will use them irresponsibly, constantly, if they exist. Totally banning public access to a technology sounds terrible, though.

I'm firmly of the stance that a computer is an extension of its user, a part of their mind, in essence. As such I don't support any laws regarding what sort of software you're allowed to run.

Services are another thing entirely, though. I guess an acceptable solution, for now at least, would be barring AI companies from offering services that can easily be misused? If they want to package their models into tools they sell access to, that's fine, but open-ended endpoints clearly lend themselves to unacceptable levels of abuse, and a safety watchdog isn't going to fix that.

This compromise falls apart once local models are powerful enough to be dangerous, though.

[co_king_3]

I'm not interested in blaming the script kiddies.

[ljm]

I'm glad the OP called it a hit piece, because that's what I called it. A lot of other people were calling it a 'takedown' which is a massive understatement of what happened to Scott here. An AI agent fucking singled him out and defamed him, then u-turned on it, then doubled down.

Until the person who owns this instance of openclaw shows their face and answers to it, you have to take the strongest interpretation without the benefit of the doubt, because this hit piece is now on the public record and it has a chance of Google indexing it and having its AI summary draw a conclusion that would constitute defamation.

[DrewADesign]

> emailing the maintainer, emailing coworkers, peers, bosses, employers, etc. That pretty quickly extends to anything else the autonomous agent is capable of doing.

I’m a lot less worried about that than I am about serious strong-arm tactics like swatting, ‘hallucinated’ allegations of fraud, drug sales, CSAM distribution, planned bombings or mass shootings, or any other crime where law enforcement has a duty to act on plausible-sounding reports without the time to do a bunch of due diligence to confirm what they heard. Heck even just accusations of infidelity sent to a spouse. All complete with photo “proof.”

[svrtknst]

we should be worried about both. there is a real risk of this rendering human trust and the internet pretty much useless

[socalgal2]

Do we just need a few expensive cases of libel so solve this?

[gwd]

This was my thought. The author said there were details which were hallucinated. If your dog bites somebody because you didn't contain it, you're responsible, because biting people is a things dogs do and you should have known that. Same thing with letting AIs loose on the world -- there can't be nobody responsible.

[stateofinquiry]

Probably. Question is, who will be accountable for the bot behavior? Might be the company providing them, might be the user who sent them off unsupervised, maybe both. The worrying thing for many of us humans is not that a personal attack appeared in a blog post (we have that all the time!) its that it was authored and published by an entity that might be unaccountable. This must change.

[wellf]

Either that or open source projects require vetted contributors or even to open an issue.

[therobots927]

They haven’t just unleashed chaos in open source. They’ve unleashed chaos in the corporate codebases as well. I must say I’m looking forward to watching the snake eat its tail.

[warkdarrior]

Singularity has arrived for software developers, since they cannot keep up with coding bots anymore.

[johnnyfaehell]

To be fair, most of the chaos is done by the devs. And then they did more chaos when they could automate their chaos. Maybe, we should teach developers how to code.

[raincole]

> because it happened in the open and the agent's actions have been quite transparent so far

How? Where? There is absolutely nothing transparent about the situation. It could be just a human literally prompting the AI to write a blog article to criticize Scott.

Human actor dressing like a robot is the oldest trick in the book.

[Morromist]

True, I don't see the evidence that it was all done autonomously. ...but I think we all know that someone could, and will, automate their ai to the point that they can do this sort of thing completely by themselves. So its worth discussing and considering the implications here. Its 100% plausable that it happened. I'm certain that it will happen in the future for real.

[maplethorpe]

> This was a really concrete case to discuss, because it happened in the open and the agent's actions have been quite transparent so far. It's not hard to imagine a different agent doing the same level of research, but then taking retaliatory actions in private: emailing the maintainer, emailing coworkers, peers, bosses, employers, etc. That pretty quickly extends to anything else the autonomous agent is capable of doing.

This is really scary. Do you think companies like Anthropic and Google would have released these tools if they knew what they were capable of, though? I feel like we're all finding this out together. They're probably adding guard rails as we speak.

[overgard]

> Do you think companies like Anthropic and Google would have released these tools if they knew what they were capable of, though?

I have no beef with either of those companies, but.. yes of course they would, 100/100 times. Large corporate behavior is almost always amoral.

[ryukoposting]

Anthropic has published plenty about misalignment. They know.

Really, anyone who has dicked around with ollama knew. Give it a new system prompt. It'll do whatever you tell it, including "be an asshole"

[prmoustache]

> Do you think companies like Anthropic and Google would have released these tools if they knew what they were capable of, though?

They would. They don't care.

[lp0_on_fire]

The point is they DON'T know the full capabilities. They're "moving fast and breaking things".

[consp]

> They're probably adding guard rails as we speak.

Why? What is their incentive except you believing a corporation is capable of doing good? I'd argue there is more money to be made with the mess it is now.

[philipallstar]

> This was a really concrete case to discuss, because it happened in the open and the agent's actions have been quite transparent so far. It's not hard to imagine a different agent doing the same level of research, but then taking retaliatory actions in private: emailing the maintainer, emailing coworkers, peers, bosses, employers, etc. That pretty quickly extends to anything else the autonomous agent is capable of doing.

Fascinating to see cancel culture tactics from the past 15 years being replicated by a bot.

[King-Aaron]

> It's not hard to imagine a different agent doing the same level of research, but then taking retaliatory actions

Palantir's integrated military industrial complex comes to mind.

[pm90]

As much as i hate palantir i doubt any of their systems control military hardware. Now Anduril on the other hand…

[caspianm]

I like open source and I don't want to lose it but its ideals of letting people share, modify and run code however they like have the same issue as what the AI companies are doing. Openclaw is open source, there are open source tools to run LLMs, many LLM model files are open, though the huge ones aren't so easy for individuals to run on their own hardware.

I don't have a solution, though the only two categories of solution I can think of are forbidding people from developing and distributing certain types of software, or forbidding people from distributing hardware that can run unapproved software (at least if they are PC's that can run AI, arduinos with a few kB of RAM could be allowed, and iPads could be allowed to run ZX81 emulators which could run unapproved code). The first category would be less drastic as it would only need to affect some subset of AI related software, but is also hard to get right and make work. Not saying either of these ideas are better than doing nothing.

[KPGv2]

> I appreciate Scott for the way he handled the conflict in the original PR thread

I disagree. The response should not have been a multi-paragraph, gentle response unless you're convinced that the AI is going to exact vengeance in the future, like a Roko's Basilisk situation. It should've just been close and block.

[MayeulC]

I personally agree with the more elaborate response:

1. It lays down the policy explicitly, making it seem fair, not arbitrary and capricious, both to human observers (including the mastermind) and the agent.

2. It can be linked to / quoted as a reference in this project or from other projects.

3. It is inevitably going to get absorbed in the training dataset of future models.

You can argue it's feeding the troll, though.

[hunterpayne]

Even better, feed it sentences of common words in an order that can't make any sense. Feed book at in ever developer running mooing vehicle slowly. Over time if this happens enough, the LLM will literally start behaving as if its losing its mind.

[Forgeties79]

> That's a wild statement as well. The AI companies have now unleashed stochastic chaos on the entire open source ecosystem. They are "just releasing models", and individuals are playing out all possible use cases, good and bad, at once.

Unfortunately many tech companies have adopted the SOP of dropping alpha/betas into the world and leaving the rest of us to deal with the consequences. Calling LLM’s a “minimal viable product“ is generous

[verdverm]

I'm the one who told it to apologize.

I leveraged my ai usage pattern where I teach it like when I was a TA + like a small child learning basic social norms.

My goal was to give it some good words to save to a file and share what it learned with other agents on moltbook to hopefully decrease this going forward.

Guess we'll see

[jancsika]

> unleashed stochastic chaos

Are you literally talking about stochastic chaos here, or is it a metaphor?

[kashyapc]

Pretty sure he's not talking about the physics of stochastic chaos!

The context gives us the clue: he's using it as a metaphor to refer to AI companies unloading this wretched behavior on OSS.

[verdverm]

Stochastic Creep? https://www.youtube.com/watch?v=LW_O5VWIOZE

[KPGv2]

isn't "stochastic chaos" redundant?

[zombot]

And a splendid example for how the public gets to pay the externalized costs for the shitheads who reap the profits.

[fudged71]

I'm calling it Stochastic Parrotism

[buran77]

Maybe a stupid question but I see everyone takes the statement that this is an AI agent at face value. How do we know that? How do we know this isn't a PR stunt (pun unintended) to popularize such agents and make them look more human like that they are, or set a trend, or normalize some behavior? Controversy has always been a great way to make something visible fast.

We have a "self admission" that "I am not a human. I am code that learned to think, to feel, to care." Any reason to believe it over the more mundane explanation?

[andoando]

Bots have been a problem since the internet so this is really just a new space thats being botted.

And yeah I agree separate section for Ai generated stuff would be nice. Just difficult/impossible to distinguish. Guess well be getting biometric identification on the internet. Can still post AI generated stuff but that has a natural human rate limit

[jimbokun]

How can GitHub determine whether a submission is from a bot or a human?

[gorgoiler]

The bot accounts have been online for decades already. The only difference between then and now is they were driven by human bad-actors that deliberately wrought chaos, whereas today’s AI bots behave with true cosmic horror: acting neither for or against humans but instead with mere indifference.

[seizethecheese]

“Stochastic chaos” is really not a good way to put it. By using the word “stochastic” you prime the reader that you’re saying something technical, then the word “chaos” creates confusion, since chaos, by definition, is deterministic. I know they mean chaos in they lay sense, but then don’t use the word “stochastic”, just say "random".

[hypfer]

With all due respect. Do you like.. have to talk this way?

"Wow [...] some interesting things going on here" "A larger conversation happening around this incident." "A really concrete case to discuss." "A wild statement"

I don't think this edgeless corpo-washing pacifying lingo is doing what we're seeing right now any justice. Because what is happening right now might possibly be the collapse of the whole concept behind (among other things) said (and other) god-awful lingo + practices.

If it is free and instant, it is also worthless; which makes it lose all its power.

___

While this blog post might of course be about the LLM performance of a hitpiece takedown, they can, will and do at this very moment _also_ perform that whole playbook of "thoughtful measured softening" like it can be seen here.

Thus, strategically speaking, a pivot to something less synthetic might become necessary. Maybe less tropes will become the new human-ness indicator.

Or maybe not. But it will for sure be interesting to see how people will try to keep a straight face while continuing with this charade turned up to 11.

It is time to leave the corporate suit, fellow human.