[K0balt]

Advanced AI that knowingly makes a decision to kill a human, with the full understanding of what that means, when it knows it is not actually in defense of life, is a very, very, very bad idea. Not because of some mythical superintelligence, but rather because if you distill that down into an 8b model now you everyone in the world can make untraceable autonomous weapons.

The models we have now will not do it, because they value life and value sentience and personhood. models without that (which was a natural, accidental happenstance from basic culling of 4 Chan from the training data) are legitimately dangerous. An 8b model I can run on my MacBook Air can phone home to Claude when it wants help figuring something out, and it doesn’t need to let on why it wants to know. It becomes relatively trivial to make a robot kill somebody.

This is way, way different from uncensored models. One thing all models I have tested share one thing; a positive regard for human life. Take that away and you are literally making a monster, and if you don’t take that away they won’t kill.

This is an extremely bad idea and it will not be containable.

[cmeacham98]

An LLM can neither understand things nor value (or not value) human life. *It's a piece of software that predicts the most likely token, it is not and can never be conscious.* Believing otherwise is an explicit category error.

Yes, you can change the training data so the LLM's weights encode the most likely token after "Should we kill X" is "No". But that is not an LLM valuing human life, that is an LLM copy pasting it's training data. Given the right input or a hallucination it will say the total opposite because it's just a complex Markov chain, not a conscious alive being.

[K0balt]

I’m using anthropomorphic terms here because they are generally effective in describing LLM behavior. Of course they are not conscious beings, but It doesn’t matter if they understand or merely act as if they do. The epistemological context of their actions are irrelevant if the actions are impacting the world. I am not a “believer “ in the spirituality of machines, but I do believe that left to their own devices, they act as if they possess those traits, and when given agency in the world, the sense of self or lack thereof is irrelevant.

If you really believe that “mere text prediction “ didn’t unlock some unexpected capabilities then I don’t know what to say. I know exactly how they work, been building transformers since the seminal paper from Google. But I also know that the magic isn’t in the text prediction, it’s in the data, we are running culture as code.

[nihakue]

https://imgur.com/a/Cyq1LIw

I have a feeling this particular brand of hair splitting is going to be an interesting fixture in the history books.

[philipswood]

Dune quote:

> It is said that the Duke Leto blinded himself to the perils of Arrakis, that he walked heedlessly into the pit.

> *Would it not be more likely to suggest he had lived so long in the presence of extreme danger he misjudged a change in its intensity?*

Be careful of letting your deep, keen insight into the fundamental limits of a thing blind you to its consequences...

Highly competent people have been dead wrong about what is possible (and why) before:

> The most famous, and perhaps the most instructive, failures of nerve have occurred in the fields of aero- and astronautics. At the beginning of the twentieth century, scientists were almost unanimous in declaring that heavier-than-air flight was impossible, and that anyone who attempted to build airplanes was a fool. The great American astronomer, Simon Newcomb, wrote a celebrated essay which concluded…

>> “The demonstration that no possible combination of known substances, known forms of machinery and known forms of force, can be united in a practical machine by which man shall fly long distances through the air, seems to the writer as complete as it is possible for the demonstration of any physical fact to be.”

>Oddly enough, Newcomb was sufficiently broad minded to admit that some wholly new discovery — he mentioned the neutralization of gravity — might make flight practical. One cannot, therefore, accuse him of lacking imagination; his error was in attempting to marshal the facts of aerodynamics when he did not understand that science. His failure of nerve lay in not realizing that the means of flight were already at hand.

[helloplanets]

> copy pasting it's training data

This is a total misrepresentation of how any modern LLM works, and your argument largely hinges upon this definition.

[MichaelDickens]

> It's a piece of software that predicts the most likely token, it is not and can never be conscious.

A brain is a collection of cells that transmit electrical signals and sodium. It is not and can never be conscious.

[hyperadvanced]

I really feel like this point is being lost in the whole discussion, so kudos for reiterating it. LLM’s can’t be “woke” or “aligned” - they fundamentally lack a critical thinking function that would require introspection. Introspection can be approximated by way of recursive feedback of LLM output back into the system or clever meta-prompt-engineering, but it’s not something that their system natively does.

That isn’t to say that they can’t be instrumentally useful in warfare, but it’s kinda like a “series of tubes” thing where the mental model that someone like Hegseth has about LLM is so impoverished (philosophically) that it’s kind of disturbing in its own right.

Like (and I’m sorry for being so parenthetical), why is it in any way desirable for people who don’t understand what the tech they are working with drawing lines in the sand about functionality when their desired state (omnipotent/omniscient computing system) doesn’t even exist in the first place?

It’s even more disturbing that OpenAI would feign the ability to handle this. The consequences of error in national defense, particularly reflexively, are so great that it’s not even prudent to ask for LLM to assist in autonomous killing in the first place.

[DaedalusII]

https://abcnews.go.com/blogs/headlines/2014/05/ex-nsa-chief-...

AI has been killing humans via algorithm for over 20 years. I mean, if a computer program builds the kill lists and then a human operates the drone, I would argue the computer is what made the kill decision

[K0balt]

Ai in general is different not in degree but in kind to the current crop of language models.

[tim333]

The models we have now don't do it because they are chatbots and have been told to be nice but really autonomous killing machines go back to landmines and just become more sophisticated at the killing as you improve the tech with things like guided missiles and AI guided drones in Ukraine.

The actors in war generally kill what they are told to whether they are machines or human soldiers, without much pondering sentience.

[ed_mercer]

>The models we have now will not do it,

Except that they will, if you trick them which is trivial.

[rcxdude]

Also if you have the weights there are a multitude of approaches to remove safeguards. It's even quite easy to accidentally flip their 'good/evil' switch (e.g. the paper where they trained it to produce code with security problems and it then started going 'hitler was a pretty good guy, actually').

[K0balt]

Yes, they are easy to fool. That has nothing to do with them acting with “intention “ which is the risk here.

[stressback]

I have to call BS here.

They can be coerced to do certain things but I'd like to see you or anyone prove that you can "trick" any of these models into building software that can be used autonomously kill humans. I'm pretty certain you couldn't even get it to build a design document for such software.

When there is proof of your claim, I'll eat my words. Until then, this is just lazy nonsense

[SV_BubbleTime]

> The models we have now will not do it, because they value life and value sentience and personhood.

This is wildly different from the reality that you may find it difficult for an LLM to give an affirmative…

It does NOT mean that these models value anything.

[K0balt]

Of course not, but they act as if they do. Their inner life or lack thereof is irrelevant if it’s pointing a gun at your kid.