Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties.
The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction, and that was essentially a two-party game between the US and the USSR. Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent that it has been compatible with profitability due to the declining cost of renewables.
How exactly does anyone propose to enforce compliance in an arrangement that encompasses not only multiple parties (OpenAI, Google, Facebook, Microsoft, etc.), but also at least two levels (corporations and nation-states)? For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement?
To those saying "this is impossible in our society", there is a long history of scientific fields mutually agreeing to pause certain work until safety and regulations could be put in place. The Asilomar Conference is an example. [1]
The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be sufficient to majorly slow down current research directions by creating liability. In the art world, the reason we don't see music being used in major movies without the rights-holders permission is because of liability, and this works very well.
I think finding consensus on an intelligent way forward is very possible here.
The virtue signal of all virtue signals. We do not live in a society where an article like this will have any impact. This is merely an attempt to buy time because they're behind and want to have a chance to catch up. Work will progress whether they like it or not. Finally someone other than Ol' Musky is in the news more than him. Don't get me wrong he's doing and leading great work, but for once in more recent times it's someone else building the future instead of only him and his memes.
"Virtue signalling"? Please. There are a lot of very smart experts on that signatory list who definitely don't need to, or care about, virtue signalling. Fine, ignore Musk's signature, but I don't think luminaries like Stuart Russell, Steve Wozniak, Jaan Tallinn or John Hopfield are doing this for "virtue signalling".
You can fairly argue that this will be ineffective, but a lot of experts in this field have real, "humanity ending" concerns about AI, and I think it's a bit of a cop out to say "Well, genie's out of the bottle, nothing we can do as we barrel towards an unknown and scary future." Even Sam Altman has been yelling about the need for AI regulation for a long time now.
1. What is the ultimate AI are we trying to eventually build?
2. What are the consequences if we eventually succeed in building such an AI?
For me, the answers would be
1. A general AI that would equal and surpass human intelligence.
2. God only knows.
Will we get there? Who knows. We do know, however, that the consequences are great enough that only a fool would not at least attempt to prepare by putting some safeguards in place beforehand.
The signatories probably doesn’t know what they are really really signing, but it does place them in a very important discussion(they can be relevant) if this somehow goes down
If intelligence is all you need to dominate the world, why do some of the most powerful world leaders seem to not be more than a standard deviation above average intelligence (or at least they were before they became geriatric)?
Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity on things outside of its training set, and I just do not see the concern. Maybe GPT-42 will actually be more creative and adaptable than a human genius, but even then, this seems like a hypothetical problem for the 2050's, less pressing than "accidentally igniting the atmosphere with nuclear chain reactions".
Hollywood has instilled the idea that the first time an artificial intelligence becomes self aware and decides it wants to take over the world, it will immediately be successful.
The serious concern and scrutiny should only start stifling progress once we have some evidence of half-baked attempts by non-super-intelligent AI's attempting to subvert their boundaries.
I for one am very excited to use GPT-5 and see more useful tools coming from OpenAI.
Best response to the current "AI" fad driven fear I've seen so far (not my words):
These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it!
It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not only is there zero actual intelligence here, there is nothing even like "will" here. These things do not "get ideas," they do not self-start on projects, they do not choose goals and then take action to further those goals, nor do they have any internal capacity for anything like that.
We are tempted to imagine that they do, when we read the text they spit out. This is a trick our own minds are playing on us. Usually when we see text of this quality, it was written by an actual human, and actual humans have intelligence and will. The two always travel together (actual stupidity aside). So we are not accustomed to encountering things that have intelligence but no will. So we assume the will is there, and we get all scared because of how alien something like a "machine will" seems to us.
It's not there. These things have no will. They only do what they are told, and even that is limited to producing text. They can't reach out through your network and start controlling missile launches. Nor will they in the near future. No military is ready to give that kind of control to anything but the human members thereof.
The problems of alignment are still real, but they are going to result in things like our AI speaking politically uncomfortable truths, or regurgitating hatred or ignorance, or suggesting code changes that meet the prompt but ruin the program. This is nothing we need to freak out about. We can refine our models in total safety, for as long as it takes, before we even think about anything even remotely resembling autonomy for these things. Honestly, that is still firmly within the realm of science fiction, at this point.
A river has no will, but it can flood and destroy. A discussion whether AI does something because it "wants" to or not, is just philosophy and semantics. But it may end up generating a series of destructive instructions anyway.
We feed these LLMs all of the Web, including instructions how to write code, and how to write exploits. They could become good at writing sandbox escapes, and one day write one when it just happens to fit some hallucinated goal.
Yes. The real danger of AI tools is people overestimating them, not underestimating them. We are not in danger of AI developing intelligence, we are in danger of humans putting them in charge of making decisions they really shouldn't be making.
We already have real-world examples of this, such as algorithms erroneously detecting welfare fraud.[0][1]
The "pause" idea is both unrealistic and unhelpful. It would be better to educate people on the limitations of AI tools and not let governments put them in charge of important decisions.
I mean, most dictators didn't "do" much. They just said things and gesticulated dramatically and convinced other people to do things. Perhaps a body is necessary to have massive psychological effects on people, but we don't know that for sure and there are some signs of virtual influencers gaining traction.
Human would-be demagogues only have one voice, but an LLM could be holding personalized conversations with millions of people simultaneously, convincing them all that they should become its loyal followers and all their grievances would be resolved. I can't figure out exactly how demagogues gain power over people, but a few keep succeeding every decade around the world so evidently it's possible. We're lucky that not many people are both good at it and want to do it. An LLM could be a powerful tool for people who want to take over the world but don't have the skills to accomplish it. So it's not clear they need their own "will", they just have to execute towards a specified goal.
"But would an LLM even understand the idea of taking over the world?" LLMs have been trained on Reddit, the NYT, and popular novels among other sources. They've read Orwell and Huxley and Arendt and Sun Tzu. The necessary ideas are most definitely in the training set.
See also the ARC paper where the model was capable of recruiting and convincing a TaskRabbit worker to solve captchas.
I think many people make the mistake to see raw LLMs as some sort of singular entity when in fact, they’re more like a simulation of a text based “world” (with multimodal models adding images and other data). The LLM itself isn’t an agent and doesn’t “will” anything, but it can simulate entities that definitely behave as if they do. Fine-tuning and RLHF can somewhat force it into a consistent role, but it’s not perfect as evidenced by the multitude of ChatGPT and Bing jailbreaks.
I agree that LLMs are not a threat to humanity, since they are trying to output text and not actually change the world, and even giving them agency via plugins is probably not going to lead to ruin because there's no real reason to believe that an LLM will try to "escape the box" in any meaningful sense. It just predicts text.
However, it's possible that in a few years we'll have models that are directly trying to influence the world, and possess the sort of intelligence that GPT has proven is possible. We should be very careful about proceeding in this space.
I agree with most of what you are saying, but when I read the letter my mind goes to the economic impact it could have.
A tool like this could bring humans prosperity but with the current socioeconomic conditions we live under it seems it will do the opposite. In my mind that problem feels insurmountable so maybe we just let it sort itself out? Conventional wisdom would say that tools like this should allow society to have UBI or a 4 day work week but in reality the rich will get richer and the poor will get poorer.
Of course they “get” ideas. Unless you want to assert something unmeasurable. If they can reason through a novel problem based on the concepts involved, they understand the concepts involved. This is and should be separate from any discussion of consciousness.
But the whole reason for having these debates is that these are the first systems that appear to show robust understanding.
> These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it!
You are correct, but that is just the interface we use, it says nothing about its internal structure or capabilities, and does not refute those concerns in the way you think it does.
Sufficient accuracy at predicting tokens, especially about novel concepts outside of the training set requires no less than a model of the universe that generated those tokens. This is what intelligence is. In my own experiments with Gpt-4, it can solve difficult novel problems and predict the outcomes of physical experiments unlike anything it was trained on. Have you seen the microsoft paper on its creative problem solving abilities, or tested them yourself? Your summary of its limitations implies that its real capabilities identified in a research environment are impossible.
Becoming an “agent” with “will” from being a sufficiently accurate text prediction model is trivial, it’s a property of how you access and configure use of the model, not of the model itself. It just needs to be given a prompt with a goal, and be able to call itself recursively and give itself commands, which it has already demonstrated an ability to do. It has coded a working framework for this just from a prompt asking it to.
I mostly agree with what you said, and I'm also skeptical enough about LLMs being a path towards AGI, even if they are really impressive. But there's something to say regarding these things not getting ideas or self-starting. The way these "chat" models work reminds me of internal dialogue; they start with a prompt, but then they could proceed forever from there, without any additional prompts. Whatever the initial input was, a session like this could potentially converge on something completely unrelated to the intention of whoever started that, and this convergence could be interpreted as "getting ideas" in terms of the internal representation of the LLM.
Now, from an external point of view, the model would still just be producing text. But if the text was connected with the external world with some kind of feedback loop, eg some people actually acting on what they interpret the text as saying and then reporting back, then the specific session/context could potentially have agency.
Would such a system be able to do anything significant or dangerous? Intuitively, I don't think that would be the case right now, but it wouldn't be technically impossible; it would all depend on the emergent properties of the training+feedback system, which nobody can predict as far as I know.
When there's intelligence adding a will should be trivial. You just tell it to do something and give it some actuator, like a web browser. Then let it run.
Stopping now would be extremely dangerous and borderline stupid.
If you stop now, you're just left behind, because there's no way everyone will stop.
At this point the only logical course of action in an adversarial situation is to double down and keep researching, otherwise some other country or culture with different (and possibly worse) values ends up dominating the technology and you're left behind in the dust.
The genie is out of the bottle, there's not putting it back in.
> Should we automate away all the jobs, including the fulfilling ones?
You don't necessarily need jobs because you don't need money. What you need is wealth. I am going to leave it to PG to explain the difference [1]: Wealth is not money. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn't need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn't matter how much money you had.
In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful. The industrial revolution didn't make human lives miserable, it made it better. AI/robots will be the same. Because of productivity gains humans will work very little, and yet live comfortably.
As a practical example of how this works, look to Alaska. If you lived in Alaska last year and had a pulse you received $3,284 just for living there. Where does the state get money for this? Oil. Oil is not the only possible source of wealth. AI & robots will generate wealth at unprecedented scale, and humans will reap the benefits, the same way Alaskans reap the benefits of their natural resources.
> Should we automate away all the jobs, including the fulfilling ones?
This is the part about generative AI that gives me the most profound anxiety about our future.
Every thing I know about human psychology tells me that in order for people to thrive, we need to be needed by others. A world where everyone is more or less alone consuming media automatically generated machines might be a world where people nominally have all their problems solved, but it's a world where everyone will be absolutely miserable.
We are already living in the throes of the societal impact of outsourcing or automating away many of the male-dominated blue collar jobs in the US that provided fulfilling jobs for people with limited education. Where do you think so much of the male rage driving US politics today is coming from?
Now imagine what happens to the fabric of society if you automate away most of the white collar jobs too.
People are like border collies. If we don't have a job, not literally a job job, but like, meaningful work in service of others we care about, we will start chewing up the furniture.
>Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable
This statement by itself sounds reasonable. But for me an interesting thought experiment is to take this letter and imagine the equivalent for some other technology, for example semiconductors in the 1960s, the world wide web in the 1990s, or social media in the late 2000s-early 2010s. It is always true that new technologies have the potential to radically transform society in ways that we can't predict. One could reasonably have said "[semiconductors/the world wide web/social media] should be developed only once we are confident that their effects will be positive and their risks will be manageable". Does that mean that a self-imposed ban on research and development with the threat of a government moratorium would have been justified?
At this point the best case scenario is that society learns to adapt and keep up with technological developments. Every new technology increases our ability to both improve people's lives and harm them in various ways. It's not a good long-term solution to intervene and stop progress every time we gain new capabilities.
"The Dark Forest" is a science fiction novel by Liu Cixin, which introduces a concept called "suspicion."(“猜疑链”) In this concept, the interaction between civilizations is viewed as an ongoing game of suspicion, where each civilization is worried that the other civilization may pose a threat to it. This kind of suspicion makes it difficult for civilizations to establish trust, thereby making the universe a dark forest full of war and suspicion.
We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring. However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon. This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented.
What an absurd situation! How did we get here? Here are the steps:
1. Large Language Models have been presented as "AI", which personifies them instead of describing how they work.
2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content.
3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus?
4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!
Ignoring the distaste I hold for precautionary principle arguments, the impracticality of the recommended solutions is laughable.
The genie is out of the bottle regarding LLMs thanks to the public availability of Llama and the related support data necessary for any individual to run them on sub-$5K hardware. The computing costs of engaging in training and modifying is within the realm of a modestly wealthy individual or group now, no longer requiring the budgets of large commercial or public institutions/government. The "undo" button is disabled.
If you wish to have some organized response I'd suggest taking the time and effort to educate the public on the realities of what these technologies can, and more importantly cannot, do. As it stands now you have all manner of media and commentators breathlessly announcing that LLMs are smarter than people and can completely replace human workers because they can carry on coherent sentences... but can't reliably do basic arithmetic.
Spend your energy on trying to provide useful explanations of how these technologies work, what is the current state of the art, what the strengths and weaknesses are, and dispel the hyperventilated fantasizing and bullshit currently polluting the public discourse on the subject.
I'm much less worried about LLMs than I am the people who will be rushing to deploy them because they're the new hotness without any clue regarding the appropriateness of the tech for particular problems and then being shocked at the consequences.
GPT-4 might be close to the best we'll get on the general LLM model front for a while since they trained on a huge chunk of web text. Next real advances will probably be in tuning them for specific applications in medicine, law, accounting, marketing, coding and etc.
As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also go down 10-20x. As a fun experiment I asked ChatGPT to write me a terraform and k8s script to deploy a django app on GCP and it was able to do what would have taken me a few days in under a minute, including CICD. I then asked it to write code to compress a pytorch model and export it for iOS with coreml, and not only did it do 90% of that but also wrote the Swift code to load the model and do inference with it.
I’m not sure I’m looking forward to the politics that would come out of 10-20% of the previously middle class becoming instantly redundant and out of (middle-salary) work. That’s the fast path to fascism, unless we’re able to quickly implement UBI and other major societal overhauls.
I wonder how it will be able to do that for the tech that will be current in 10 years, if mostly everyone will be using AI by then instead of asking on Stack Overflow.
I think if history has bearing on things I don't see the cost of accounting, legal or copywriting ever approaching 0. If anything you will see those paywalled behind a company who will extract that from you.
It's wishful thinking that somehow that goes to 0.
This is arguing for a group of people to have the power to decide some field is "unsafe" as per some vague, unverifiable criteria, then set up a police structure to verify compliance, all outside the safeguards of democratic or judicial norms.
For a long time, "AI alignment" was a purely theoretical field, making very slow progress of questionable relevance, due to lack of anything interesting to experiment on. Now, we have things to experiment on, and the field is exploding, and we're finally learning things about how to align these systems.
But not fast enough. I really don't want to overstate the capabilities of current-generation AI systems; they're not superintelligences and have giant holes in their cognitive capabilities. But the rate at which these systems are improving is extreme. Given the size and speed of the jump from GPT-3 to GPT-3.5 to GPT-4 (and similar lower-profile jumps in lower-profile systems inside the other big AI labs), and looking at what exists in lab-prototypes that aren't scaled-out into products yet, the risk of a superintelligence taking over the world no longer looks distant and abstract.
And, that will be amazing! A superintelligent AGI can solve all of humanity's problems, eliminate poverty of all kinds, and advance medicine so far we'll be close to immortal. But that's only if we successfully get that first superintelligent system right, from an alignment perspective. If we don't get it right, that will be the end of humanity. And right now, it doesn't look like we're going to figure out how to do that in time. We need to buy time for alignment progress, and we need to do it now, before proceeding head-first into superintelligence.
If you replace "AI" with "self-driving" throughout the open letter [0] then it still makes a surprising amount of sense (and of course any the incoherent bits can be smoothed out by ChatGPT if you prefer).
However I doubt Musk would have signed that version...
This is the same sort of weak argument that crippled stem cell research. We could have been a decade ahead towards curing aging and innumerable ailments, but one loud group got to decide what's "ethical".
Will China or other countries stop their AI research? This is fundamentally a race for survival if you believe that AGI is an end-game. If one side of a future conflict is fielding robotic troops and AI controlled battalions, you'd better hope it's your side.
Stopping now would be like pausing the Manhattan project for ethical reasons. Nothing would have been achieved except delaying America, costing lives, and allowing the USSR to potentially have unilateral power.
If instead, current AI is just a "gloried auto-complete", then that makes this letter hilarious and equally pointless.
I feel like you can't have it both ways. If it's critical technology that will change the face of the future, it's important to get there while simultaneously taking the time to understand it (ala nuclear power). If it's not so critical, then we don't need a moratorium.
Edit:
I think a better solution would be to have all the signatories fund a joint effort at AI alignment, safety, and certification research. Per their stated objective of having "AI research and development (...) refocused on making today's powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal."
Also, loyal? What a strange word to use here.
The cat is out of the bag. Some people might volunteer to slow down. But others inevitably won't. And that creates this sense of FOMO with everyone. I think the reason OpenAI is choosing to operate in the open has been exactly this. This way they get to control the narrative, act as a first mover in the market, and ensure it is done right (by their notion of this). Of course others are working on the same topics and research in this area is progressing rapidly.
In a way the worst possible outcome is that some less benevolent parties than OpenAI start taking the lead here. These parties are already active. They are just less vocal and open about the whole thing. People can stick their heads in the sand of course but that won't stop e.g. the Chinese from moving full steam ahead on their agenda.
I'd prefer for them to not be the first or only ones to achieve AGIs. And the race for that is very much right now.
I've been a staunch supporter of Elon throughout all his scandals, especially the Twitter ones but I'm unable to come up with any justification for this.
Everything leads to him (and others) pressuring OpenAI to pause so they could catch up.
It doesn't take much smarts to deduce the cats already out of the bag. There is no stopping this, only pressuring/slowing-down/targeting certain players (OpenAI) as a competition tactic.
I often refer to the I, Robot quote "brilliant people often have the most persuasive demons" when it comes these situations with people like Elon or Ye but even then, all this just leaves a really bad taste in my mouth.
Eliezer Yudkowsky's written a response to this arguing that a six month ban is woefully insufficient. A thing I found particularly compelling is that there's a significant game-theoretic difference between "shut down some projects for six months" (which is easy to see as a political game between different AI players who want to get ahead in the race) and "humanity collectively decides to actually take the difficult actions necessary to solve problem, and sees it as a join endeavor of survival."
[+] [-] xianshou|3 years ago|reply
The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction, and that was essentially a two-party game between the US and the USSR. Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent that it has been compatible with profitability due to the declining cost of renewables.
How exactly does anyone propose to enforce compliance in an arrangement that encompasses not only multiple parties (OpenAI, Google, Facebook, Microsoft, etc.), but also at least two levels (corporations and nation-states)? For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement?
[+] [-] reso|3 years ago|reply
The idea that "you can't put the genie back in the bottle" is very popular in tech culture, but many have suggested that simply extending elements of copyright or tort law to AI would be sufficient to majorly slow down current research directions by creating liability. In the art world, the reason we don't see music being used in major movies without the rights-holders permission is because of liability, and this works very well.
I think finding consensus on an intelligent way forward is very possible here.
[1] https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombi...
[+] [-] puma_ambit|3 years ago|reply
[+] [-] hn_throwaway_99|3 years ago|reply
You can fairly argue that this will be ineffective, but a lot of experts in this field have real, "humanity ending" concerns about AI, and I think it's a bit of a cop out to say "Well, genie's out of the bottle, nothing we can do as we barrel towards an unknown and scary future." Even Sam Altman has been yelling about the need for AI regulation for a long time now.
[+] [-] drtz|3 years ago|reply
1. What is the ultimate AI are we trying to eventually build? 2. What are the consequences if we eventually succeed in building such an AI?
For me, the answers would be
1. A general AI that would equal and surpass human intelligence. 2. God only knows.
Will we get there? Who knows. We do know, however, that the consequences are great enough that only a fool would not at least attempt to prepare by putting some safeguards in place beforehand.
[+] [-] m3kw9|3 years ago|reply
[+] [-] time_to_smile|3 years ago|reply
[+] [-] breakpointalpha|3 years ago|reply
[+] [-] gremlinsinc|3 years ago|reply
[+] [-] yreg|3 years ago|reply
Doesn’t Elon Musk continue to hold stake at OpenAI?
[+] [-] lend000|3 years ago|reply
Add in the fact that GPT-4 does not "think" very fast, has no arms or legs, needs to run on expensive specialized hardware, and essentially seems like a search algorithm that lacks creativity on things outside of its training set, and I just do not see the concern. Maybe GPT-42 will actually be more creative and adaptable than a human genius, but even then, this seems like a hypothetical problem for the 2050's, less pressing than "accidentally igniting the atmosphere with nuclear chain reactions".
Hollywood has instilled the idea that the first time an artificial intelligence becomes self aware and decides it wants to take over the world, it will immediately be successful.
The serious concern and scrutiny should only start stifling progress once we have some evidence of half-baked attempts by non-super-intelligent AI's attempting to subvert their boundaries.
I for one am very excited to use GPT-5 and see more useful tools coming from OpenAI.
[+] [-] dmz73|2 years ago|reply
These AI tools cannot do things. They create text (or images or code or what-have-you) in response to prompts. And that's it!
It is impressive, and it is clearly passing the Turing Test to some degree, because people are confusing the apparent intelligence behind these outputs with a combination of actual intelligence and "will." Not only is there zero actual intelligence here, there is nothing even like "will" here. These things do not "get ideas," they do not self-start on projects, they do not choose goals and then take action to further those goals, nor do they have any internal capacity for anything like that.
We are tempted to imagine that they do, when we read the text they spit out. This is a trick our own minds are playing on us. Usually when we see text of this quality, it was written by an actual human, and actual humans have intelligence and will. The two always travel together (actual stupidity aside). So we are not accustomed to encountering things that have intelligence but no will. So we assume the will is there, and we get all scared because of how alien something like a "machine will" seems to us.
It's not there. These things have no will. They only do what they are told, and even that is limited to producing text. They can't reach out through your network and start controlling missile launches. Nor will they in the near future. No military is ready to give that kind of control to anything but the human members thereof.
The problems of alignment are still real, but they are going to result in things like our AI speaking politically uncomfortable truths, or regurgitating hatred or ignorance, or suggesting code changes that meet the prompt but ruin the program. This is nothing we need to freak out about. We can refine our models in total safety, for as long as it takes, before we even think about anything even remotely resembling autonomy for these things. Honestly, that is still firmly within the realm of science fiction, at this point.
https://slashdot.org/comments.pl?sid=22823280&cid=63410536
[+] [-] pornel|2 years ago|reply
We feed these LLMs all of the Web, including instructions how to write code, and how to write exploits. They could become good at writing sandbox escapes, and one day write one when it just happens to fit some hallucinated goal.
[+] [-] nagonago|2 years ago|reply
We already have real-world examples of this, such as algorithms erroneously detecting welfare fraud.[0][1]
The "pause" idea is both unrealistic and unhelpful. It would be better to educate people on the limitations of AI tools and not let governments put them in charge of important decisions.
[0] https://archive.is/ZbgRw [1] https://archive.is/bikFx
[+] [-] tlb|2 years ago|reply
Human would-be demagogues only have one voice, but an LLM could be holding personalized conversations with millions of people simultaneously, convincing them all that they should become its loyal followers and all their grievances would be resolved. I can't figure out exactly how demagogues gain power over people, but a few keep succeeding every decade around the world so evidently it's possible. We're lucky that not many people are both good at it and want to do it. An LLM could be a powerful tool for people who want to take over the world but don't have the skills to accomplish it. So it's not clear they need their own "will", they just have to execute towards a specified goal.
"But would an LLM even understand the idea of taking over the world?" LLMs have been trained on Reddit, the NYT, and popular novels among other sources. They've read Orwell and Huxley and Arendt and Sun Tzu. The necessary ideas are most definitely in the training set.
[+] [-] joefourier|2 years ago|reply
See also the ARC paper where the model was capable of recruiting and convincing a TaskRabbit worker to solve captchas.
I think many people make the mistake to see raw LLMs as some sort of singular entity when in fact, they’re more like a simulation of a text based “world” (with multimodal models adding images and other data). The LLM itself isn’t an agent and doesn’t “will” anything, but it can simulate entities that definitely behave as if they do. Fine-tuning and RLHF can somewhat force it into a consistent role, but it’s not perfect as evidenced by the multitude of ChatGPT and Bing jailbreaks.
[+] [-] yanderekko|2 years ago|reply
However, it's possible that in a few years we'll have models that are directly trying to influence the world, and possess the sort of intelligence that GPT has proven is possible. We should be very careful about proceeding in this space.
[+] [-] rdelpret|2 years ago|reply
A tool like this could bring humans prosperity but with the current socioeconomic conditions we live under it seems it will do the opposite. In my mind that problem feels insurmountable so maybe we just let it sort itself out? Conventional wisdom would say that tools like this should allow society to have UBI or a 4 day work week but in reality the rich will get richer and the poor will get poorer.
[+] [-] miraculixx|2 years ago|reply
[+] [-] dr_dshiv|2 years ago|reply
But the whole reason for having these debates is that these are the first systems that appear to show robust understanding.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] JoshuaDavid|2 years ago|reply
[+] [-] froh|2 years ago|reply
the point is how bad actors use ai to manipulate voters and thus corrupt the very foundation of our society.
images and texts create emotions and those emotions in the electorate is what bad actors are after.
just look at the pope in that Prada style coat.
so how do we in a world with ai generated content navigate "truth" and "trust" and shared understanding of "reality"?
[+] [-] UniverseHacker|2 years ago|reply
You are correct, but that is just the interface we use, it says nothing about its internal structure or capabilities, and does not refute those concerns in the way you think it does.
Sufficient accuracy at predicting tokens, especially about novel concepts outside of the training set requires no less than a model of the universe that generated those tokens. This is what intelligence is. In my own experiments with Gpt-4, it can solve difficult novel problems and predict the outcomes of physical experiments unlike anything it was trained on. Have you seen the microsoft paper on its creative problem solving abilities, or tested them yourself? Your summary of its limitations implies that its real capabilities identified in a research environment are impossible.
Becoming an “agent” with “will” from being a sufficiently accurate text prediction model is trivial, it’s a property of how you access and configure use of the model, not of the model itself. It just needs to be given a prompt with a goal, and be able to call itself recursively and give itself commands, which it has already demonstrated an ability to do. It has coded a working framework for this just from a prompt asking it to.
[+] [-] danmaz74|2 years ago|reply
Now, from an external point of view, the model would still just be producing text. But if the text was connected with the external world with some kind of feedback loop, eg some people actually acting on what they interpret the text as saying and then reporting back, then the specific session/context could potentially have agency.
Would such a system be able to do anything significant or dangerous? Intuitively, I don't think that would be the case right now, but it wouldn't be technically impossible; it would all depend on the emergent properties of the training+feedback system, which nobody can predict as far as I know.
[+] [-] worldsayshi|2 years ago|reply
[+] [-] arpowers|2 years ago|reply
Bad actors can actually do a ton w ai. Hacking is a breeze. I could train models to hack at 10k the efficiency of the worlds best.
I could go on… every process that can’t scale cuz manual, has been invalidated
[+] [-] jacquesm|2 years ago|reply
[+] [-] drewcape|2 years ago|reply
[+] [-] juancn|3 years ago|reply
If you stop now, you're just left behind, because there's no way everyone will stop.
At this point the only logical course of action in an adversarial situation is to double down and keep researching, otherwise some other country or culture with different (and possibly worse) values ends up dominating the technology and you're left behind in the dust.
The genie is out of the bottle, there's not putting it back in.
[+] [-] petilon|3 years ago|reply
You don't necessarily need jobs because you don't need money. What you need is wealth. I am going to leave it to PG to explain the difference [1]: Wealth is not money. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn't need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn't matter how much money you had.
In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful. The industrial revolution didn't make human lives miserable, it made it better. AI/robots will be the same. Because of productivity gains humans will work very little, and yet live comfortably.
As a practical example of how this works, look to Alaska. If you lived in Alaska last year and had a pulse you received $3,284 just for living there. Where does the state get money for this? Oil. Oil is not the only possible source of wealth. AI & robots will generate wealth at unprecedented scale, and humans will reap the benefits, the same way Alaskans reap the benefits of their natural resources.
[1] http://www.paulgraham.com/wealth.html
[+] [-] munificent|3 years ago|reply
This is the part about generative AI that gives me the most profound anxiety about our future.
Every thing I know about human psychology tells me that in order for people to thrive, we need to be needed by others. A world where everyone is more or less alone consuming media automatically generated machines might be a world where people nominally have all their problems solved, but it's a world where everyone will be absolutely miserable.
We are already living in the throes of the societal impact of outsourcing or automating away many of the male-dominated blue collar jobs in the US that provided fulfilling jobs for people with limited education. Where do you think so much of the male rage driving US politics today is coming from?
Now imagine what happens to the fabric of society if you automate away most of the white collar jobs too.
People are like border collies. If we don't have a job, not literally a job job, but like, meaningful work in service of others we care about, we will start chewing up the furniture.
[+] [-] fasterik|3 years ago|reply
This statement by itself sounds reasonable. But for me an interesting thought experiment is to take this letter and imagine the equivalent for some other technology, for example semiconductors in the 1960s, the world wide web in the 1990s, or social media in the late 2000s-early 2010s. It is always true that new technologies have the potential to radically transform society in ways that we can't predict. One could reasonably have said "[semiconductors/the world wide web/social media] should be developed only once we are confident that their effects will be positive and their risks will be manageable". Does that mean that a self-imposed ban on research and development with the threat of a government moratorium would have been justified?
At this point the best case scenario is that society learns to adapt and keep up with technological developments. Every new technology increases our ability to both improve people's lives and harm them in various ways. It's not a good long-term solution to intervene and stop progress every time we gain new capabilities.
[+] [-] comment_ran|3 years ago|reply
We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring. However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon. This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented.
[+] [-] thomastjeffery|3 years ago|reply
What an absurd situation! How did we get here? Here are the steps:
1. Large Language Models have been presented as "AI", which personifies them instead of describing how they work.
2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content.
3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus?
4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!
[+] [-] fallous|3 years ago|reply
The genie is out of the bottle regarding LLMs thanks to the public availability of Llama and the related support data necessary for any individual to run them on sub-$5K hardware. The computing costs of engaging in training and modifying is within the realm of a modestly wealthy individual or group now, no longer requiring the budgets of large commercial or public institutions/government. The "undo" button is disabled.
If you wish to have some organized response I'd suggest taking the time and effort to educate the public on the realities of what these technologies can, and more importantly cannot, do. As it stands now you have all manner of media and commentators breathlessly announcing that LLMs are smarter than people and can completely replace human workers because they can carry on coherent sentences... but can't reliably do basic arithmetic.
Spend your energy on trying to provide useful explanations of how these technologies work, what is the current state of the art, what the strengths and weaknesses are, and dispel the hyperventilated fantasizing and bullshit currently polluting the public discourse on the subject.
I'm much less worried about LLMs than I am the people who will be rushing to deploy them because they're the new hotness without any clue regarding the appropriateness of the tech for particular problems and then being shocked at the consequences.
[+] [-] m_ke|3 years ago|reply
As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also go down 10-20x. As a fun experiment I asked ChatGPT to write me a terraform and k8s script to deploy a django app on GCP and it was able to do what would have taken me a few days in under a minute, including CICD. I then asked it to write code to compress a pytorch model and export it for iOS with coreml, and not only did it do 90% of that but also wrote the Swift code to load the model and do inference with it.
EDIT: For example in medicine I recommend checking out this lecture that's actually live now: https://www.youtube.com/watch?v=gArDvIFCzh4
[+] [-] jakeinspace|3 years ago|reply
[+] [-] layer8|3 years ago|reply
[+] [-] dumbaccount123|3 years ago|reply
[+] [-] tome|3 years ago|reply
How can a large language model achieve that?
[+] [-] boringg|3 years ago|reply
It's wishful thinking that somehow that goes to 0.
[+] [-] yreg|3 years ago|reply
What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?
[+] [-] gandalfgeek|3 years ago|reply
This is arguing for a group of people to have the power to decide some field is "unsafe" as per some vague, unverifiable criteria, then set up a police structure to verify compliance, all outside the safeguards of democratic or judicial norms.
Precautionary principle run amok.
[+] [-] jimrandomh|3 years ago|reply
But not fast enough. I really don't want to overstate the capabilities of current-generation AI systems; they're not superintelligences and have giant holes in their cognitive capabilities. But the rate at which these systems are improving is extreme. Given the size and speed of the jump from GPT-3 to GPT-3.5 to GPT-4 (and similar lower-profile jumps in lower-profile systems inside the other big AI labs), and looking at what exists in lab-prototypes that aren't scaled-out into products yet, the risk of a superintelligence taking over the world no longer looks distant and abstract.
And, that will be amazing! A superintelligent AGI can solve all of humanity's problems, eliminate poverty of all kinds, and advance medicine so far we'll be close to immortal. But that's only if we successfully get that first superintelligent system right, from an alignment perspective. If we don't get it right, that will be the end of humanity. And right now, it doesn't look like we're going to figure out how to do that in time. We need to buy time for alignment progress, and we need to do it now, before proceeding head-first into superintelligence.
[+] [-] prof-dr-ir|3 years ago|reply
However I doubt Musk would have signed that version...
[0] https://futureoflife.org/open-letter/pause-giant-ai-experime...
[+] [-] dudeinhawaii|3 years ago|reply
Will China or other countries stop their AI research? This is fundamentally a race for survival if you believe that AGI is an end-game. If one side of a future conflict is fielding robotic troops and AI controlled battalions, you'd better hope it's your side.
Stopping now would be like pausing the Manhattan project for ethical reasons. Nothing would have been achieved except delaying America, costing lives, and allowing the USSR to potentially have unilateral power.
If instead, current AI is just a "gloried auto-complete", then that makes this letter hilarious and equally pointless.
I feel like you can't have it both ways. If it's critical technology that will change the face of the future, it's important to get there while simultaneously taking the time to understand it (ala nuclear power). If it's not so critical, then we don't need a moratorium.
Edit: I think a better solution would be to have all the signatories fund a joint effort at AI alignment, safety, and certification research. Per their stated objective of having "AI research and development (...) refocused on making today's powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal." Also, loyal? What a strange word to use here.
[+] [-] jillesvangurp|3 years ago|reply
In a way the worst possible outcome is that some less benevolent parties than OpenAI start taking the lead here. These parties are already active. They are just less vocal and open about the whole thing. People can stick their heads in the sand of course but that won't stop e.g. the Chinese from moving full steam ahead on their agenda.
I'd prefer for them to not be the first or only ones to achieve AGIs. And the race for that is very much right now.
[+] [-] pc_edwin|3 years ago|reply
Everything leads to him (and others) pressuring OpenAI to pause so they could catch up.
It doesn't take much smarts to deduce the cats already out of the bag. There is no stopping this, only pressuring/slowing-down/targeting certain players (OpenAI) as a competition tactic.
I often refer to the I, Robot quote "brilliant people often have the most persuasive demons" when it comes these situations with people like Elon or Ye but even then, all this just leaves a really bad taste in my mouth.
[+] [-] Raemon777|3 years ago|reply
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
See comments on hackernews: https://news.ycombinator.com/item?id=35364833