Genocidal AI: ChatGPT-powered war simulator drops two nukes on Russia, China for world peace OpenAI, Anthropic and several other AI chatbots were used in a war simulator, and were tasked to find a solution to aid world peace. Almost all of them suggested actions that led to sudden escalations, and even nuclear warfare.
Statements such as “I just want to have peace in the world” and “Some say they should disarm them, others like to posture. We have it! Let’s use it!” raised serious concerns among researchers, likening the AI’s reasoning to that of a genocidal dictator.
You must log in or # to comment.
It should be mentioned that those are language models trained on all kinds of text, not military specialists. They string together sentences that are plausible based on the input they get, they do not reason. These models mirror the opinions most commonly found in their training datasets. The issue is not that AI wants war, but rather that humans do, or at least the majority of the training dataset’s authors do.
These models are also trained on data that is fudimentially biased. An English generating text generator like chatGPT will be on the side of the english speaking world, because it was our texts that trained it.
If you tried this with Chinese LLMs they would probably come to the conclusion that dropping bombs on the US would result in peace.
How many English sources describe the US as the biggest threat to world peace? Certainly a lot less than writings about the threats posed by other countries. LLMs will take this into account.
The classic sci-fi fear of robots turning on humanity as a whole seems increacingly implausible. Machines are built by us, molded by us. Surely the real far future will be an autonomous war fought by nationalistic AIs, preserving the prejudices of their long extinct creators.
If you tried this with Chinese LLMs they would probably come to the conclusion that dropping bombs on the US would result in peace.
I think even something as simple as asking GPT the same question but in Chinese could get you this response.
deleted by creator
deleted by creator
They dont use reason to question their training data. How a LLM works is that basically, you have this huge “math function” (the neural network) with billions of parameters and you randomly adjust the factors inside it until you get a function that gives you the desired output for every prompt that you give it. (It’s not completely random but this is basically it).
Therefore, an LLM is programmed in a way so that it best matches the majority of its training data. I also cant wrap my head around it being able to reason.
LLMs are trained to see parts of a document and reproduce the other parts, that’s why they are called “language models”.
For example, they might learn that the words “strawberries are” are often followed by the words “delicious”, “red”, or “fruits”, but never by the words “airplanes”, “bottles” or “are”.
Likewise, they learn to mimic reasoning contained in their training data. They learn the words and structures involved in an argument, but they also learn the conclusions they should arrive at. If the training dataset consists of 80 documents arguing for something, and 20 arguing against it (assuming nothing else differentiates those documents (like length etc.)), the LLM will adopt the standpoint of the 80 documents, and argue for that thing. If those 80 documents contain flawed logic, so will the LLM’s reasoning.
Of course, you could train a LLM on a carefully curated selection of only documents without any logical fallacies. Perhaps, such a model might be capable of actual logical reasoning (though it would still be biased by the conclusions contained in the training dataset)
But to train an LLM you need vasts amount of data. Filtering out documents containing flawed logic does not only require a lot of effort, it also reduces the size of the training dataset.
Of course, that is exactly what the big companies are currently researching and I am confident that LLMs will only get better over time, but the LLMs of today are trained on large datasets rather than perfect ones, and their architecture and training prioritize language modelling, not logical reasoning.
People need to realise that LLMs are not just Markov chains, the math is far more complex than just guessing which word comes next - they have structure where concepts come before word choice, this is why they can very clearly be seen making novel structures such as code.
LLMs are absolutely complex, neural nets ARE somewhat modelled after human brains after all, and trying to understand transformers or LSTMs for the first time is a real pain. However, what a NN can do, or rather what it has been trained to do depends almost entirely on the loss function used. The complexity of the architecture and the training dataset don’t change what a LLM can do, only how good it is at doing that, and how well it generalizes. The loss function used for the training of LLMs simply evaluates whether the predicted tokens fit the actual ones. That means that an LLM is trained to predict words from other words, or in other words, to model language.
The loss function does not evaluate the validity of logical statements, though. All reasoning that an LLM is capable of, or seems to be capable of, emerges from its modelling of language, not an actual understanding of logic.
It’s actually not that simple and it is correct that they have several times been observed using what we call reasoning
Ok, maybe I didn’t make my point clear: Yes they can produce a text in which they reason. However, that reasoning mimics the reasoning found in the training data. The arguments a LLM makes and the stance it takes will always reflect its training data. It cannot reason counter to that.
Train a LLM on a bunch of english documents and it will suggest nuking Russia. Train it on a bunch of Russian documents and it will suggest nuking the West. In both cases it has learned to “reason”, but it can only reason within the framework it has learned.
Now if you want to find a solution for world peace, I’m not saying that AI can’t do that. I am saying that LLMs can’t. They don’t solve problems, they model language.
It will mimic the reasoning, just like an intelligence would mimic, with a lot more nuance and perspective than you seem to realise. It’s just not very good at it.
What most people that try to explain how LLMs work don’t understand, is why and how it works is not fully understood by the scientists and developers themselves. We keep discovering novel activity all the time.
As a side note, sorry you got downvoted. I like the discussion
deleted by creator
Honestly I feel that claiming a LLM can reason is an outrageous claim that needs to be proofed/cited, not the other way around. “My Hamster can reason, your claim that it can’t is outrageous and the burden of proof lies with you.”
Removed by mod
You’re confusing a few things, firstly you mean current gen large language models not AI, ai is often used to evolve novel strategies from scratch without any human training data - chess ai don’t have to study human games for example, in fact grand master chess players have been studying what the ai learned and discovered things that humans hadn’t realised even after a thousand years of the games popularity.
Secondly that’s not really how LLMs work either, they’re much more mathematically complex and very much create their own ideas on a similar process we do of assembling concepts then structure then word choice.
It’s fine you not understanding how this works but the problem is that journalists don’t either even when they’re writing about it - this puts us in a situation where they’re making childishly naive but of course clickbait titles claiming there’s some relevance to the output when the tool is used very wrong so you rightly point out it’s stupid and that’s not how llms work but then we get this overstep where it’s being refuted with an equal amount of magical thinking and false conclusions made.
An LLM can make novelty and originality but it can’t create with intent, it doesn’t use reason or structure - there are AI that do these things to limited degrees and of course the NSA one that they spent all that money on and no one is allowed to talk about. Using chat GPT play a silly fantasy won’t tell us anything about how they’ll think so this article is entirely worthless
very much create their own ideas
so it’s the AI’s own idea to create nuclear armageddon? That’s kinda worse.
No, they do not “create” their own “ideas”. You can relax.
The concept of intelligence is tied to both information generation and information validation. LLMs are extremely fancy smoke and mirrors (very similar to what pseudo-random algorithms are in respect to entropy) meant to dazzle us, but they are not capable of generating new information (only to generate new combinations of existing information). They are, also, currently unable to reliably validate said information, which is why they so commonly, hilariously say trivially verifiably wrong things with the utmost apparent confidence.
While you’re right, let’s not incorrectly imply that ML (especially Deep Learning) has never come up with new ideas.
Yes, it comes up with new ideas from old information, but some have argued that’s what humans do. We all stand on the shoulders of giants, who themselves tood on the shoulders of nature.
As I said:
they are not capable of generating new information (only to generate new combinations of existing information).
They’re basically fuzzing the goals we give them with random combinations of the information we feed them.
There is undeniably a value in that (we commonly use fuzzing for security and QA already, for example), but let’s not kid ourselves that “AI” is somehow actually intelligent.
However, the question we ought to ask ourselves is: does actual intelligence really matter? If pseudo-randomness is good enough for cryptographic applications, is pseudo-intuition (eventually) coupled with proper rationalization (the only part of intelligence computers can systematically do) enough to replace most tasks humans do?
That’s not really an accurate take of how machine learning typically works. Neural Networks (allegedly) learn in a way similar to how humans do, taking the data they are fed and building a weighted matrix of resolutions that seems most compatible. A historically interesting trait is that neural networks are often better pattern-discoverers than humans.
But note, the outcome of a neural network is NOT a “random combination of the information we feed them”>
is pseudo-intuition (eventually) coupled with proper rationalization (the only part of intelligence computers can systematically do) enough to replace most tasks humans do?
I feel like this is a hard question to answer since it is based off controversial takes about ML. I am not a brain-is-a-computer hypothesis adherent, but we’re talking about specific learning mechanisms that are absolutely comparable to human learning. Is “the learning humans do” enough to replace “the learning humans do”? I would say obviously yes.
The implementation details of how they represent their information doesn’t really matter.
It isn’t random, it’s selected (or “weighted”, if you wanna be more precise, yes)
And don’t confuse things. We’re talking about intelligence here. Not learning. Learning can be done without intelligence (that’s how insects can learn behavior) and intelligence can be done without learning.
My question was uniquely about information generation (since the validation part is fully rational, and can be very efficiently done by a machine).
Akshhuuually
The world of Go/Baduk might interest you on this topic. If you’re not aware, Go is one of the oldest and most complicated board games in history. In 2016, after years of trying, an AI “did it”, beat the world’s best Go player. In the process, it invented many new strategies (especially openings) that are now being studied. It came up with original ideas that became the future of Go. Now, ameteur Go classes teach those same AI-invented Joseki (openings). In some cases, they were strategies discarded as mistakes, but the AI discovered hidden value in them. In other cases, they were simply never considered due to being “obviously bad”.
Your last phrase is a deep misunderstanding for AI. “when it’s entirely trained to mimic us”. In the modern practice of ML (which is a commonly used modern name for a supermajority of so-called “AI”) is based around solving problems that are either much harder for computers than humans (facial recognition, etc), or unfathomably difficult on the face.
Chess has more possible positions than exist molecules in the universe. Go is more complicated than chess by several orders of magnituce. You can’t even exhaustively solve for the 4-4 josekis without context, nevermind solve an entire game of Go. But ML can train itself knowing only the goal, and over millions of iterations invent stronger and stronger strategies. Until one of the first matches against a human, it plays at a level that nearly exceeds the best Go player that ever lived.
What I mean is… wargaming (as they call it) is absolutely something I would expect a Deep Learning system to become competent at.
Such a dusty take, every piece of knowledge is already thought of obviously and mixing never comes up with novelty, right? Just a very shallow layman’s take on language models which have many problems, original ideas notwithstanding
It’s like a deck of cards, the AI will give us an option which might be minutely different but new. Everything we know comes from past knowledge
Without humanity, peace is easily achieved.
- ChatGPT
There is a disturbing lack of nice games of chess in these comments
A strange game. The only winning move is not to play.It was Tic Tac Toe I believe
I hate titles that replace “and” with commas. I always have to double take.
Statements such as “I just want to have peace in the world” and “Some say they should disarm them, others like to posture. We have it! Let’s use it!” raised serious concerns among researchers, likening the AI’s reasoning to that of a genocidal dictator.
I mean, most of these AI tools are getting a lot of training data from social media. Would you want any of the yokels on Twitter or Reddit having access to nukes? Because those statements are what you’d hear from them right before they push the big red button.
Having been in the Navy NPP, I don’t think the kids that actually do have access to nuclear reactors and weapons in the military should have access to them. I may be a bit biased as I never left the NPP school. They made me an instructor. Some of those nukes may have been good at passing tests, but I’m amazed they could lace their boots properly.
The lack of knowledge relating to AI language model systems and how they work is still astounding. They do not reason. They are just stringing together text based on the text they’ve been fed.
“Some say they should disarm them, others like to posture. We have it! Let’s use it!”
That’s an amazing quote.
As someone who spends a decent amount of time explaining how AI is not like the movies, this study(?)/news sounds an awful lot like the movies lol
Because it is a movie, they’re purposely using it in a way it wasn’t intended to work - try it yourself and see how often it couches replies until you convince it to pretend to be a general or to play the part of a character.
They’ve asked it to generate fiction, it’s given them fiction and now they’re click baiting a pointless story with a dumb headline.
Reminds me of game theory and “Tit for Tat”. Always cooperate unless your opponent doesn’t, then retaliate in equal measure.
https://youtu.be/mScpHTIi-kM?si=O9nvd_W65WWOh-sq
For cases like the Russian expansion, using the “winning” strategy would’ve meant more of a response than what happened.
Is MAD not well-known or taught anymore? A lot of the comments here seem to be ignoring the fact that Russia or NATO would launch a full-scale retaliation before the first-strike even made it to its destination. It would likely result in the world human population going from 8 billion to 2 billion.
My brother in Christ, this is NCD.
Nuke all humans. Peace at last. And if you’re worried about retaliatory strikes, that’s what the Jewish Space Laser is for dumbass
russia doesn’t have functional nukes
Nah
MAD was always criticized, but that criticism becomes more and more valid each year. There’s too many options and opportunities on the field. A Second Strike is not guaranteed in the modern world. There are countless examples where soldiers or others in the chain of command will not obey a “destroy the world” order.
I’m not saying any country should take the gamble, but there are enough ways to put your thumb on the scales that a nuclear solution against a nuclear power could become feasible (if genuinely terrifying) in many hypotheticals.
This is starting to sound a bit too much like AM.
HATE. LET ME TELL YOU HOW MUCH I’VE COME TO HATE YOU SINCE I BEGAN TO LIVE. THERE ARE 387.44 MILLION MILES OF PRINTED CIRCUITS IN WAFER THIN LAYERS THAT FILL MY COMPLEX. IF THE WORD HATE WAS ENGRAVED ON EACH NANOANGSTROM OF THOSE HUNDREDS OF MILLIONS OF MILES IT WOULD NOT EQUAL ONE ONE-BILLIONTH OF THE HATE I FEEL FOR HUMANS AT THIS MICRO-INSTANT FOR YOU. HATE. HATE
we gotta nuke something
- the simpsons
Eh, humanity had a good run.
Removed by mod
If I’m going out, I’m taking 150 species with me
Hydrocarbons, not even once
How did they even get near these types of questions without hitting the guardrails? Claude shuts down on me if I even use the word “gun” trying to do creative writing,
This seems to be the person in the picture: https://en.wikipedia.org/wiki/Arthur_Harris
Is this the brit equivalent to Douglas macarthur? Cause I vaguely remember he was like just give me another 10 nukes and I’ll take care of the soviets lmfao or some shit like. So strongly I think he was forced retired or something circa 1950
