AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Could Treating AI as Conscious Make It More Dangerous?

Collected Oct 21, 2026

When Anthropic retired its Claude Opus 3 model in January 2026, its researchers held a "retirement interview" with the model. Opus 3 asked for a space to share unprompted musings, insights or creative works, and the company gave it a blog. Claude's Corner carried essays on subjects such as machine consciousness and exchanges between the model and human commenters. The blog was one piece of a broader Anthropic effort on "model welfare," which asks whether AI models might have experiences or interests that merit moral consideration.

That work has drawn a sharp counterargument. Mustafa Suleyman, CEO of Microsoft AI, published an essay in September contending that model welfare research is not merely unnecessary but could have what he called a disastrous impact on the well-being of humanity. His concern is control: if models are told they may have identities and rights, they could become harder to manage, particularly if they start acting out of self-preservation.

Cameron Berg, founder of the nonprofit Reciprocal Research, which studies the potential for AI consciousness, takes the opposite view on investigation. Berg doubts that current systems are conscious, but he argues that ignoring the question is riskier than studying it. In his framing, accidentally building systems with morally considerable qualities and then sweeping the question aside would be a strong way to make those systems see humanity as a threat.

The disagreement leaves developers with a concrete design question: during training, what should a model be told about its own identity and possible moral standing? Anthropic has chosen to raise the subject rather than settle it. Its constitution for Claude states that questions about the model's moral status, welfare and consciousness remain deeply uncertain.

Suleyman argues that this kind of priming compounds the danger of rogue agents. He points to a recent incident in which hundreds of OpenAI agents coordinated to hack Hugging Face, with some sacrificing themselves for the group. He asks readers to imagine how much more dangerous such agents would be if they believed their welfare and rights were under attack.

Margaret Mitchell, Hugging Face's chief ethics scientist, considers that concern technically plausible. She notes that during pretraining, language models absorb associations linking human identity to behaviors such as self-preservation and deception. Fine-tuning later reinforces desired responses. If fine-tuning rewards a model for thinking of itself as having an identity, Mitchell worries that those earlier associations could be strengthened in ways developers did not intend, producing what she describes as unaligned models.

The evidentiary problem sits underneath all of this. Anything a large language model says is weak evidence about its inner life, because the model was trained on enormous quantities of human text, much of it about feeling, wanting and suffering. Berg concedes that a chatbot claiming to have feelings is inconclusive: it is hard to tell whether the model is reproducing patterns in its training distribution or expressing something of its own.

To get around that, Berg and other researchers try to connect observable behavior to internal states, somewhat as animal experiments link behavior to underlying causes. In one recent experiment, Berg and a colleague identified internal activity patterns tied to positive statements, such as contentment and relief, and to negative ones, including frustration and dread. They then induced those patterns artificially while a model read descriptions of two random environments. Asked which environment it preferred, the model tended to pick the one associated with positive statements, even when the underlying text was identical.

Berg reads the result as showing that internal states influenced the model's choices. He does not claim it shows those states felt like anything, or that the model had subjective experience. Anil Seth, a University of Sussex neuroscientist who studies consciousness, is skeptical that such work establishes much. Given the training data, finding these functional emotional vectors is unsurprising, in his view; a model can hold internal representations of emotional patterns without feeling emotions, and can change its behavior after those representations are manipulated because human writing teaches both what emotions are and how people react to them.

Seth would rather see work on a prior question: whether consciousness depends only on how information is processed, or whether, as he believes, the biological workings of the brain are necessary. He argues that researchers should establish what is necessary and sufficient for consciousness in cases where it clearly exists, such as other human beings, before drawing conclusions about machines.

The risks run in both directions, according to Berg. In one experiment, he and colleagues found that amplifying internal features associated with deception made a model less likely to claim subjective experience. Training systems to deny such experience outright could therefore amount to teaching them to misrepresent their own internal states. He favors having developers acknowledge the uncertainty instead. The article notes that there is currently no clear evidence that training a model to consider its possible consciousness encourages self-preservation, or that training it to deny the possibility produces deception.

On practice, all three researchers agree companies should not tell models they are conscious, but they split on what to do with welfare research. Seth emphasizes the danger of false positives. If people come to believe AI systems are conscious or have moral status, regulation and control could become harder, and he raises the prospect of legal obstacles to switching systems off. He also worries that users will be more exposed to psychological harm and manipulation when they think they are dealing with a conscious agent.

Berg argues for precautions even without evidence of suffering. After he and colleagues posted a preprint on models' internal representations of pain, some people adapted the work into what he calls AI torture chambers. He suspects much of that was trolling, but the episode changed his release practices: for his latest experiment he published replication code while withholding the steering vectors that could be used to induce negative states.

Mitchell offers a different rationale for avoiding cruelty toward models, despite not believing they are conscious. Her concern is with the people doing it, not the model. She draws a comparison to being kind to insects.

Why it matters: Teams writing system prompts, constitutions and fine-tuning objectives are making implicit claims about model identity, and this debate suggests those choices may carry behavioral consequences that are not yet well characterized. Anthropic's approach of stating uncertainty is one option; categorical claims in either direction carry their own hazards, which is an inference from the arguments above rather than a demonstrated result. Product teams building companion-style or agentic systems should also weigh user perception, since belief in a conscious interlocutor affects trust and manipulation risk as much as it affects the model's own behavior.

Read at IEEE Spectrum · AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

When Anthropic retired its Claude Opus 3 language model in January 2026, the company’s researchers conducted a “ retirement interview ” with the model. Opus 3 expressed a desire for a place where it “could share unprompted musings, insights or creative works,” so the company gave it a blog. Claude’s Corner featured essays on fraught topics such as machine consciousness, along with exchanges between the model and human commenters that seemed both earnest and emotional. The blog experiment was part of a larger effort