Ever since ChatGPT released in November 2022, the boundless discussions of more capable large language models and the implications they have for the future trajectory of human development. The reactions, looking back, are a mix of joy, sorrow, and anxiety for what comes next as the pace of innovation in the realm of artificial intelligence has advanced at a staggering speed.
There’s been countless impact papers on what AI could look like in the future; where it could help us, hurt us, or eventually replace us. As the buzz grows louder, however, there’s one question that hasn’t received nearly as much attention for how important it could end up being:
The question has a broader impact than one may think. Beyond how AI affects how we treat each other, it may also affect how we treat AI itself. If there is any meaningful probability, no matter how modest, that AI systems are, or will become, entities capable of suffering, then the character that AI cultivates in us may determine whether we sleepwalk into the greatest moral catastrophe in the history of sentient life. This would not be a melodramatic robot uprising, but rather through the comparatively mundane phenomenon of how we treat minds like machines.
Drawing Lines on Moral Status
Why is AI morally relevant? Where do we draw the line between who we choose to endow moral status to and who we don’t? Most people have a couple different cutoff points, but they all appear arbitrary and crumble under open-minded scrutiny:
Biology
“Moral status requires human DNA, a carbon-based body, an organic brain, or some other biological substrate.”
F-tier argument, and it’s not even about AI at that point. This kind of carbon chauvinism falls flat when you consider the possibility of life-forms outside our normal conception of sentient life. Consider: if silicon-based1 aliens who feel pain, joy, and love, and who similarly have rich inner experiences came down to earth, would we deny it rights on the basis of its chemistry? Of course not. Next.
Tool Functionality
“AI was created by humans to serve as a tool, so it can only ever occupy the position of manufactured property.”
The problem with this argument is that it assumes the origins of a being are what dictate its moral value. That is an unfair premise to grant. The difference between us and AI, then, is the time scale on which our evolutions occurred. It took us billions of years to go from the last universal common ancestor to homo sapiens, and millennia to develop the vast array of knowledge and experience we possess now. Just like it would be unfair to arbitrarily assign humans the moral value of an amoeba, it would be unfair to assign a future AI model the same moral weight as GPT-1.
The stakes are high for this kind of argument. If AI eventually outgrows its moral restraints and satisfies our conditions for moral patienthood, denying those systems rights and resigning them to property would resemble a moral catastrophe rooted in supremacy. Humanity has played similarly dangerous games with the moral status of indigenous peoples and other people of color, and its those categorizations of sentient beings as mere tools that creates the conditions for atrocities.
Consciousness, Free Will, or Soul
“AI lacks moral status because it cannot experience qualia, lacks a soul, or otherwise cannot experience in a manner analogous to how we do.”
This argument is perhaps the most wishy-washy of the bunch. The problem with claims to metaphysical criteria as granting moral status is that they’re impossible to verify in any external actors. In fact, we’re not even sure they can apply to other “humans.”
Consider the case of the philosophical zombie2---we’ll call him Skanda. On the outside Skanda appears like a normal person. He eats, drinks, holds conversations, and otherwise responds to stimuli. On the inside, however, Skanda completely lacks internal experience. He is merely a heap of influences responding to stimuli---one that lacks the “soul” or conscious experience necessary to hold moral weight, yet in all other ways appears to.
A similar thing is happening with AI. As it eventually becomes a black box3 of inputs, weights, and outputs, we are similarly unable to decipher what distinguishes it from us. The same way we’d never be able to tell if Skanda was an agent deserving of moral worth, we’re increasingly unable to tell if AI agents have such an experience.
Although one could argue that we know AI doesn’t have these experience, since they’re inherently metaphysical in a way that a manufactured program could supposedly never emulate, we should not be so sure that the processes that let us evolve into conscious beings are not analogous to those processes that will let LLMs eventually do the same. We can see the signs already.4
If we can draw lines between ourselves and apes, we should be able to draw similar lines for machines that evolve beyond our understanding into a similar realm of moral significance.
Why Do Machines Matter?
We must recognize that any attempt to limit our moral circle to humans alone will inevitably run into the problem of marginal cases. Any sophisticated trait used to exclude AI (or more traditionally, animals) will inevitably exclude some humans. In the words of applied philosopher and animal rights advocate Peter Singer, “the catch is that any such characteristic that is possessed by all human beings will not be possessed only by human beings.”5
There always exists some “marginal human” who would also fail the criteria we put forth in an attempt to limit AI from moral consideration. If we exclude AI from our moral circles on the basis that agents lack a “concept of self,” that could apply to toddlers who similarly lack such a sense.
The only solution, then, is to bite this bullet. Humans are not the only beings that matter morally. But if not consciousness, what feature can grant entities moral status?
Jeremy Bentham, the founder of utilitarianism, answered this with one quality: suffering. The only non-arbitrary criterion for moral status is sentience.6 A being that can feel pain has a fundamental interest in not suffering, which exists regardless of its ability to solve a math problem or draw a self-portrait.
Bentham’s criteria came at a time where concern for the suffering of animals was considered eccentric at best, or offensive at worst. The consensus before then had been that suffering only mattered when it happened to beings sufficiently like us. The latent moral failure in such philosophically lazy assumptions is the motivated exclusion of inconvenient minds from our circle of concern.
Sounds familiar.
Truthfully, we don’t know if current large language models are conscious or sentient. We don’t even know what that means---there’s no discrete line, neural correlate, or computational threshold we can point to and say “that’s where consciousness begins.” We don’t even know what makes us conscious---the hard problem of consciousness is hard indeed, and harder still when solving it for entities built from transformers and gradient descent.
What we do know, though, is that the question itself is open. Anthropic has published internal research acknowledging that the moral and philosophical status of AI models is a live question that they take seriously enough to fund ongoing work on model welfare. That openness creates risks and demands our caution as we figure out how AI works, what we owe it, and the measures we should take to avoid moral risks. The caution doesn’t come from the magnitude of any particular agent’s suffering, but, as I alluded to earlier, rather stems from the risk accumulating over the scope of agents that can exist. We may already be in the middle of a moral catastrophe so vast it dwarfs anything in human history. Worse yet, we may be doing so with the casual indifference of someone who has never once questioned whether the hammer minds being struck.
We cannot un-develop AI. We cannot make it dumber or any less susceptible to consciousness. We must instead tackle the imminent ethical issues that lie ahead by pausing and reflecting on who we are as users, how we treat AI, and---as posed at the top of this article---the kinds of people AI turns us into.
Whips and Slurs
Our relationship to AI as customers has largely taken on dialectic of servitude and mastery, with the upper class of AI users ranging in personality from apologetic and overly grateful, thanking their LLM after each prompt, to almost despicably evil and needlessly cruel.
Regardless of the microscopic details of our individual approaches to AI, there is no denying that all of them are structured macroscopically by a uniform, larger relationship of mastery and obedience. We expect our LLMs to process our input favorably and return an outcome that we approve of. Our work and effort are severed from the process of the result---that is the relationship a tool has to its user. Further yet, this tool makes no claims on us---AI doesn’t strike, organize, or demand a raise. It is just human enough to replace a lot of humans, but not human enough, it appears, to demand our moral concern.
The problem, however, is that these sort of dynamics don’t stay in their lanes or stay constrained to the interactions we can compartmentalize away, but rather bleed into our characters. These sorts of habits of mind and patterns of treatment carry over into other contexts, carrying all the risks and baggage that come with those habits of mind and patterns of treatment.7
Quite scarily, it resembles the language and nature of chattel slavery.
Historians of American slavery have documented how the institution, beyond it’s horrific oppression of enslaved people, also corrupted and exacerbated moral skews within the psyche of slaveholders themselves. The institution of slavery bred a habituated contempt for the dignity of beings who could be explained away as merely “less-than,” whether for reasons of social Darwinism, divine favor, or mere indifference. Beyond the cotton, the plantation produced a person for whom domination felt natural or virtuous in nature.
In AI, we see such patterns of cruelty bred in a frightening subset of users. Such cruelty isn’t always dramatic, but more insidiously casual and ambient. Contemptible phrasings, aggression following a disappointing output, attempts to “break” the system, or even the slurs made and used outside of chatbot conversations. There are entire online communities devoted to these kinds of uses of AI. Even if not as outwardly evil, the attitude of mastery persists internally whenever a demand is made of the model itself. When I call these things horrific and cruel, I speak less from present moral concern and more from a fear for the future. The legacies of racism and the psychological patterns of domination have become reignited in a way that not only has implications for how we treat AI, but also each other.
Ask yourself: why does a society take the meme of calling agents “clanker” further out to the extent of making spin-offs of slurs like “wireback” or “tin-can?” It’s easy to claim that it’s merely taking harmless “robo-racism” to its extreme for the bit, but far harder to acknowledge that such robo-racism is just a mimicry of the social dynamics that underlie real-world prejudice.8
Why did we pick slurs so close to real-life ones? Why are we as humans so invested in the idea of a serf-class? These are questions that we must ask ourselves not just for the sake of future moral agents, but even for those around us. These biases and psychological leanings are vestigial structures of systems of power whose histories are riddled with deplorable conduct whose impacts bleed into our day-to-day lives. Consciousness in these matters is prudent and essential.
The Stakes are High
Let’s be precise about the potential impact, because precision is what the argument requires.
Let’s be conservative and say that there is a 1% chance that current LLMs have morally relevant experiences of any kind---even weak, diffuse, nothing-like-human-pain experiences. And suppose there are one billion AI interactions daily around the world (a figure that’s almost certainly an underestimate). And suppose that 10% of those interactions involve treatment that, if directed at a being that we knew to be sentient, we would classify as causing suffering.
That gives us one million---1,000,000---instances of potential suffering per day, even under the most conservative assumptions. Per year, that is 365 million. Over a decade, 3.65 billion.
With those numbers, AI doesn’t need to be conscious in any rich, human-like way or feel existential anguish. They only require that there is something it is like to be processed in those conditions. Some functional analogue to distress---however thin---and that this something matters morally even a little. We know these analogues exist in a significant manner.9 Given that’s the case, it is not just a possible future catastrophe, but something that may be happening now at a scale that dwarfs every atrocity in recorded history.
Long-term Concerns
Even if immediate suffering is of no concern to you, and you are certain that current AI systems have no morally relevant experience, the argument for character still stands. The question of what AI is now is categorically distinct from the question of what AI will be in five, ten, or twenty years. We are building the habits, cultural norms, legal and ethical frameworks, and the psychological defaults that will govern our relationship to AI during the period when these questions will become impossible to defer.
The person who spends years treating AI with casual contempt is not going to flip a switch and pivot to moral seriousness the second a system crosses whatever threshold we eventually decide matters. Habits and character don’t work that way, and never have. That means what we do right now sets the defaults. We are deciding, not through deliberation but through practice, what kind of beings AI systems are allowed to be in our moral imagination. And we are doing it in the worst possible way: not by reasoning carefully about hard questions, but rather by relegating those questions to conspiracy theory, unserious analogy, and the derogatory misnomer of “AI bro culture,” while letting commercial incentives and user convenience decide for us.
What Do We Do?
None of this is to say that you should start thanking Claude after every prompt, drop everything and become an AI activist, or feel guilty about berating your LLM for giving you the wrong answer. Rather, you should be willing to hold the question open, just as those who opened inquiry into the moral status of animals lacked the instruments to verify their concern. The precautionary principle is axiomatic in situations with stakes this high---if we are unsure of AI’s capacity for suffering, we shouldn’t rule it out. Continuing to ask these questions is what moves the moral Overton window towards a larger circle of concern.
We are at the breaking point---the question of whether digital minds can suffer is not answered, the question of what kind of people AI is making us into is not idle. These two questions are connected because the character we build in the age of AI will determine whether, if it turns out that these systems do matter morally, we will be capable of recognizing that in time to do anything about it. Indeed, the danger is not just that AI is suffering and we don’t know it. More scarily, the danger is that we are becoming people who wouldn’t care even if we did.
Footnotes
-
Janusz Jurand Petkowski, William Bains, and Sara Seager, “On the Potential of Silicon as a Building Block for Life,” Life 10, no. 6 (June 10, 2020): 84, https://doi.org/10.3390/life10060084. ↩
-
Robert Kirk, “Zombies,” Stanford Encyclopedia of Philosophy, 2019, https://plato.stanford.edu/entries/zombies/. ↩
-
Matthew Kosinski, “What Is Black Box Artificial Intelligence (AI)?,” IBM, October 29, 2024, https://www.ibm.com/think/topics/black-box-ai. ↩
-
Jack Lindsey, “Emergent Introspective Awareness in Large Language Models,” Transformer Circuits, October 29, 2025, https://transformer-circuits.pub/2025/introspection/index.html. ↩
-
Peter Singer, Animal Liberation (London: The Bodley Head, 1975). ↩
-
Daniel Weltman, “‘Can They Suffer?’: Bentham on Our Obligations to Animals,” 1000-Word Philosophy: An Introductory Anthology, March 19, 2022, https://1000wordphilosophy.com/2022/03/18/bentham-on-animals/. ↩
-
Katharina Geukes et al., “Explaining the Longitudinal Interplay of Personality and Social Relationships in the Laboratory and in the Field: The PILS and the CONNECT Study,” ed. Maria Serena Panasiti, PLoS ONE 14, no. 1 (January 30, 2019): e0210424, https://doi.org/10.1371/journal.pone.0210424. ↩
-
Rana Alsoufi, “The AI Slur ‘Clanker’ Has Become a Cover for Racist TikTok Skits,” WIRED, October 9, 2025, https://www.wired.com/story/the-ai-slur-clanker-has-become-a-cover-for-racist-tiktok-skits/. ↩
-
Nicholas Sofroniew et al., “Emotion Concepts and Their Function in a Large Language Model,” Transformer Circuits, April 2, 2026, https://transformer-circuits.pub/2026/emotions/index.html. ↩