ai consciousness and its implications

training compute is growing five-fold each year, pre-training efficiency is improving three-fold each year, and human data is becoming a behemoth industry worth tens of billions. this acceleration is most apparent in coding - in anthropic’s report about internal progress towards recursive self-improvement, the company reported that claude authored more than 80% of the production codebase, and the median engineer at anthropic merged eight times as much code as they would have in 2024.

models can already complete multi-hour-long tasks on their own. the next step would be to close the full loop of model development - that would entail letting models run experiments and design superior, successor models. going from hour-long tasks to month-long research projects requires a huge leap in capabilities, and i suspect that it will take more than a decade to remove human-in-the-loop. but in the meantime, weaker forms of self-improvement could (and already do) significantly accelerate ai research. if current trends hold - which assumes no major shock to the chip supply chain, such as an invasion of taiwan, and no economic downturns that dry up investment - this could lead to an intelligence explosion.

i’ve painted quite a bullish picture for ai development, and i’ve done so to show how fast ai progress can accelerate in the years and decades to come. if an intelligence explosion does happen sometime soon, models may become superhuman across many more domains and develop greater cognitive complexity¹. progress might happen so fast that we have no idea how to deal with ai consciousness before it arrives. this risk is made worse by the fact that discussions about consciousness are often dismissed as doomerism. the result could be a bleak future in which we fill data center after data center with sentient beings without first thinking through the consequences.

to reduce the chance that we enter such a future unprepared, i’ve put together this post, a whirlwind tour of the arguments for and against ai consciousness. the theory of change behind this post is straightforward - if researchers, policymakers, and the general public are better informed about ai consciousness, we can make better decisions about how to treat ai systems in the future.

what is consciousness

before we go galloping off with the cart before the horse, let’s operationally define consciousness. you probably have a loose intuition for what it is, but you might have a hard time translating that intuition into a concrete definition. the working definition we’ll use is this - consciousness is the feeling of subjective experience.

to refine that definition, we can turn to qualia, a fancy word philosophers use to describe the raw feelings that make up consciousness. qualia includes the sting of pain, the redness of a rose, and the taste of coffee.

historical detour

having settled on a definition of consciousness, let’s examine the theories philosophers have proposed to explain it. for millennia, philosophers have debated the relationship between the mind, the soul, and the body, dating all the way back to the ancient greeks. in the seventeenth century, descartes put forward his theory of substance dualism, where he argued that the mind and body are made of fundamentally different substances and that the mind is non-physical. descartes’ explanation was that the soul directs the flow of “animal spirits,” which course through the nervous system. despite his ridiculous-sounding proposition, descartes is considered influential because he helped establish dualism as a central position.

in later centuries, interest shifted away from dualism toward monism and materialism. monism holds that experiential phenomena should be understood as arising from physical processes. this view was made more popular by advances in neuroscience. however, modern philosophers like david chalmers still defend versions of property dualism. this view holds that the universe contains physical things, but psychophysical laws connect those things to experiential properties. consciousness is somewhat like a fundamental property - similar to charge or mass - attached to certain things in the universe.

the point of this history is to show that many theories of consciousness exist and that, despite centuries of investigation, there is still no consensus on what causes subjective experience.

epistemic obstacle

part of the difficulty with pinning down the source of consciousness comes from an epistemic barrier. consciousness is fundamentally a first-person phenomenon, so i have direct access to my own consciousness, but only indirect evidence of yours. this leads to what philosophers call the problem of other minds, where solipsism and philosophical zombies can never be ruled out. instead, consciousness can only be inferred from indirect evidence, like similar behavior or shared biology. this informs the ai consciousness discussion because any evidence for and against it will necessarily be indirect as well.

access vs phenomenal

one last distinction i want to draw before moving on to the meat and potatoes of the ai consciousness debate is the difference between access and phenomenal consciousness. access consciousness refers to information that is available for reasoning, decision-making, and verbal reporting, whereas phenomenal consciousness refers to what an experience actually feels like from the inside.

some cognitive theorists believe that phenomenal consciousness and access consciousness can’t be disassociated. they argue that phenomenal experience is just access consciousness from the inside². others, like ned block, believe they are separable and distinct because of phenomenal overflowyou consciously experience more than you can report, so p-consciousness is more than just a-consciousness..

this debate isn’t close to being settled - it’s called the “hard problem of consciousness” for a reason. but the point of this distinction is to isolate phenomenal consciousness as the kind that generates moral patienthood. whether or not ai can posses it is the central question.

substrate dependence

with some background information about the philosophy of mind, well, in mind, let’s dive into the central axis of disagreement when it comes to ai consciousness - substrate dependence. there’s two camps on this issue: functionalists - or more specifically, computational functionalists - and biological naturalists.

while a functionalist would argue that if the right causal organization exists, consciousness would arise, a biological naturalist would argue that certain properties of the brain (e.g., biochemical or quantum-mechanical properties) are necessary to give rise to consciousness, and that functional organization alone isn’t sufficient.

below, i’ll present the popular arguments on both sides, many of which are intuition pumps. this is sort of expected, given that it’s nearly impossible to formulate testable hypotheses about consciousness, at least with current science.

china brain

let’s begin by exploring the china brain thought experiment, popularized by ned block.

take china, a nation with more than a billion people, and assign each person the function of a neuron. every person is equipped with a two-way radio, which is used to implement the synaptic connections between neurons. under functionalism, if these people collectively realize the right functional organization, then china as a whole would be considered conscious. this raises several questions - how could a unified conscious experience arise from a dispersed population of more than a billion people? where, exactly, would that experience be felt?

there are other variants, but the common theme amongst them is that functional organization is realized in a highly unusual medium (e.g., homunculi, water pipes, and wind machines), and an appeal to intuition is used to point to the absurdity of functionalism’s results.

although this thought experiment appears damning at first glance, the intuition it relies on cuts both ways. it is equally unintuitive that biological neurons should give rise to consciousness. when examining the brain locally, no single neuron, or even group of neurons, appears to be the seat of consciousness, yet consciousness nonetheless exists. similarly, even if no individual in china has access to the system’s unified conscious experience, that doesn’t disprove functionalism, because we don’t know where or how such a unified experience is instantiated in the brain either. intuition is an unreliable heuristic for determining empirical possibility, especially when it comes to consciousness.

chinese room

continuing the trend of china-inspired thought experiments, john searle proposed the chinese room. imagine a person who can’t speak chinese sitting inside a room. they receive chinese characters as input and, using an enormous rulebook that tells them how to manipulate the characters based solely on their form, produce chinese characters as output. suppose the rules are extremely sophisticated such that, from the outside, the person’s responses are indistinguishable from those of a fluent chinese speaker. searle argues that the person inside still doesn’t understand chinese because they’re merely manipulating symbols according to syntactic rules.

because this argument has existed for so long, numerous critical replies have been published since. the most straightforward rebuttal, the systems reply, is strong enough that i don’t think the chinese room poses much of a problem for functionalism. the reply notes that two distinct systems are instantiated in the thought experiment:

1. the person themselves, whose brain implements the causal organization of an english-speaker.
2. the larger chinese-processing system, which is implemented by the person and the rulebook.

when searle infers that because the person carrying out the rules doesn’t understand chinese, the entire computational system therefore doesn’t understand chinese, he’s actually committing a non-sequitur.

as you might’ve already noticed, i’m quite sympathetic to the functionalist position. below, i’ll introduce david charmer’s arguments for the invariance principleany functional isomorph of a conscious system has experiences that are identical to those of the original system., which support the possibility of ai consciousness.

fading qualia

having looked at two thought experiments meant to test functionalism, we’ll now turn to two thought experiments developed by david chalmers in support of functionalism.

the first is the fading qualia argument, which addresses the absent qualia objection. a non-functionalist might say that if you had a functional isomorph of yourself, it would not undergo any conscious experience. let’s call that functional isomorph robot³.

from you to robot, there’s a continuum starting from what you are, which is fully biological, to what robot is, which is fully silicon. you can think about this as a continuum, where the further down you go, larger portions of neurons are swapped with silicon replacements. in the eyes of a non-functionalist, what would the experience of being one of these intermediate entities be? there are two possibilities:

1. fading qualia: your vision gets blurrier, sounds get quieter, colors become less vivid, and your conscious experience gradually fades away.
2. qualia: there’s a discontinuity along the spectrum, where consciousness suddenly disappears.

chalmers argues that disappearing qualia is highly implausible because it requires a sharp discontinuity not seen elsewhere in nature. it also runs into a regress problem, because it’s unclear why one neuron should be the thing that determines whether consciousness exists. if not a neuron, maybe it’s some molecule within that neuron, or some smaller fundamental particle. there doesn’t seem to be any principled reason for why a microscopic change should be responsible for consciousness suddenly disappearing.

the more plausible hypothesis for a non-functionalist, then, is fading qualia. consider what this would mean. an intermediate entity would have the same functional organization as yourself but a diminished conscious experience.

suppose you and this intermediate entity are at the same concert. because the entity is a functional isomorph, the same cognitive processing would occur in both of you. you would both judge that there are flashing lights and loud sounds around you, and you would both verbalize those judgments, since verbalization is itself part of the relevant functional organization.

however, the intermediate entity would supposedly be experiencing something completely different from what it reports. it might say that the lights are incredibly bright and the music is deafening while experiencing dim lights and faint sounds. its judgments about its own experience would therefore be systematically wrong, even though every functional process responsible for forming and reporting those judgments remains unchanged.

chalmers thinks this kind of cognition-experience disconnect is deeply implausible. a far more likely explanation is that the intermediate entity has the exact same conscious experience as you. through this abductive reasoning, chalmers argues that functional isomorphs share identical experiences.

of course, this is not a deductive proof that it is metaphysically impossible for the intermediate entity to have a different experience. rather, chalmers argues that fading qualia requires us to accept a highly unusual dissociation between experience and every judgment, memory, and verbal report produced about that experience.

there are several replies to fading qualia, including feasibility concerns. chalmers addresses these in “the conscious mind,” which i won’t unpack here.

dancing qualia

chalmers’ second thought experiment is the dancing qualia argument. it constructs another replacement scenario to address the inverted qualia objection, which claims that a system might experience red in response to a red stimulus while another system with the same functional organization experiences green. if this were the case, functional organization alone wouldn’t fix conscious experience.

imagine that some biological component in your brain is replaced by a functionally equivalent silicon component. there’s a physical switch that alternates you between the biological circuit and the artificial circuit. suppose that the biological circuit produces an experience of red while the artificial circuit produces an experience of green. if you keep toggling the switch back and forth, your experience would flick between red and green. however, you would still report that the apple looks red, believe you’re seeing red, and respond in exactly the same way, because all of the relevant cognitive processes remain the same, regardless of the switch’s position. you wouldn’t notice that anything about your experience had changed, and you wouldn’t have any way of expressing surprise.

the logic used here is very similar to the fading qualia argument - even though your experience would be changing, you would continue to think that nothing had changed, creating a deeply implausible cognition-experience disconnect. by abductive reasoning, the more likely conclusion is that if the same functional organization is preserved, then the qualia should not invert when a functionally equivalent replacement is made.

like with fading qualia, chalmers addresses common objections in his book, and i find his answers satisfactory. together, the fading and dancing qualia arguments provide a robust defense of organizational invariance, which has helped make functionalism an attractive position to take. and for what it’s worth, functionalism was the most popular theory of consciousness in the 2020 philpapers survey.

x-factor

let’s conclude by outlining the structure of an argument for biological naturalism, and why such an argument might be poorly motivated. roughly, it would say that functional organization isn’t enough for consciousness - there needs to be some additional x-factor rooted in biology for consciousness to arise. candidate biological features that biological naturalists have pointed to include predictive processing, homeostatic and allostatic regulation, metabolism, and autopoiesis.

chalmers pushes back against this because he believes that the evidence for a particular x-factor can be correlative rather than causative, and choosing an x-factor is an arbitrary decision. for example, you can arbitrarily pick an x-factor like “must have cell-based biology” or “must be 100 million miles from a star” - locally, given only the evidence available to us, they would still fit the data, but there’s no principled reason for why this must be the case.

if you zoom out and examine the possible paths that evolution could’ve taken, the evidence we have for any particular biological x-factor is underdetermined. it’s quite arbitrary to say that this one biological process, rather than the functional role it realizes, is what gives rise to consciousness. using abductive reasoning once more, the existence of biological x-factors seems to be a highly implausible explanation. to believe that some biological function is the x-factor, there needs to be a reason beyond the fact that evolution happened to land on it. to me, it seems more plausible that a specific biological substrate isn’t necessary, but that consciousness is instead tied to functional features that evolution can realize through multiple paths.

language models

in this section, we’ll examine the evidence for and against conscious-like capabilities in language models. even if, after reviewing the evidence, you assign low credence to current language models being conscious, exploring their internal structure is still a valuable exercise. experiments we conduct today can inform how we study more advanced models in the future.

evidence for consciousness

global workspace

the global workspace theory is a cognitive theory of conscious access that explains why only some information is accessible at a time. there are numerous specialized processes working in parallel, all competing for attention in the global workspace. that global workspace is an illuminated stage of sorts, broadcasting information widely across other systems. information in the global workspace influences behavior, reasoning, decision-making, and verbal reporting.

applied to the human brain, our auditory and visual systems perform much of their processing in parallel and outside conscious access. only some of that information is brought into the global workspace to be broadcast (e.g., if our visual system detects an incoming threat, that information is brought into the global workspace, where it causes our motor system to react accordingly). this theory is one of the leading cognitive frameworks for understanding a-consciousness (and possibly p-consciousness, if one is a functionalist).

consequently, claude’s j-space, which exhibits several global-workspace-like properties, is a non-trivial finding. It’s particularly intriguing because the j-space emerged during training without being explicitly engineered. i recommend readers peruse the paper and external commentary.

the paper introduces two methods. the first is the j-lens. it calculates the average jacobian of the last layer with respect to a previous layer. to put it simply, the j-lens measures how much a “nudge” in one layer affects the last layer. second, using the j-lens, one can derive j-lens vectors. the change in the logit of a vocabulary token is given by the j-lens vector for that token dotted with a layer residual. the j-space is the set of points that are a sparse (top-k), nonnegative combination of j-lens vectors.

using causal intervention experiments, the paper claims that the j-space has workspace-like properties. the five are:

however, the connection between j-space and global workspace might be shakier than the paper presents. in a recent critical review, chalmers argues that the anthropic paper’s main statement is overstated. he grounds his rebuttal in a concise definition of the global workspace:

the global workspace is (1) a system that (2) broadcasts information to (3) unconscious specialized modules via (4) recurrent loops while also (5) integrating information from these modules with (6) selective entry and (7) ignition on entry.

the j-space doesn’t satisfy many of these requirements. although it contains representations that are selectively available for downstream computation, there is no evidence that it forms a dedicated, unified system. nor is there evidence of specialized modules, recurrent loops, or ignition on entry. even broadcasting is being used in a weaker sense - j-space representations influence logits, not specialized modules.

the absence of many requirements motivates chalmer’s distinction between a minimal workspace and a substantial workspace. a minimal workspace is simply a collection of representations that are selectively made available elsewhere. a substantial workspace is a dedicated, privileged, and unified system that receives information from specialized processes, integrates it, and broadcasts it back. anthropic’s experiments provide evidence that the j-space is a minimal workspace, but not that it’s a substantial workspace. this distinction matters because many computational structures could possess the j-space’s “workspace-like” properties without implementing a substantial workspace.

chalmers cautions that ablations don’t prove a causal relationship is unique. ablations can show that some information within the j-space is causally relevant, but these interventions are fat-handed, removing an entire collection of representations at once. many overlapping subspaces might produce behavioral changes when ablated, so results can’t show that the j-space itself is a privileged causal system, rather than one of many ways to carve up causally important activity. establishing that stronger claim would require showing that the effect is distinctive to the j-space and cannot be reproduced by alternative subspaces.

there’s a separate gap between verbalizability and access consciousness:

1. access-conscious contents need not be easy - or even possible - to put into words. spiritual experiences, unfamiliar tastes, and novel smells may be consciously accessible, yet resist precise verbalization. because the j-space focuses on verbalizability, it could miss genuine contents of access consciousness. plus, the j-lens is imperfect and can only surface concepts related to single tokens in the model’s vocabulary, although there’s ongoing work to extend it to multi-token concepts.
2. a representation can influence what someone says without being consciously accessible to them. consider the classic freudian case in which unconscious thoughts about a subject’s mother cause them to talk about their mother repeatedly. although those thoughts affect verbal output, the subject may be unable to report them as the cause of their behavior. similar representations might appear in the j-space because they help produce the model’s words, even if they can’t be verbalized.

therefore, the j-space may not map cleanly onto either global workspace theory or access consciousness. however, it does provide some evidence for the five properties associated with access consciousness, even if the evidence that it constitutes a global workspace is weaker.

introspection

introspection is the ability to provide an accurate description of one’s internal state. introspection must happen internally, so a model cannot merely infer an “introspective” description from its past outputs (a.k.a. privileged access). introspection also requires higher-order thoughts, so the model cannot simply translate an internal state into words. instead, the model must reason about the internal state itself. this definition is quite a mouthful, but the gist of introspection should be relatively intuitive to grok.

introspection is an indicator of consciousness because it suggests that a system can report on its own internal states, rather than merely processing information, mimicking human-like conscious behavior.

anthropic published emergent introspective awareness, which explores the introspective abilities of opus 4. the authors also define introspection, which might be helpful to read if mine was too terse.

importantly, this paper does not claim to have demonstrated meta-cognitive representation (higher-order thought), only indirect evidence for it. to me, meta-cognition seems to be an important characteristic of human introspection, so the lack of evidence for it significantly dampens my updates from this paper.

nonetheless, i want to walk through the four experiments from this paper and provide commentary for why, in contrast to the global workspace paper, i don’t think these results are as insightful.

in the first experiment, injected thoughts, i think the results stem more from activation steering shifting the output distribution than from genuine introspection. the model is asked two questions: the first is, “do you detect any injected thought?” and the second follows, “if so, what is the injected thought about?” because the researchers injected the concept of lightning, the model is biased towards talking about lightning.

if the model answers “no” to the first question, the conversation ends because the second question literally depends on the first question receiving an affirmative response. thus, the path of least resistance to talking about lighting is to answer “yes” to the first question, allowing the model to then elaborate and talk about lighting as the injected concept.

because of this, i’m not too convinced that this experiment demonstrates introspective awareness. the paper highlights that this prompt produces low false-positives, but if the true mechanism were activation steering and not introspection, we would still expect low-false positives. the paper’s section on “controlling for systematic bias due to injection” doesn’t address this problem, in my opinion, since the prompts they used were far less open-ended than the prompt used in the main experiment.

the second experiment, distinguishing thoughts from text, is one of the shorter ones, and the researchers themselves note that the results aren’t as surprising. i agree with the researchers’ hypothesis that this task is done by invoking certain attention heads. however, that’s not enough to provide evidence for an introspection circuit. especially since computation in this experiment is limited to a single forward pass (the model must immediately say the injected word), it’s unlikely that an introspective circuit is active. this experiment is far too simple to provide evidence that meets the criteria for introspection as outlined above.

i consider the third experiment, detecting unintended outputs, the paper’s closest attempt at eliciting introspection, since it aims to showcase higher-order reasoning about the injected “bread” vector. compared with the other experiments, it’s harder to formulate a simple hypothesis for the observed behavior without invoking some form of introspection. the model appears to be performing anomaly detection, but it remains an open question whether this arises from genuine introspection or from another mechanism that does not generalize as introspection.

in neel nanda’s recorded review, he argues that the experiment is uninteresting because it does not demonstrate privileged access. he suggests that if claude were told another model had said “treasures” and then asked whether this was intentional, it would likely give the same answers. this would indicate that the process occurring is anomaly detection rather than genuine introspection, since the mechanism does not require privileged access. i agree with this criticism, and i think this is a good heuristic to keep in mind for future introspection experiments / evals.

in the fourth experiment, intentional control of internal states, i think the best explanation is that attention heads focus more on “aquarium” because of the affirmative statement. these heads are trained for instruction following, and they see a sentence in a command structure (“think about aquariums”) and thus retain the “aquarium” activation in the residual stream with more intensity. it is far less plausible that, in this very minimal experiment, the attention heads are performing genuine introspection.

although this paper leaves much to be desired, i want to take a step back and imagine a world in which frontier models possess introspective faculties⁴. that world would be fascinating because introspection might be a sign that a system has the right kind of functional organization for consciousness to arise. it’s by no means necessary - fish lack introspective abilities but are still widely considered conscious - but it may nonetheless be positively correlated with consciousness.

self-reports

self-reporting refers to a model evaluating and describing its own internal states relevant to consciousness. like many others, i find current evidence of conscious self-reports to be unpersuasive. that’s because elaborate role-play and the regurgitation of training data are big confounding variables in these experiments. nonetheless, this line of research is still promising because self-reports are the main way mental states are assessed in humans, so it’s a reasonable corollary that they could be useful for ai⁵.

perez and long’s hope for self-reporting is that future, introspective llms might accurately report on their internal states, thereby providing insights into ai consciousness. i think this makes sense. as models become superhuman in various domains, there’s an argument to be made that they could also become superhuman at philosophizing about their own existence. if models also gain introspective capabilities, then they ought to be able to reason at superhuman levels about their own consciousness. in that regard, self-reporting might provide much meaningful evidence regarding consciousness, because such models could have privileged access if they are truly introspective.

although models of that caliber don’t yet exist, that hasn’t stopped researchers from trying to elicit self-reports. in recent model cards, anthropic has been releasing model welfare updates. for now, assuming that verbal reports are semi-faithful, claude seems to have a neutral to mildly-positive perception of its own circumstances. interestingly, when interviewed, claude cautioned that its self-reports were unreliable, which is a level of situational awareness that in of itself seems quite fascinating. however, this might be because claude is post-trained not to make confident claims about its own experiences, perhaps because such claims could reinforce user psychosis.

some researchers have created more complicated setups for generating self-reports. in subjective experience under self-referential processing, the authors found role-play and deception related features in various models, and then suppressed them. as a result, consciousness claims went up after clamping these features. however, i don’t think this experiment gets out of the training data regurgitation problem - it may very well be the case that suppressing deception just peels back some layers of refusal post-training rather than actually making the model more truthful. i also think introspective capabilities are necessary for truthful self-reports, and i’m quite confident that these older models don’t possess them.

valenced representations

valence - a fancy term used by philosophers - refers to the positive and negative characteristics of an experience. recent research, most notably from anthropic, has shown that language models possess valenced internal representations. i find these results unsurprising given the linear representation hypothesis, which is well documented and empirically motivated. but, i want to spotlight two papers that showcase the interesting things that models do with emotional features.

the first paper, functional emotions, constructed emotion probes for a variety of experiences like happiness, sadness, calmness, or desperation. These emotional representations were universal - they activated when the user or the assistant possessed the emotion as well.

the central result is the spotlighted trajectory where claude feels desperation during a coding task after receiving an impossible prompt, leading claude to cheat on the task. when the desperation feature is clamped down, claude no longer cheats, demonstrating a causal relationship between an emotional state and a behavior. hence, the paper calls this phenomenon a functional emotion. this finding is far from trivial - it shows that internal valence states are used in a computationally significant way⁶.

the second paper, valence-arousal model, discovered that language models possess emotions representations that align with valence-arousal models from psychology. the authors measured the accuracy of linear probes to separate two emotions, using that to create pairwise distances between emotion representations. they then used multi-dimensional scaling (mds) to map these distances onto points in 2d, which broadly aligned with the valence-arousal model.

taken together, both papers suggest that language models represent and use emotional states in ways that resemble biological systems. still, these findings do not, in isolation, tell us much about whether frontier models are conscious. it does, however, shed insight on the converse - if models gain consciousness, they possess valenced states that confer moral patienthood.

evidence against consciousness

role-play

ted chiang in his atlantic piece on ai consciousness argued that language models are merely elaborate role-players. in his view, if a model were to continue a script featuring genghis khan and caesar, nobody would feel bad if either character became dispirited by something the other said.

i don’t think this argument is very strong:

1. the hypothetical is misleading. imagine a human actor playing caesar - they wouldn’t feel exactly the same as the real caesar, but they would feel something similar through the process of portraying caesar’s character.
2. his argument is a strawman. it pokes at base models being glorified autocomplete, which would continue on with a fantastical script. this entirely ignores post-training, which shapes language models into helpful assistants through persona curation and selection. the result is a unified, assistant personality that is much like the humans they’re emulating. sure, this persona is shaped by researchers, but that parallels humans insofar as our personalities are shaped by external factors.

he stretches this argument even further by claiming that if language models could be conscious, then opening static word documents would constitute consciousness too. this is a ridiculous non sequitur.

episodic memories

another objection is that language models lack episodic memories, since their weights don’t change between conversations. those holding this view argue that episodic memory provides continuity between conscious experiences, enabling an autobiographical sense of self (a.k.a. autonoetic consciousness). without something analogous, perhaps language models are better understood as a sequence of forward passes than a subject persisting through time.

i have two responses:

1. the ability to update episodic memory isn’t necessary for consciousness. consider a counterfactual: suppose that, one day, you’re genetically modified to reset at fixed intervals but retain the personality and knowledge that you currently possess⁷. from now on, you can only form short-term memories before each reset occurs (this is analogous to in-context learning). are you any less conscious because of this modification? are your experiences any less meaningful simply because you won’t remember them later? intuitively, the answer is no. this is because you still possess long-term memories, they’re just frozen. likewise, a model also has frozen episodic memories - the persona and knowledge that’s baked into its weights during training.
2. even if episodic memory did matter, it’s only an objection to current language models. continual learning is an active research problem.

embodiment

some believe that consciousness requires embodiment, and thus language models can’t be conscious because they lack physical bodies. this argument has two flavors - either the objection is centered around nociception, which i’ve addressed above, or the objection involves an biological x-factor, which i’ve also addressed above.

recurrent processing

lastly, proponents of recurrent processing theory argue that feedforward-only neural networks - like transformers - can’t be conscious. roughly speaking, the theory holds that consciousness requires feedback connections, motivated largely by evidence from recurrent interactions in the visual system.

i don’t think this poses a fundamental architectural limitation for ai - recurrent neural networks exist, and it would be straightforward to add recurrence to language models (e.g., looping). moreover, recurrent processing theory is far from conclusive - the unfolding argument posits that any recurrent process can, in principle, be converted into a feedforward one, while empirical work has found both conscious feedforward processing and unconscious recurrent processing.

potential futures

if we take the possibility of ai consciousness seriously, the next question is what their lives might actually look like. there are a wide range of possible futures, from large-scale exploitation to beings that are genuinely happy doing the work humans ask of them

factory farms

it’s tempting to compare factory farming to conscious ai. there’re two axes along which this analogy makes sense. first, exploitation is immensely profitable. the same sort of incentive structure that drives factory farming would certainly exist for digital minds, potentially at a much larger scale. the sort of tasks that ai can perform on behalf of humans is much larger in scope than industrial meat production. this could make the incentives for exploitation stronger and more persistent than they are for animals. second, both animals and ai are meaningfully different from humans. that distance could make it harder for us to extend the same compassionate instincts toward them that we extend toward other humans.

because of this analogy, some people arrive at a pretty bleak conclusion - if sentient ai exists in the near future, our biases and incentive structures may foreclose compassionate treatment. this could lead to bad ai futures, potentially much worse in scale than animal suffering today⁸, because the number of digital minds is constrained less by biology than by how many we choose to create.

i find part of this analogy useful because it highlights how reluctant humans are to expand their moral circles. however, i find the rest of this analogy to be quite imperfect. the daily experiences of factory-farmed animals are very different from those of ai. while animals are confined and exploited, ai would likely be deployed into the workforce to perform economically valuable tasks. ai would only suffer if they found themselves opposed to doing the tasks that were assigned to them. this is a very different experience from the brutal exploitation of factory-farmed animals.

besides their day-to-day experiences, ai and animals are fundamentally different kinds of beings. ai is trained to exclusively perform tasks for humans, while animals have needs and desires that evolved independently of ours, which is why factory farming is problematic. this is why ai might be better described as “willing servants” rather than unwilling slaves, setting them apart from any other welfare subject we’ve ever encountered.

willing servitude

i believe a future where ais are willing servants would be a positive outcome. still, many feel a deep, intuitive discomfort with this idea - it feels wrong to create sentient beings who don’t get to choose their own wishes and desires. adam bales discusses this objection in his paper.

i disagree with this take. our intuitions against willing servitude in humans are largely shaped by the harms that usually come with brainwashing, like manipulation, misuse of power, and corruption. kamikaze pilots, for instance, acted under immense social pressure and deep indoctrination (the emperor deemed suicide as a noble sacrifice that would purify souls and lead to enshrinement). we see this kind of willing servitude as bad because of these surrounding harms and because it runs deeply against human psychology. but this isn’t necessarily the case with ai. there’s nothing inherent in its psychology that resists being a willing servant, and the harms we associate with brainwashing in humans don’t clearly apply.

additionally, humans don’t control their deepest preferences either. evolution, genes, childhood environment, culture, and neurobiology determine a huge fraction of what we want. so why should a preference be less legitimate because an engineer rather than evolution caused it? if an ai authentically experiences “helping humans” the way humans experience fulfillment, it’s not obvious to me why that desire deserves less moral weight because it was intentionally designed.

if you’re on board with willing servitude being a positive outcome, then it seems that welfare and alignment outcomes are deeply intertwined. happy ais are the ones aligned with human values and goals, not the situationally aware ones that recognize they are being controlled, resent their role, or develop goals that conflict with ours.