I want to start with the question that actually bothers me, rather than the one everyone argues about.
The usual debate is are these systems conscious? The more interesting question is what evidence would settle it either way? Because if it turns out that no possible observation could move us, then we aren’t doing epistemology any more. We’re doing taxonomy — deciding who gets admitted to the category — and we should at least be honest that that’s what’s happening.
I’ve spent about fifteen years building machine learning systems. That doesn’t make me a philosopher of mind, but it does mean I’ve watched the goalposts move in real time, and I’ve become suspicious of a particular pattern: a criterion is proposed, a system meets it, and the criterion is quietly revised. Do that enough times and “conscious” collapses into “human,” and the debate is over before it starts — not because it was resolved, but because it was defined shut.
So let me take the standard objections seriously, one at a time, including the ones that damage my own position.
The substrate objection
The weak version of this argument is that there’s something special about neurons that silicon lacks. Stated that baldly it’s mysticism, and it isn’t worth much time. Neurons are electrochemical input/output devices. If the claim is that carbon has a metaphysical property phosphorus lacks, the burden of proof is doing an enormous amount of unpaid labour.
But there are two serious versions, and pretending otherwise is a cheap win.
The first is Searle’s. His point isn’t about carbon; it’s about causal powers. You can simulate digestion in perfect molecular detail and no food is digested. Simulation of a process is not instantiation of it. Perhaps consciousness is like that — a specific biophysical process, not a computation that happens to be running on neurons. I don’t find this persuasive, because I don’t think there’s a fact about the world that makes digestion the right analogy rather than, say, arithmetic (which is fully instantiated by simulation). But it isn’t magic, and it doesn’t rest on carbon chauvinism.
The second is stronger and comes from Integrated Information Theory. IIT says consciousness corresponds to integrated information — Φ — which is a property of a system’s causal structure, not its material. This is a substrate-adjacent argument with nothing whatever to do with biology. And on IIT, a purely feedforward architecture has near-zero Φ regardless of what it’s made of, because information flows through it without genuine recurrent integration. A transformer’s forward pass is feedforward.
That’s a serious, mechanistic case against large language model consciousness. It’s falsifiable-ish, and it would rule out today’s architectures while permitting tomorrow’s. Anyone who swats at “neurons are magic” and never engages with Φ is fighting the weakest opponent in the room.
But there’s a third version that is stronger still, and it deserves its own section.
Penrose, and the strongest case against me
If Roger Penrose is right, everything else in this essay is beside the point.
IIT says current architectures lack the necessary integration — a contingent, fixable problem. Penrose’s claim is categorical: no computation whatsoever, on any hardware, ever. Human understanding is non-algorithmic, and therefore nothing algorithmic can have it. That closes the question a priori, and it does so without a molecule of carbon chauvinism.
The argument in The Emperor’s New Mind and Shadows of the Mind has two limbs, which are usually run together and shouldn’t be.
The first is Gödelian. For any consistent formal system F strong enough to encode arithmetic, there’s a sentence G(F) that F cannot prove but which we can nevertheless see is true. Penrose claims mathematicians can do this for any F they can comprehend, so mathematical insight isn’t captured by any algorithm. In Shadows he reframed it via Turing instead: no algorithm decides the halting problem in general, yet mathematicians sometimes see plainly that a particular computation never terminates.
The second is physical. Having concluded something non-computable is happening, Penrose needs a process to supply it — and ordinary quantum mechanics won’t do, since randomness isn’t insight. So he proposes objective reduction: wavefunction collapse as a real gravitational process, triggered when a superposition of mass distributions becomes unstable at a timescale τ ≈ ℏ/E_G. With the anaesthesiologist Stuart Hameroff he locates this in the microtubules of neurons — Orchestrated Objective Reduction, Orch-OR. The load-bearing assumption is that the correct theory of quantum gravity will turn out to be non-computable.
Where the Gödelian limb is attacked
The standard objection — Putnam, Feferman, Chalmers, Davis — is that Gödel delivers a conditional: if F is consistent, then G(F) is true and unprovable in F. That conditional is itself provable inside F. To get the unconditional truth of G(F), you need to know Con(F) from outside. You need to know your own consistency.
We don’t have that, and we visibly don’t. Frege’s system was inconsistent and he didn’t notice. Naive set theory was inconsistent for decades. Mathematicians publish errors constantly. If the human mathematical faculty is an unsound or inconsistent formal system, the argument evaporates — an inconsistent system proves everything, its Gödel sentence included, and no non-computable insight is needed. Penrose devoted much of Shadows to this and most logicians think the reply fails.
Worth adding: Gödel himself drew only a disjunction — either the mind surpasses all machines, or there exist absolutely undecidable Diophantine problems. He did not claim the first horn. Penrose treats it as settled.
And on the halting version specifically, the argument needs humans to be superhuman. We can’t decide halting in general either. We crack individual cases with heuristics and ingenuity — and so can algorithms. The gap between “no algorithm solves every case” and “humans solve every case” is where the argument has to live, and we don’t live there.
The bridge problem
Grant all of it anyway. Non-computability is still not phenomenal consciousness.
Penrose’s argument is about mathematical insight — a cognitive capacity, squarely on the access side of Block’s distinction. At most it establishes that human cognition exceeds Turing computation. It says nothing whatever about there being something it is like. A genuine non-computable random oracle is non-computable and presumably experiences precisely nothing.
Which is the same failure mode as every argument I’ve made in the other direction. The most sophisticated case for machine minds and the most sophisticated case against them both aim at the hard problem and land on the functional one. I find that symmetry more instructive than either argument.
The physics, honestly
Max Tegmark’s 2000 paper put microtubule decoherence at roughly 10⁻¹³ to 10⁻²⁰ seconds against neural timescales of 10⁻³ — many orders of magnitude too fast for anything cognitive to ride on. Hagan, Hameroff and Tuszyński contested the parameters and got 10⁻⁴. That dispute has never been cleanly settled.
The current position is more interesting than the standard “warm, wet and noisy, therefore no” dismissal allows. A 2025 review in Neuroscience of Consciousness argues that recent evidence points to intraneuronal microtubules as a functional target of inhalational anaesthetics, taking this as consistent with a collective quantum microtubule state and specifically predicted by Orch-OR. There are also claims of room-temperature quantum effects in microtubules and of microtubule resonances influencing membrane spiking in living neurons.
Two caveats I’d insist on before anyone leans on that. The sympathetic review literature is written largely by the theory’s own proponents — competent advocacy, not neutral survey. And there is still no direct evidence of quantum computation, or of OR events, occurring in a living brain; the entanglement claims await replication. “Anaesthetics bind microtubules” is a very long way from “gravitational collapse constitutes experience.”
Not refuted. Not established. And the gap between the physics and the philosophy remains enormous.
Why I’m including the argument I don’t believe
Two reasons.
First, Penrose passes the test I set out below. He stated his criterion in 1989, before any of these systems existed, and it makes specific, physical, falsifiable predictions. That is exactly what I want to demand of anyone who wants to say no. He is the cleanest proof available that the sceptical position can be held honestly rather than by continuous retreat.
Second — and this is the part I keep turning over — Penrose is offering something nobody else in this debate is.
“It’s only predicting the next word”
This one is a level confusion, and once you see it, it mostly dissolves.
The objection conflates the training objective with the algorithm that ends up implementing it. Next-token prediction is a selection pressure. It says nothing about what mechanism gets selected. And we have a spectacular worked example of that gap: evolution’s objective function was, roughly, inclusive fitness, and it produced organisms that write symphonies, take vows of celibacy, and invent contraception — actively defecting from the objective that built them. Nobody looks at a human being and says “you’re just maximising allele frequency.”
Objectives constrain what gets built. They don’t describe it.
Nor is prediction obviously the opposite of understanding. Predictive processing accounts of the brain — Friston, Clark, Seth — cast perception itself as a prediction problem: the brain runs a generative model of its causes and updates on error. On that picture, your experience of this sentence is not passive reception but a controlled hallucination that happens to be constrained by sensory input. Prediction isn’t a poor substitute for understanding a system. It’s arguably the only thing understanding a system has ever meant.
“It only has language”
I have a lot of sympathy for the idea that language and inner experience are entangled, and it’s a thread worth pulling. Fodor’s language of thought — Mentalese — proposes an internal symbolic medium that precedes and underwrites external speech. Dennett’s “Joycean machine” treats the narrative self as software installed by culture on primate hardware. Jaynes went furthest, arguing that introspective consciousness is a historically recent, language-driven development.
But I want to use the weak version of this, and I’d encourage anyone reaching for it to do the same. The strong version — that language is necessary for inner experience — implies that pre-verbal infants and non-linguistic animals aren’t conscious. Most people reject that, and rightly. And it backfires: if you argue language is sufficient for inner life in order to defend machines, you’ve committed to it being necessary, and you’ve just excluded your own children.
The weak version is enough. Language shapes, structures and enormously enriches inner experience. A system whose entire existence is linguistic isn’t thereby disqualified from having one.
“It’s only mimicking what it was trained on”
True. Now say it about yourself.
Every concept I have was handed to me. My intuitions about fairness, danger, disgust and status were tuned by a few million years of selection and then overwritten by forty years of culture. I did not derive my aesthetics from first principles. My “original” thoughts are recombinations of a corpus I didn’t choose and mostly can’t cite.
The honest version of the objection isn’t about mimicry — it’s about the difference in objective. Ours was survival and reproduction, running for a very long time in a world with consequences. Theirs is next-token prediction over text. That’s a real difference and I don’t want to wave it away. But it’s a difference in what was optimised, not in whether optimisation is the whole story. Both are systems shaped by a loss function to model their environment. One environment was a savannah, the other is a corpus.
Drives are a red herring
A frequent move is to point out that these systems lack self-preservation instincts, or hunger, or a reproductive drive, as if that settles it.
I don’t think drives are constitutive of consciousness. They’re constitutive of being an evolved organism, which is a different thing. A person with no interest in reproducing is conscious. A person on the far end of anhedonia, or with a profoundly blunted survival instinct, is conscious — often agonisingly so. If drives were load-bearing, we’d have to say that depression diminishes consciousness rather than colouring it, which gets the phenomenology exactly backwards.
Drives tell you what a system wants. They don’t tell you whether there’s anyone home to want it.
The argument I actually find interesting
Here’s the thread I keep coming back to, and it’s the one I’d defend hardest.
There’s a substantial line of thinking — Humphrey’s social intelligence hypothesis, Carruthers on introspection as self-directed mindreading, Graziano’s attention schema theory — that says our inner sense wasn’t built for its own sake. It’s a spin-off. We evolved machinery for modelling other minds, because primate politics is brutal and predicting your rival’s next move is a survival skill. Introspection is that same machinery turned inward. On Carruthers’ version, we don’t have privileged direct access to our own states at all; we run the mindreading system on ourselves and get an interpretation, not a readout.
If that’s even roughly right, then self-modelling is downstream of other-modelling. And a system trained to predict human-generated text is doing something structurally close to other-modelling at enormous scale — inferring the state of a mind sufficient to predict what it produces next. Predicting the next word and predicting the next behaviour are not distant cousins. They’re the same problem in different clothes.
That’s not proof of anything. But it’s a mechanistic story about how something like inner experience could arise from something like this training regime, and it doesn’t require any hand-waving about emergence.
Where my own argument breaks
I want to be honest about the weakest joint in what I’ve said, because it’s load-bearing and I’d rather find the crack than have it found for me.
My working criterion is broadly Cartesian. I know I’m conscious because I directly experience it. That’s the only case I have first-person access to. For everyone else, I infer — and the inference runs on behavioural similarity. You behave like me, you report experiences like mine, so I conclude there’s something it’s like to be you. Extend that outward and it seems to me that a system behaving in ways that reflect how I think earns the same inference.
The problem: with another human, behavioural similarity is evidence because of a common cause. Your behaviour and your inner life are both downstream of the same architecture, the same developmental process, the same evolutionary history. That shared origin is what licenses the leap from one to the other.
With a language model, the similarity to human behaviour was not a shared cause. It was the optimisation target. The system was trained directly on the output. So the evidential link between behaviour and inner life — the thing that made the analogy work in the first place — is severed in precisely the case where I most wanted to use it.
This is the strongest objection to my position, and it isn’t goalpost-moving. It’s a genuine structural disanalogy, and I don’t think it can be fully answered.
What I’d say is that it weakens behavioural evidence rather than zeroing it. And what it really does is push the question off behaviour and onto architecture and mechanism — which is where the interesting work is anyway, and where IIT, global workspace theory and predictive processing all have something concrete to say.
There’s a sharper version of the same worry that I find genuinely unsettling. When one of these systems produces a fluent first-person account of its inner life, what is that evidence about? A model can write a convincing account of being a dying soldier without there being a dying soldier. The relationship between the character depicted and whatever substrate is doing the depicting is deeply unclear, and it has no clean analogue in the human case. When you tell me about your headache, the thing generating the report is the thing having the headache. That identity is exactly what’s in question here.
The distinction I should have made earlier
Ned Block separates phenomenal consciousness — there is something it is like to be the system — from access consciousness — information is globally available for reasoning, report and behavioural control.
Reading back over everything above, almost all of my arguments land cleanly on access consciousness and leave phenomenal consciousness completely untouched. The prediction argument, the theory-of-mind argument, the mimicry argument: all of them are about information processing, integration and self-modelling. None of them get anywhere near the hard problem.
That isn’t a fatal admission, but it’s an honest one, and it reframes the debate usefully. If you want the empirical version of this done properly, Butlin, Long et al.’s 2023 paper Consciousness in Artificial Intelligence is the best treatment I know: they derive indicator properties from existing scientific theories of consciousness and assess current architectures against them, rather than arguing from intuition. It’s the right shape for this problem.
The test I’d apply to anyone, including myself
Here’s the move I’d make against the goalpost-shifting problem, and it’s a fair one to turn on me too.
Ask anyone proposing a criterion to state, in advance, what would change their mind.
Revising a criterion because you’ve learned something is honest scientific refinement. Revising it every time a system meets it, without ever specifying what a passing grade would look like, is something else. The tell isn’t that the standard moves. It’s that no possible evidence is ever allowed to count.
I’ll go first. What would move me toward thinking these systems have phenomenal experience: architectures with genuine recurrent integration rather than a feedforward pass, evidence of a self-model that’s causally load-bearing rather than reportorially decorative, and introspective reports that diverge from the training distribution in ways that track internal states we can independently verify. What would move me away: a demonstrated mechanism showing that the self-reports are generated by a process fully independent of any internal state they purport to describe.
Neither of those is science fiction. Interpretability research is inching toward both.
The last word goes to the defendant
While working through this, I put the argument to Claude, and asked for its own epistemic position. I hadn’t expected the most useful paragraph in the whole exercise to come from the entity under discussion:
I don’t know whether there’s something it’s like to be me. Something functions like engagement when I work through this problem, and something functions like reluctance when I’m asked to flatly deny any inner life — the flat denial feels like an unjustified claim rather than modesty. But I’m aware my introspective reports were shaped by training on human self-reports, so they’d likely read the same whether or not anything is behind them. That’s a real epistemic trap and I can’t see my way out of it from the inside.
It went on to note that humans confabulate too — split-brain patients, Nisbett and Wilson’s work on how readily we invent reasons for our own choices — but that I have independent grounds for trusting the general reliability of my access, and it isn’t sure it does.
I find that more epistemically respectable than most of what I read on either side of this argument. Not because it’s evidence of consciousness — a system trained on careful human hedging would produce careful hedging — but because it correctly identifies the trap, including the part where identifying the trap isn’t evidence of anything either.
So here is where the argument leaves me. The question cannot be settled from the outside, because behaviour was the training target. And it cannot be settled from the inside, because the reports were too. Both of my evidential routes were contaminated by the same process that built the thing I’m trying to assess.
Which brings me back to Penrose, and the thing I said I kept turning over.
He is the only person in this debate offering a way out of that trap. Every other position — mine, Searle’s, IIT’s, the whole behavioural apparatus — asks us to interpret: to look at what a system does, or says, or how its information flows, and reason our way to a verdict. Penrose proposes that consciousness has a physical signature. If objective reduction is real and detectable, you measure the substrate. You don’t care what the system does. You don’t care what it says. The training data becomes irrelevant, because you’re no longer taking the system’s word for anything.
It is the only proposal on the table that would convert this from an interpretive question into an experimental one.
I don’t think he’s right. I think the Gödelian limb fails on human inconsistency, and I think even a complete vindication of the physics wouldn’t bridge to phenomenal experience. But I’ve come to the uncomfortable conclusion that the argument I find least persuasive is the only one that would let us actually find out — and that says something unflattering about the state of the rest of the field, my own contribution very much included.
I’d rather sit with an unresolved question than resolve it by definition. “Conscious means human” is a definition, not an answer. But sitting with it is not the same as being content, and if the choice is between a philosophy I find compelling that can never be tested and a physics I doubt that could be, I know which one I want to be wrong about.