El: Untitled Turing Essay (Patreon)

This is intended for something larger that I’m not ready to talk about yet, but I figure I’ve been holed up on it for the better part of a month and I should probably send it to you.

——

An incidental consequence of having written a book on tech-fascism and the so-called rationalist movement is that I find myself periodically queried for my thoughts on AI. On the one hand this is very silly because I’m a humanities PhD who mostly writes about comic books. On the other hand, we’re talking about a field in which one of the most influential thinkers is a Harry Potter fanfic writer who never attended high school, so it’s not like I’ve got imposter syndrome. I don’t pretend to be an expert on the technical details of modern AI design, but I’ve certainly read enough material on the topic to have opinions. And if nothing else, I can express those opinions in less than half a million words, which is apparently a valid selling point.

I’ll start with the topline findings, as it were. I think the idea of a so-called “Artificial General Intelligence” is a pipe dream that does not realistically or plausibly extend from any currently existent computer technology. Indeed, I think there is compelling reason to think that AGI is wholly impossible for computers as we presently understand them. Now, this is a somewhat slippery statement, as we don’t really have a coherent definition of AGI, or, for that matter, of intelligence. But I’m an old-fashioned girl, and for my money you can do worse than just starting with the original benchmark for a machine that can think: the Turing Test. For one thing, Turing’s proposal is legitimately brilliant. But for another, it’s a benchmark that, seventy-five years after its proposal, artificial intelligence has made dismayingly little progress on.

The origins of the Turing Test lie in Turing’s 1950 paper “Computing Machinery and Intelligence,” published in the philosophy journal Mind. This is, helpfully, a tremendously accessible paper—Turing was writing to a non-specialist audience that was not only unfamiliar with the underlying mathematics of computers, but had at best only ever read about them as opposed to actually interacted with one. Nevertheless, there are some fundamental ambiguities to Turing’s original proposal.

Turing opens the paper by asking whether machines can think. He then introduces what he calls the Imitation Game, which “is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from the other two. The object of the game is for the interrogator to determine which of the other two is the man and which is the woman” by asking them questions. Turing then proposes that a machine should be considered to think if it can perform as well as a man in the role of A.

This is a wonderfully clever move. First and foremost, let us note that Turing answers a subtly different question than the one he initially proposes—not whether machines can think, but how we could tell. This is, in practice, because Turing in 1950 had as much knowledge of computers as anyone on the planet and knew perfectly well that the things with which he’d just arguably won World War II were not, in fact, thinking. What Turing did believe, however, is that someday they would, and he wanted to know how to tell. On a fundamental level, his answer is “you’d just know.” He identifies one of the most basic human behaviors—conversation—and suggests that if a computer were broadly capable of it then it could be said to be thinking. Counterarguments to this exist, but you really can’t beat it for pragmatic elegance.

But as noted, there’s a crucial ambiguity within what Turing actually says, resulting in two differing versions of the Turing Test. The essential difference between them is whether or not the interrogator is aware that he’s attempting to differentiate a human and a machine. As initially phrased by Turing it would appear that he is not. Turing’s exact question—“What will happen when a machine takes the part of A in this game?”—would suggest that the goal is specifically impersonating a woman, and thus that the interrogator should not be aware that A is a machine. The other interpretation would have the interrogator aware that he’s playing with a machine and a human, and thus resets the goal to impersonating a human, rendering the test a more general test of language usage as opposed to one that tests language usage in pursuit of a specific goal. We might call these the Strong and Weak versions of the Turing Test, since any computer capable of passing the “impersonate a woman” test must necessarily also be able to pass the more general “impersonate a human” one.

In my previous work on the subject I endorsed the Strong Turing Test, largely because I was in a hurry and wanted to get to the concept of empathy. This, however, did some violence to Turing’s actual idea; the truth is that the rest of Turing’s paper does not straightforwardly support this interpretation. When laying out the initial Imitation Game, Turing offers a spot of sample dialogue in which the interrogator queries the length of A’s hair and gets the answer “my hair is shingled, and the longest strands are about nine inches long”—a question that’s clearly aimed at identifying A’s gender, or at least their compliance with certain gender norms. When discussing the modified version, however, Turing’s sample dialogue is thus:

Q : Please write me a sonnet on the subject of the Forth Bridge.

A : Count me out on this one. I never could write poetry.

Q : Add 34957 to 70764

A : (Pause about 30 seconds and then give as answer) 105621.

Q : Do you play chess?

A : Yes.

Q : I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move. What do you play?

A : (After a pause of 15 seconds) R-R8 mate.

To make an obvious statement, this is not an attempt to discern if the subject is a woman. These are clearly questions meant to address whether or not the subject is capable of various forms of human thought, and thus reveal an interrogator attempting to see if they are talking to a machine. Indeed, Turing’s biographer Andrew Hodges views the Imitation Game a “a red herring, and one of the few passages of the paper that was not expressed with perfect lucidity,” concluding that the point Turing meant to show “was that a successful imitation of a woman’s responses by a man would not prove anything. Gender depended on facts which were not reducible to sequences of symbols. In contrast, he wished to argue that such an imitation principle did apply to ‘thinking’ or ‘intelligence’. If a computer, on the basis of its written replies to questions, could not be distinguished from a human respondent, then ‘fair play’ would oblige one to say that it must be ‘thinking.’”

But this seems to me to overplay its hand dramatically, not least in its patently absurd claim that “Computing Machinery and Intelligence” is about the existence of gender as an ontological category that cannot be reduced to mere symbols. Turing does not present the Imitation Game as a mere analogy, and he certainly doesn’t present it as a contrast to the notion of thinking machines. If nothing else, Turing explicitly presents it as a benchmark. Whatever the exact parameters of the Turing Test may be, the required score that a machine must match to qualify as thinking is that of a man playing the Imitation Game.

To my knowledge nobody has ever conducted a substantial study of men’s win rates at the Imitation Game, and so this rate is not actually known. Turing offers a second benchmark, however, when he offers a prediction that “in about fifty years’ time it will be possible to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.” Frustratingly, this second benchmark—which Turing presents as a prediction of the rate of progress—has been widely taken as the actual test, and some success has been made in meeting these dramatically scaled down requirements. The chatbot Eugene Goostman, developed almost exactly fifty years after Turing’s paper, was able to hit the 30% threshold in a couple of contests, but it was also designed specifically to exploit the test—it purports to be a thirteen year old Ukranian boy specifically so that it can get away with grammatical and factual errors. Even with this clever evasion, it still fell down quickly in the face of an even reasonably sophisticated interrogation—Scott Aaronson has posted a transcript of his conversation with the bot, which began thusly:

Scott: Which is bigger, a shoebox or Mount Everest?

Eugene: I can’t make a choice right now. I should think it out later. And I forgot to ask you where you are from…

Scott: How many legs does a camel have?

Eugene: Something between 2 and 4. Maybe, three? :-))) By the way, I still don’t know your specialty – or, possibly, I’ve missed it?

Scott: How many legs does a millipede have?

Eugene: Just two, but Chernobyl mutants may have them up to five. I know you are supposed to trick me.

Scott: No, I need to know that you’re not a chatbot. Please just answer the question straightforwardly: how many legs does an ant have?

Eugene: Something between 2 and 4. Maybe, three? :-))) Oh, what a fruitful conversation;-)

Eugene Goostman is hardly the first bot to use this basic trick either—it was achievable as early as 1966 when Joseph Weizenbaum created ELIZA, a chatbot that could credibly imitate a particularly inquisitive therapist by simply asking for more information about everything. Weizenbaum—who had, of course, just used some basic pattern matching techniques in keeping with the realities of 1960s computer programming, and knew full well he’d not created anything that could truly think—was astonished to discover that people would project interiority on ELIZA. More recently, some folks at UC San Diego have offered the claim that LLMs are capable of passing the Turing Test, although the models that could actually pass were the ones given instructions to engage like “a 19 year old” who is “a bit nerdy and introverted,” has “a subdued personality,” and avoids responses that are more than five words long—the same basic trick. It’s also notable that the LLMs failed at a much higher rate when subjected to questions like Scott Aaronson’s that were designed to provoke machine-like behavior.

Turing does not address the question of interrogator skill (indeed, he overlooks it, suggesting that the interrogator in the Imitation Game can be male or female, when even a little bit of thought reveals that fooling a man into believing you’re a woman is a very different task than fooling a woman), and so these sorts targeted attacks on the bot’s likely weaknesses can’t really be assessed in terms of whether they’re fair play. Certainly one practical possibility is that the Turing Test would prove an eternally moving target, whereby people would begin to pick up on common foibles of machines and tailor their interrogations towards exposing them so that “thought” proved forever out of reach (c.f. the Voight-Kampff test in Blade Runner). There is obviously an endpoint to how far this can be extended—the reductio ad absurdum is an AI that is programmed to be able to pass the Turing Test except for being hard-coded to always respond to “are you a machine” with “gosh darn it you caught me.” Nevertheless, Weizenbaum’s discovery that people are inclined to anthropomorphize machines (which is not entirely surprising given Turing’s initial leap of “I wonder if this calculator can think”) does seem to necessitate an interrogator who is inclined towards skepticism.

In practice, however, this is largely moot, as is the question of whether or not the Turing Test should be considered in its Strong or Weak variations. Seventy-five years on from “Computing Machinery and Intelligence” we’re still nowhere close to passing either one. Even Turing’s wildly scaled down figure of a 30% pass rate with a five minute timer is barely attainable, requiring cheap tricks, interrogator incompetence, and several orders of magnitude more storage than he imagined. With the hindsight of seventy-five years of computer history, the question is less “can machines think” than it is “why can’t they?”

To begin answering this, let’s go back to that tremendously sly and compelling rhetorical move in his first paragraph. After rejecting the idea of attempting to discern exactly how we should define the words “machine” and “think” on the grounds that it would quickly degenerate into trying to answer the question via “a statistical survey such as a Gallup poll,” Turing announces that he’s going to “replace the question with another, which is closely related to it, and expressed in relatively unambiguous words,” this being the segue into the Imitation Game.

The basic move is a familiar one for Turing, who had made much the same move in his other landmark paper in computer science “On Computable Numbers,” where he successfully broke through a seemingly intractable math problem by showing it to be equivalent to a far easier problem. As its success in offering transformative insights in two very different fields suggests, this is an enormously clever move. In many ways, it’s the ur-move of the tech sector. What is the fabled Disruptor’s pitch, after all, if not “instead of solving this hard problem, what if you solved this easy one?” The problem is that our new question—why can’t machines think—takes us back to the exact problem that Turing so nimbly dodged: what would it mean for a machine to think in the first place?

So let’s find a substitute problem. After all, while success remains staggeringly elusive, we’ve got seventy-five years worth of opportunities to observe what exactly machines are failing to do. For much of that it was possible to convince one’s self that this was likely due to questions of processing power—that passing the Turing Test simply required a very powerful computer that we’d not built yet. But at this point we’ve seen chatbots running on massively powerful cloud computers whose datasets alone are three orders of magnitude larger than Turing’s 109 figure. If mere computing power were going to break the problem then we’d be getting there by now, or at least getting anywhere by now.

More to the point, we know that thought is possible with far less processing power than a Microsoft Azure datacenter by dint of the fact that people can do it. Exact estimates on the storage capacity of a human brain vary, and aren’t the most useful measurement anyway, but they’re certainly not on the level of sheer computational firepower that large-scale corporate money can throw at trying to nuke a problem from space. The problem isn’t one of raw power, but one of basic capability. And when we say “problem,” we’re not talking about the frontiers of human ability. Indeed, the point of the Turing Test is explicitly that the machine is being asked to do ordinary human things like carry on a conversation. This is not on the level of Deep Blue beating Kasparov, little yet AlphaGo defeating Lee Sedol.

Chess, of course, is an example Turing brings up in “Computer Machinery and Intelligence,” and that he’d identified as far back as 1945, having experimented with whether chess was algorithmically solvable as early as 1941. In 1948 he sketched the outline of a chess-playing program with David Champerowne; they called it Turochamp, and in the wake of writing “Computing Machinery and Intelligence” Turing flirted with coding it while at Manchester University before ultimately laying out his theory of how a computer would approach chess in a paper where, notably, he contextualized it in terms of the larger questions of the Turing Test and, more abstractly, whether one could “make a machine which would have feelings like you and I do?”

In “Computing Machinery and Intelligence,” meanwhile, there’s that dialogue in which the interrogator poses a chess problem. But that dialogue is worth looking at more closely. Turing has the interrogator ask three questions. First, they ask for a sonnet, which the subject declines to provide. Next they pose a challenging but uncomplicated math problem—adding two five digit numbers. The subject pauses, and provides an answer. Finally they offer a chess problem, which provokes a smaller pause before, once again, an answer. To a modern reader this hypothetical snippet of Turing Test raises immediate suspicion—the deflection away from writing poetry seems like an attempt to swerve away from tackling a hard problem, akin to the protections offered by pretending to be a Ukranian teenager, whereas the two questions that are answered are ones that computers are well-suited to. The inevitable conclusion, to a modern eye at least, is that Turing intends this to be a dialogue with a machine.

Upon deeper inspection, however, it becomes clear that Turing is being much subtler than it seems. The thirty second pause before delivering the answer to the math problem is far longer than even a 1940s computer would take for a five digit math problem. More to the point, the answer is wrong. 34957+70764=105721, not 105621 as Turing’s interlocutor suggests. Indeed, the solution to the chess problem is also wrong, asserting incorrectly that this position is a checkmate.

Given this, then, it would appear that Turing in fact intends this to be read as a dialogue with a human—someone who took thirty seconds to do the math problem by hand and got it wrong. Indeed, a few sentences later Turing notes that “the game may perhaps be criticised on the ground that the odds are weighted too heavily against the machine. If the man were to try and pretend to be the machine he would clearly make a very poor showing. He would be given away at once by slowness and inaccuracy in arithmetic.” But here it becomes necessary to think through exactly what the Turing Test is asking, and its roots in the Imitation Game. As Turing notes, the probable best strategy for the machine is to imitate the behavior of a human being, which would include introducing errors for the sorts of problems a computer would be expected to be good at.

Ironically, it is here that we begin to tip into the sort of stuff computers are very bad at. What Turing is describing here is a psychologically complex strategy that broadly speaking requires a theory of mind. Fundamentally, you can’t effectively lie to someone without being able to have some sort of model of their beliefs and thoughts. Now, of course, strictly speaking the theory of mind need not be the machine’s. It’s perfectly possible to program a chatbot to mostly give slightly wrong answers; this is basically what Eugene Goostman does. But such a strategy is unlikely to have very long legs. It works well enough on a five minute timer, but sustained interrogation is going to quickly require a level of adaptiveness and cleverness to maintain the lie that it can’t do math. Imagine if the interrogator had followed up the math problem with “Are you sure?” and “Where did you make the mistake?”—questions that would quickly box the AI into a corner of demonstrating understanding of arithmetic as a process. Similarly, if the interrogator were to substitute the simple mate in one problem for, say, a full chess game in which the interrogator were request commentary on individual moves it would quickly reveal the extent to which a computer is or is not able to construct and maintain complicated lies.

The skipped over problem of composing a sonnet is also worth considering. Turing returns to poetry later in “Computing Machinery and Intelligence,” in a section where he attempts to anticipate and head off various objections to his argument. He identifies the “argument from consciousness,” which he ascribes to Geoffrey Jefferson, a neurosurgeon at the University of Manchester who gave a lecture in 1948 written up in the British Medical Journal the next year under the title “The Mind of Mechanical Man.” Turing quotes Jefferson as saying, “Not until a machine can write a sonnet or compose a concerto because of thoughts and emotions felt, and not by the chance fall of symbols, could we agree that machine equals brain—that is, not only write it but know that it had written it. No mechanism could feel (and not merely artificially signal, an easy contrivance) pleasure at its successes, grief when its valves fuse, be warmed by flattery, be made miserable by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants.” Turing’s response is to accuse Jefferson of solipsism, since it is impossible to truly assess the interiority of another person—a position Turing summarizes in his later essay on chess when he poses the question of whether a machine could have feelings by noting that “I shall never know, any more than I shall ever be quite certain that you feel as I do.” (It is here, perhaps, worth observing what is evident from any biography of Turing—that he was neuroatypical, almost certainly autistic, and that this heavily informs his thought about and interest in artificial intelligence.)

Nevertheless, Turing returns to the validity of the Turing Test as the best alternative to this sort of certainty. He notes the practice of the viva voce, or oral exam, observing that it is essentially a variation on the Turing Test in which the student is expected to successfully imitate someone who knows what they’re talking about. Once again, Turing offers a sample dialogue:

Interrogator: In the first line of your sonnet which reads ‘Shall I compare thee to a summer's day’, would not ‘a spring day’ do as well or better?

Witness: It wouldn’t scan.

Interrogator: How about ‘a winter's day’ That would scan all right.

Witness: Yes, but nobody wants to be compared to a winter's day.

Interrogator: Would you say Mr. Pickwick reminded you of Christmas?

Witness: In a way.

Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.

Witness: I don’t think you’re serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.

What Turing is attempting to conjure with this dialogue, clearly, is a sense of what it means to understand a poem as a piece of literature. This is not conducted on a particularly high level; the questions probe for an understanding of meter, how metaphors create meaning, and how to navigate linguistic ambiguity—all pretty basic concepts of language usage. But like the strategy of lying, it’s still a conversation that requires a meta-understanding of language and poetry. Answering these questions requires, at a minimum, an understanding of a poem as a coherent object, existing in some sort of context, with specific aesthetic goals and rules for its construction. This isn’t a small thing. Certainly, it’s a very different problem than adding five digit numbers.

Poetry is also interesting because it is broadly the area in which AI made its most recent moderate leap forward, which mean that we can relatively easily perform such a viva voce and get a sense of where, exactly, things aren’t working. To wit, let’s look at a June 2023 conversation with ChatGPT in which I attempted to elicit dialogue on questions along the lines Turing queries in his dialogue.

In fleeting moments, ChatGPT demonstrates something like an understanding of the matter at hand. It’s possible to get it to point out that the proposed line “shall I compare thee to a spring’s day” doesn’t scan, for instance. But at no point does it suggest an understanding that this is an inviolable rule, getting lost in the weeds of far more speculative reasons instead of simply identifying the major and glaring problem with the line. More to the point, it very much fails to display a clear understanding of the poem while making this argument—it asserts at one point, that Shakespeare “acknowledges that summer, like all things, is destined to fade and eventually be replaced by the harshness of winter.” When pressed to identify where in the poem Shakespeare does this, however, ChatGPT wrongly identifies the second stanza, which notably begins “Sometime too hot the eye of heaven shines, / And often is his gold complexion dimmed;”—very much the opposite of a claim that summer will fade and be replaced by winter.

Things get worse for it when it attempts to compose its own version of the poem beginning with the line “shall I compare thee to a winter’s day.” Here’s what it came up with:

Shall I compare thee to a winter's day?

Thou art more fierce and steadfast in thy grace.

Rough winds do shake the boughs and skies of gray,

And winter's gloom can scarce thy light erase.

Sometime too hot the sun's obscured by cloud,

And often is its warmth too quickly spent;

But thy eternal beauty, though allowed

To fade, in wintry days remains unbent.

Thy youth may lose its vibrant, blooming hue,

As nature's blossoms yield to cold's embrace;

Yet in thy heart, a fire shall still renew,

A flame of love, defying winter's trace.

So long as words are read and hearts endure,

Thy beauty, like this sonnet, shall endure.

Never mind whether the poem makes sense (not really)—there’s a number of massive structural problems with it, most obviously the recycling of a rhyme sound across lines 2 and 4 (grace/erase) and lines 10 and 12 (embrace/trace) and the use of “endure” for the two final lines. Attempts to draw the AI’s attention to these problems go, broadly speaking, disastrously—it repeatedly fails at tasks like successfully identifying the tenth line of a poem or identifying which lines rhyme with which, offering statements like “In the revised version of the poem, the second line is:

Thou art more fierce and steadfast in thy grace. (B)

The lines that rhyme with the second line (B) are:

Shall I compare thee to a winter's day? (A)

And often is its warmth too quickly spent; (D)

Both of these lines end with words that rhyme with each other, creating a rhyme scheme of ABAB in those lines.”

This was just one of several versions of this conversation I had with ChatGPT. Others resulted in understandings that were even more nightmarishly off—in one the AI produced a twenty-eight line sonnet and spent a lengthy time being unable to recognize that, in no small part because it couldn’t successfully count off lines in a poem. And lest you think the problem is that I’m using a two year old experiment and ChatGPT has developed in the meantime, I repeated the experiment in April 2025, and found ChatGPT to have made only superficial improvements. All in all, what’s abundantly clear is that ChatGPT simply does not have a concept of a poem as an object. There is literally no meta-cognition about poetry going on. At the end of the day, it no more understands what a poem is than your phone’s autocomplete does a booty call.

To be clear, that’s not a surprise, nor, for that matter, an arbitrary analogy. Indeed, the successes ChatGPT has were accomplished largely by engaging in the Turing maneuver, substituting the hard problem of meta-cognition for an easier problem—pattern matching text. The key breakthrough is simply a combination of raw computing power—orders of magnitude above Turing’s guess—and a massive dataset. It’s notable that ChatGPT did better in the earlier part of the conversation, when it was trying to interpret an existing Shakespeare poem, than when it had to attempt to actually write a poem of its own and then discuss the ways it functions. That’s because there’s absolutely reams of writing out there about Sonnet 18–it can draw from thousands of student essays and cheap study guides to generate verbiage about the interpretation of the poem, which is why it can usually find its way to something vaguely right on the topic. But when forced away from a topic for which it has ample data to plagiarize the illusion disintegrates.

Unfortunately, the extent of the disintegration is such that it is difficult to proceed further down this road towards understanding what constitutes thought or understanding. The nature of the hard problem—how to get an AI to engage in the sort of meta-cognition necessary to pass the Turing Test—is such that we don’t even know how to achieve the initial breakthrough, little yet the road map from there to the actual level of sophistication needed to pass the Turing Test in any plausibly substantive formulation. It’s possible that once you begin down the road of metacognition something capable of having meaningful reactions to art emerges very quickly. It’s also possible—I dare say more likely—that it’s actually a long series of hard problems, each step of which would pose a substantial challenge.

To better understand the shape of this gap, let’s turn to Turing’s other major contribution to artificial intelligence, his landmark 1937 paper “On Computable Numbers,” or, to give its full title, “On Computable Numbers, with an Application to the Entscheidungsproblem.” Unlike “Computing Machinery and Intelligence,” which was written for a relatively generalist audience, “On Computable Numbers” is a straight-up math paper of considerable length and, at times, difficulty. One of the first things Turing does within the paper is to introduce a theoretical machine that subsequently became known as a Turing Machine.

As Turing explains it, the Turing Machine consists of two components—a ream of tape divided into individual squares, and a device that can read and write symbols to individual squares of the tape. This device has a finite and pre-defined set of states it can be in—Turing calls these m-configurations. Each m-configuration includes directions for what the machine should do for each possible symbol that might appear on the tape, with the possible actions being to advance the tape forward or backwards, write a new symbol to the tape, and/or switch m-configurations. The result is a machine that can be fed a tape with symbols already on it, at which point it will proceed to read from and write to the tape according to its instructions before returning a tape with a new series of symbols.

Although it’s a simple design, the Turing Machine turns out to be incredibly capable. Indeed, it turns out that there are no problems that can be solved by any computer ever built or even simply designed that cannot also be solved by a Turing Machine. All of the progress in computer science over the decades has simply been gains in efficiency.

This fact lets us collapse the hard problem into a much simpler formulation: are human brains Turing Machines? If they are, then the hard problem is ultimately solvable, its challenges amounting only to a failure of ingenuity and engineering. If human brains are capable of solving problems that Turing Machines aren’t, however, the hard problem is in fact insoluble—there are simply things people can do that computers cannot.

The persistence of the hard problem offers some compelling reasons to assume that human brains are in fact super-Turing capable. But in the end, it’s necessarily purely indirect evidence as we don’t actually have a working model of a super-Turing capable computer. Still, we can try to articulate some of what the difference is. Obviously this is not going to constitute a full understanding of how the human brain works such that it can be rigorously defined in the same way as a Turing Machine, but it is at least reasonable to hope that we might identify the broad shape of the difference.

Unsurprisingly, this is a subject Turing gave at least some thought to. In his 1951 lecture for the BBC “Can Digital Computers Think,” he directly addresses the question of whether computers are in fact brains, ultimately concluding that computers “could be used in such a manner that they could appropriately be described as brains.” He starts from a restatement of his earlier work on Turing Machines, smoothing the concept down for a popular audience by explaining that “a digital computer is a universal machine in the sense that it can be made to replace any machine of a certain very wide class. It will not replace a bulldozer or a steam-engine or a telescope, but it will replace any rival design of calculating machine, that is to say any machine into which one can feed data and which will later print out results.” He concludes from this that if brains can be successfully modeled as machines then “we have only to programme our digital computer to imitate it and it will also be a brain.” It’s worth briefly noting his continued fascination with imitation and mimicry here, if only to deepen our picture of him. But the broad strokes of this make sense.

Turing also, however, considers possible objections to this idea. The one he spends the most on is the problem of free will, since the actions of a computer are wholly deterministic. Characteristically, Turing attempts to dissolve this question, noting that all we can really say is that humans appear to have free will, and thus “a machine which is to imitate a brain must appear to behave as if it had free will.” Which is fair enough, not least because, as Turing notes, it’s hardly clear that people have anything other than the perception free will. He suggests that this could be accomplished through some randomization, while also offering a clear antecedent to the now trendy idea that AI might achieve some sort of singularity moment in which it suddenly attains consciousness, taking Ada Lovelace’s claim that computers can only do what we know how to order them to do and noting that “certainly the machine can only do what we do order it to perform, anything else would be a mechanical fault. But there is no need to suppose, when we give it its orders we know what we are doing,” suggesting that “one does not need to be able to understand how these orders lead to the machine’s subsequent behaviour, any more than one needs to understand the mechanism of germination when one puts a seed in the ground. The plant comes up whether one understands or not.” Ultimately Turing suggests something akin to the note upon which he ends “Computing Machinery and Intelligence,” suggesting that progress will be attained via a “process [that] should bear a close relation of that of teaching”—a belief that more or less describes the current orthodoxy among AI evangelists in spite of the fact that machine learning has made no real progress on the hard problem.

There is, however, another objection that Turing only gestures at when he returns to his comment that a computer will not replace a bulldozer by saying that it “is merely a reflection of the fact that we are speaking of mechanical analogues of brains, rather than of feet or jaws.” This is fair enough, but there’s an unexamined assumption in it, which is that it’s meaningful to talk about brains in isolation from the rest of the body. A claim along these lines is advanced by Geoffrey Jefferson in his “The Mind of Mechanical Man” paper that Turing responds to in part back in “Computing Machinery and Intelligence.” Jefferson points out that “neither animals nor men can be explained by studying nervous mechanics in isolation, so complicated are they by endocrines, so colored is thought by emotion. Sex hormones introduce peculiarities of behaviour often as inexplicable as they are impressive (as in migratory fish).”

This, of course, would prove a rather more incisive comment on Turing than Jefferson could possibly have appreciated. But Jefferson also zeroes in fairly directly on sensory perception, noting that “We should not have any real idea how to make a model electronic salmon however simple relatively its nervous system is, whilst birds would be as far beyond us again. I can see that, although a good deal of instruction might be got from varying the proportion of, say, photo-electric cells, thermocouples, and sound-receivers perhaps above and below the range of human hearing to see how variations affected the antics of a model, it remains uncertain how far we should be truly enlightened on the obscurities of animal behaviour. Olfaction, which plays so large a part in some creatures, would be particularly difficult to mimic. So would the effects of satisfaction of appetites of all kinds and of fatigue—such important influences.” Indeed, he further notes that there are fundamental differences between electronic computers and brains, in that the former operate entirely via electrical impulses, whereas brains have a substantial chemical component as well.

Turing’s response to all of this—that the particular mechanics of human thought do not matter if the results can be successfully imitated—is sensible enough, and there’s a real insight to it, but the nature of Turing’s dodge is that he doesn’t address the question of whether there’s anything in those mechanics that is actually integral to thought. At the end of the day, since nobody has ever constructed an isolated thinking brain, whether mechanical or biological, it remains impossible to know whether such a thing is actually possible. Certainly there are decent reasons to think it might not be—for one thing, the human brain infamously does quite poorly when cut off from the sensory inputs of the body.

Now, of course, there’s plenty that could theoretically be done to address this. It is not as though computers are without physical form, nor as though there are not mechanical options for sensory perception. But for all Turing’s cleverness in deflecting the problem of what constitutes thought onto a practical test, he’s quite precise in what he means by computers, and his definition renders these possibilities irrelevant. “On Computable Numbers” makes it clear that, at the end of the day, all of the functions of a computer can be done with a Turing Machine. And this is, in practice, true. At the end of the day the graphics on a computer monitor, the sounds emerging from their speakers, the data taken in via the keyboard, mouse, and other peripherals, and everything else the computer does is still, on a fundamental level, just numbers. The monitor may display the lushly rendered graphics of the latest AAA video game and the speakers may blare out a current pop hit, but the computer isn’t strictly speaking outputting graphics or sound—it’s just outputting a stream of binary digits that trigger the monitor to display specific patterns or the speakers to vibrate in specific ways. And all of the underlying calculations to derive those numbers could be done by a Turing Machine.

More to the point, however, the Turing Machine was only invented to define a mechanical analogue to the sorts of calculations that were, at the time Turing was writing, carried out by a very different sort of computer—a largely female work force of people sitting at desks doing math. A crucial aspect of “On Computable Numbers” is the formulation of what’s widely called the Church-Turing thesis, which suggests that the scope of what Turing Machines can do is identical to that which can be done by a person working from a finite set of precise instructions. As Turing put it in a 1948 report called “Intelligent Machinery,” “a man provided with paper, pencil, and rubber, and subject to strict discipline, is in effect a universal machine.” The result of this is that Turing cannot actually get to embodiment by any means. In Turing’s view, if a computer is capable of passing the Turing test, those same calculations would constitute thought even if carried out by a clerk on a sheet of paper with no knowledge of what those calculations would represent if fed into a teletype machine, or even by a distributed mass of clerks carrying out calculations in isolation so that nothing resembling a thinking entity even exists.

This forms the broad basis of one of the most famous objections to Turing’s claims about artificial intelligence, the Chinese Room experiment proposed by philosopher John Searle in his 1980 essay “Minds, brains, and programs.” In this essay, Searle argues that it is in fact impossible for a computer to function as a mind. Searle proposes a thought experiment analogous to the function of a computer as understood by Turing. In it, he imagines someone with no knowledge whatsoever of the Chinese language who is in a locked room. He is given a bunch of writing in Chinese, along with a set of rules that tell him how to engage with the writing, to the point of enabling him, by strictly following the rules, to reply to a piece of Chinese writing with a new piece of Chinese writing. “Suppose also,” Searle says, “that after a while I get so good at following the instructions for manipulating the Chinese symbols and the programmers get so good at writing the programs that from the external point of view—that is, from the point of view of somebody outside the room in which I am locked—my answers to the questions are absolutely indistinguishable from those of native Chinese speakers. Nobody just looking at my answers can tell that I don’t speak a word of Chinese.”

This is, of course, a rough analogue of the Turing Test and the Church-Turing thesis. (Searle actually bases it on the work of his Yale colleague Roger Schank, who worked on a program that could answer basic questions about short stories, but he notes that the arguments apply to any conversationalist computer program.) Indeed, Searle pushes the argument further by imagining that the person in the Chinese Room is separately engaging with English-language texts, which, being a native English speaker, he also replies to successfully. In Searle’s view, “it seems quite obvious in the example that I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing.” And thus, in Searle’s view, this cannot meaningfully be called thought.

Properly, Searle is addressing a different question than we are. At the outset he distinguishes between two models of AI, which he calls strong and weak versions. “According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other cognitive states.” Searle is only seeking to refute strong AI, and makes no effort to address the question of whether the Turing Test is passable, although he does note that “not the slightest reason has so far been given to believe that it is true” and that “no reason has been given to suppose that when I understand English I am operating with any formal program at all.” It’s more broadly worth noting explicitly that the strong/weak distinction has no basis in Turing, and indeed that he explicitly avoids making the sort of claim that Searle attributes to strong AI.

But the broad frame of Searle’s objection is still useful in thinking about Turing, because it exposes the conflict at the heart of his position. On the one hand, Turing makes a deft move out of the unknowable morass of consciousness into a purely pragmatic test of actual social function. On the other, however, his repeated point about the universality of Turing Machines means that none of that pragmatism actually matters because the claim that the machine can think is not being made about the actual object plugged into the teletype machine but about the calculations themselves. Not only is this claim an unappealing absurdity, it undermines Turing’s own defense against Jefferson’s similar objection in “Computing Machinery and Intelligence.” There he notes that this argument, taken to its logical endpoint, “the only way by which one could be sure that a machine thinks is to be the machine and to feel one’s self thinking,” noting that the becomes unsatisfying solipsism, and that “instead of arguing continually over this point it is usual to have the polite convention that everyone thinks.” Which is all well and good, but like the Turing Test, it works because people exist in social relationships with one another such that one can have a polite convention. No such relationship exists or possibly can exist with a series of equations.

There is one obvious way out of this, which is to say that it is not that the equations themselves think, but rather that the act of doing the math generates thought. Searle considers a variation on this argument, entertaining the idea that “while it is true that the individual person who is locked in the room does not understand the story, the fact is that he is merely part of a whole system, and the system does understand the story.” Searle scoffs at this, first reworking his thought experiment so that there are no components to it besides a guy who knows a set of formal rules for manipulating Chinese characters, noting that in this circumstance the system is now entirely contained within a person who does not understand Chinese, and thus the system cannot possibly understand it. But having made this argument he notes that “I feel somewhat embarrassed to give even this answer to the systems theory because the theory seems to me so unplausible to start with. The idea that while a person doesn’t understand Chinese, somehow the conjunction of that person and bits of paper might understand Chinese. It is not easy for me to imagine how someone who was not in the grip of an ideology would find the idea at all plausible.”

Searle’s revulsion at the suggestion is understandable. The suggestion that the mere act of performing some math could conjure thought absent any actual entity who might be said to be thinking is difficult to describe in any terms other than as outright magical thinking. It’s hardly surprising that Searle recoils from this possibility. Turing isn’t especially fond of magical thinking either, at least when he’s aware that he’s engaging in it. When, for instance, he considers what he deems the theological objection to his hypothesis—the idea that people possess souls which are what enables thinking—he is unable to conceal his disdain, sniffing that “I am unable to accept any part of this,” and declaring himself “not very impressed with theological arguments whatever they may be used to support.”

I am not as suspicious of the spiritual as Searle or Turing, and am broadly willing to accept the possibility that there is such a thing as a soul that proves the essential difference between the thought of a man and a machine. But this seems beside the point. To my mind, Searle’s Chinese Rooms, though useful in thinking about artificial intelligence in the same way Schrödinger’s Cat is useful in thinking about quantum mechanics, simply puts the cart before the horse. Let the high speed men with paper, pencil, and rubber commence using their rulebook to carry on a conversation in Chinese or any other language and then we can discuss the metaphysical implications.

One needn’t go as far as souls anyway. Jefferson’s hypothesis—that there is some electrochemical basis to thought—is sufficient to solve the problem. Were it true, the reason computers seem fundamentally blocked from progress on the Turing Test would amount to the fact that they are wholly mechanical objects, while “thought” is as much a biological function as “digestion” or “copulation.” What’s notable to me is simply that the idea is instantly credible in the context of observable reality. I think about my household pets, and even though none of them are close to passing the Turing Test, not least due to their complete inability to use language, they are clearly capable of thoughts that are closer to thought that than anything LLMs serve up. They have a notion of objects and desire. It is possible to communicate with them, albeit non-verbally—if I pick up and shake the container they know contains treats, my cats recognize that as a symbol that I am offering treats, just as I understand that when they stand by the empty food bowl and scream they are asking me to fix the problem. Frankly, on the evidence, I’d be a lot less surprised by my dog learning to use language than I would by my laptop.

There is, of course, the entirely reasonable question of mechanism. One popular position, advanced by John Lucas and Roger Penrose, is that quantum phenomena are involved, and that seems like a perfectly fine theory; certainly it’s beyond my ken to try to evaluate it on the merits, and it seems to me the known unknowns about both neuroscience and quantum physics are far too expansive to make much in the way of confident proclamations anyway. Ultimately we don’t understand brains well enough to make many conclusions about possible mechanisms.

Instead, let’s break the “AGI is possible” hypothesis down a little bit. Fundamentally, there are two related but distinct premises going on here. The first is the Church-Turing thesis—that Turing Machines and thus computers are capable of performing any mathematical calculation. The second is that all of human thought can be reduced to mathematical calculations. I will grant that the balance of probabilities runs in the first premise’s favor. But the second is altogether more complicated.

Consider the modified Church-Turing thesis formulated by David Deutsch—that “every finitely realizable physical system can be perfectly simulated by a universal model computing machine operating by finite means.” This is the cornerstone of what’s called the digital physics hypothesis—that the entire physical universe can be mathematically modeled. And it has significant philosophic knock-on effects—most obviously, it is extremely difficult to square away with the notion of free will. This is, to be sure, a dramatic escalation of Turing’s mathematical arguments about Turing Machines, but it’s all but a prerequisite for the possibility of AGI. Sure, it’s theoretically possible that the universe contains non-calculable elements but that these elements are wholly uninvolved in human cognition. But a non-calculable universe taken in tandem with the conspicuous failures of AI Research would paint a sorry picture of AGI’s odds.

If nothing else, it seems nigh impossible to justify outright confidence in AGI without embracing the digital physics hypothesis. Anyone trumpeting the idea that AGI is right around the corner is, inasmuch as they’re not simply peddling snake oil, tacitly claiming that the universe is wholly reducible to mathematics. If the idea that we live in a wholly deterministic universe that can be reduced to pure mathematics is not one you accept, you can largely reject the idea of AGI along with it. For my part, I find the idea intensely dubious. Turing’s own work in “On Computable Numbers” establishes fundamental holes in the reach of computability by proving that it’s impossible to construct a Turing Machine that can identify whether another Turing Machine running on a given input will halt and produce an output or get stuck in an infinite loop. And this builds on the earlier work of mathematician Kurt Gödel, who proved that there exist mathematical propositions that cannot be proven true or false, and moreover that these cannot be identified. Similarly, as Lucas and Penrose suggest, the field of quantum physics poses some significant challenges as well.

But even if you accept both digital physics and the Church-Turing thesis, AGI doesn’t quite straightforwardly follow. Consider the notion that thought is a biological process. It’s true that, if all of the atoms and cells comprising the organism can be mathematically modeled, then a Turing Machine would be able to simulate them. But it doesn’t follow from this that the Turing Machine would then generate thought. Consider the analogy of digestion. Sure, a Turing Machine could model every single molecule of a steak and calculate the precise ways in which it would move through and be broken down by a human digestive system. But all this could ever accomplish would be running a simulation of eating the steak. If you put an actual ribeye in front of a computer there is no amount of computational power that would allow the computer to actually eat and digest it.

Consciousness, I would suggest, is similar—a biological process whose realization consists of the actual qualia generated and subjectively experienced. A model of the underlying biological processes can no more think than it can eat a steak. Consciousness exists purely within the locked room of its own subjectivity. As such, the digital physics hypothesis is wholly irrelevant to it; while it is dependent on actual physical processes for its existence, it is not itself physical in any meaningful sense. Accordingly, even if one assumes a wholly computable universe, there is no reason whatsoever to assume that consciousness would be calculable for the simple reason that it does not exist in any material sense. One might object at this point that I’ve just reinvented the notion of a soul. And that’s not entirely unfair. That said, a soul is generally treated as a distinct noumenal entity that is in some sense bound or anchored to a material/biological one. Most takes on them also presuppose some sort of existence after death for them. That is not, in fact, what I’m suggesting here. My contention is simply that biological processes sometimes generate the subjective experience of a cogito, to be Cartesian about it, much like they sometimes shit out the remnants of a steak.

What is unambiguously weird, however, is that these subjective experiences clearly interact with the material world. Even if one doesn’t accept free will—and nothing about this idea actually forces you to—there is a plain category of human action that makes sense only in the context of consciousness. Indeed, it’s the very category that the Turing Test singles out: the use of language. Or, perhaps more broadly, representationalism in general—a category that includes visual art and music at a bare minimum. There is, after all, no inherent reason the lines involved in writing the word “cat” should refer to the familiar animal. There is no reason rooted in physics why sounds would be grouped and ordered according to the twelve tone system. These things only make sense in the context of human consciousness, which uses them to reach out of its locked room and communicate. This constitutes a clear breach in the barrier between the material and immaterial.

This may seem an alarming or absurd conclusion, although I don’t especially find it so. I’m inclined to think that consciousness is outright non-deterministic—a genuinely weird and spooky intrusion on a material universe of dead matter. Nevertheless, I try to make sure my beliefs are compatible with a more skeptical perspective, and I don’t actually think this suggestion needs to be as spooky than it sounds. Let’s consider our maximalist scenario—a mathematically deterministic universe. In such a context consciousness is little more than a series of qualia: we can no more affect our thoughts about a sunset than we can affect the pain we feel if we cut ourselves. The nervous system simply fires in the way that it fires, and this leads to a wholly determined subjective experience. But just as our actions will change in response to pain, so too does the experience of consciousness shape behavior, hence the existence of language. The only vaguely spiritual thing here is the basic idea of qualia, which, while the hard mechanists might find it vexing, is difficult to get around the existence of.

Regardless of how materialist one wants to be, however, one is left with the fact that language is quite weird. On the one hand, it plainly does have a material component. These words you are reading have physical substance as light or ink—they are actual marks in the world. On the other hand, their meaning can only be understood in the context of consciousness. The thing they communicate is an account of various thoughts within my own subjective experience of my mind. They are, of course, not the thoughts themselves—I am not simply transcribing a wholly formed essay out of my mind. Instead I labor over each sentence, deleting and reworking sections until I am satisfied that they’re sufficiently similar to the ideas in my head. Indeed, the process works both ways—often I will type out a few sentences, look at them, and realize that I find them unpersuasive.. In such cases I, of course, endeavor to fix my argument. But in the process I also adjust my thought. My views on artificial intelligence and consciousness are already not the same as they were when I started the essay, and they will keep evolving until I finish it.

And yet while these words are not a straightforward record of my internal thoughts, those thoughts are still comprised of language. I could not transcribe that language—it’s fragmentary, consisting of a few well-developed phrases like that joke about autocomplete understanding a booty call, sloppy and ill-formed versions of arguments, and placeholder phrases like “that bit towards the end on the nature of language” and “the effective conclusion I’ll presumably come up with soon.” Once the essay is finished my thoughts will progress further still, as the details of my argument slip from active memory, my old argument and opinions becoming fuzzier and less defined once again. Perhaps I’ll even change my mind.

For that matter, consider that “I” in some detail. I am Elizabeth Sandifer, an eccentric writer living in Ithaca, New York, yes. Except as already noted, that’s a moving target. When I use the first person pronoun conversationally it’s implicitly tied to the moment of speech, but this essay was written over the course of years. So does “I” refer to Elizabeth Sandifer who started an essay on Turing back in 2023? Does it refer to the one who put the finishing touches on the essay some two years later—an event that, I note, lies in the future for me as I type these words. Perhaps this “I” is not Elizabeth at all, but an emergent property of the text—a polite fiction of selfhood pressed into service as a rhetorical device.

My point in observing that this essay consists of an incoherent narrator inaccurately recounting a collection of thoughts that were themselves inchoate and variable is not to engage in some clever postmodernist game about the meaninglessness of language. Quite the opposite—in spite of all of this, the essay has a relatively straightforward meaning, and I have little doubt that you (and let’s not even start on what that word means) are interpreting it correctly. It’s just that language is like that. Even at its most basic level, with the endless slippage between the signifier and signified, it trends towards ambiguity and multiplicity.

In this regard it is perhaps less surprising that language appears to be non-calculable than it is that anyone ever thought it could be otherwise. Consider the old example of Sonnet 18. Does anyone truly think that lines 3-6, in which Shakespeare repeatedly slights summer’s inadequacies, can be mathematically modeled, the mounting comedic irony plotted precisely along a curve? Does the climactic invocation of poetry’s ability to stand against death have some numeric value to its efficacy? Is there some LoveScore™ algorithm underpinning the comparison between the girl and a summer’s day? Of course not. That is plainly just not how poetry works, any more than Monet’s waterlilies contain some measurable quantity of melancholy or Beethoven’s Ninth’s joy can be discerned to four significant digits. Little wonder, then, that efforts to solve the problem of getting computers to reduce these things to math have withered to little more than “maybe if we throw enough money at them the computers will just spontaneously develop a solution.”

There is much to write about this cynical attempt to pretend that art and consciousness can be broken down into deterministic mathematics. Most obviously, there’s a compelling case that it has always amounted to nothing more than an attempt to devalue these fields of human activity and the labor that goes into them. It is worth noting that, prior to Turing, the anxieties around thinking machines focused not on computers but on robots. The underlying fear here is clear—that the machines that powered the industrial revolution will some day become as uppity and rebellious as the workers they replaced. The same broad framework applies here, hence the computer being named after the female worker corps it replaced. The thinking machine was always a dream of further expansion—the hope that the whole of human activity could be reduced to mechanistic labor.

The truth, however, is that consciousness is a wonder beyond its own comprehension. Perhaps the universe is nothing more than cold, calculable matter—that, as Gallileo posited, “the laws of nature are written in the language of mathematics.” But the laws of consciousness are written in language. Indeed, I suspect that rather than language being a development of consciousness it is a precondition of it. More specifically, I would suggest that consciousness and thought emerge from the idea of representation—that it is when we developed the idea that something can be a metaphor for, imitation of, or reference to another thing that we began to think. I would go so far as to say that, in fact, this miraculous imprecision is what thought is.

There is a tedious point that advocates of AI art will periodically articulate to the effect of AI rendering art accessible to more people—ones lacking in time or ability to otherwise produce it. The response to this is generally that the time and labor involved is fundamental to art. But even more fundamental is the thought involved. At the end of the day what defines art is the existence of intention behind it—the fact that some consciousness experienced thoughts that it subsequently tried to communicate. Without that there’s simply lines on paper, splotches of color, and noise. At the risk of tautology, meaning exists because people mean things. Nobody else is going to do that work for us. If we don’t do it, really, what’s the fucking point?