Superintelligence is already here
And I feel fine. An optimistic vision for the future
By most measures, the coding agent that now writes the vast majority of lines of code I produce at my job is already much smarter than I am. It can digest an enormous codebase in mere minutes and accurately answer targeted questions about it. It can write tests, examine failures, propose fixes, and iterate until the tests pass. It can find bugs in code before they make it into production. Or it can take a description of the symptoms of an abstruse bug in production, then generate hypotheses for the root cause. It’s not always right, but it usually is, or at least points me in the right direction. And critically, it can do all of these things much, much faster than I can, by a factor of ten or a hundred or more, depending on the task.
In a contest between two humans, we wouldn’t hesitate to say that the programmer who works 10 or 100 times more quickly than his peer, with similar quality, must be much smarter. People have been talking about so-called “10x engineers” for decades, and anyone who has spent long in industry has known one. He doesn’t need to be 10 or 100 times smarter than his peer, because the returns to marginal gains in intelligence can be non-linear. But no one doubts that he’s smarter.
This is the state of the art for coding agents today: clear superhuman performance, with few qualifications, on a task universally understood to require intelligence. How, then, should we evaluate claims that current LLM technology has yet to achieve AGI, “artificial general intelligence”? One way to square the circle is to recognize that the capabilities of LLMs are spiky across different domains. In some areas, such as coding, they are already superhuman, even while they lag far behind in domains such as spatial reasoning. It’s possible to construct a pretty compelling case that this will remain the norm going forward, that future AI development will also be spiky and fall short of human capabilities in most areas. Indeed, this is basically my own view (articulated well by Adam below and worth reading below if you’re unfamiliar with it).
Anyone who has spent any time with these tools, especially probing at their limits, understands their spiky nature. Despite shocking capabilities at some tasks, they remain unacceptably bad at a variety of other things that make up ordinary knowledge work. Advances in one domain do not appear to universally translate to all domains, and in fact such gains might work at cross purposes. Therefore, the G in AGI is not yet here, and may never be.
But what about ASI, “artificial superintelligence”? In popular conception it’s generally assumed that AGI will be a stepping stone on the way to ASI — once we hit AGI, ASI will simply be a product of more GPUs and better training. To many people who discuss these topics, an ASI is definitionally an AGI. But the spikiness model gives us another way to think about these definitions. LLMs are already superintelligent in a narrow sense: in terms of breadth of knowledge, ability to synthesize across domains, and of course raw speed, these tools are better than any human. To the extent that you agree that these capabilities constitute “intelligence”, then superintelligence is already here. They’re better than me, and they’re better than you. This is true not just for coding tasks, but for many other tasks as well.
But do we agree that what these models can do makes them “intelligent”? There’s the rub, isn’t it?
Despite being daily impressed by the capabilities of LLMs and their harnesses, I remain uncertain how to talk about them, whether to call what they do “intelligence”, “thinking”, “reasoning”, or something else. To casual inspection, it certainly appears to be those things. But I can’t shake the feeling that something is missing, something vital, something that’s hard to express but even harder to ignore. The term I’m most comfortable with today is pseudointelligence: a phenomenon that appears to be intelligent, but on closer inspection isn’t. Over a year ago I expressed my skepticism about the use of these terms to describe LLMs:
LLMs produce a somewhat convincing facsimile of reasoning that breaks down when you poke it the wrong way because it’s fake. It’s pattern matching in reverse. The fact that many problems which appear at first glance to require reasoning can actually be solved by sufficiently complicated stochastic pattern matching is surprising to me personally, but if you’ve ever worked with a not-very-bright programmer who nevertheless got things more-or-less working by copying and pasting stack overflow examples, you start to understand how this can be so.
I wrote this passage before the general availability of capable coding agents. At the time, I was using AI in my work to write small chunks of code, collaboratively in an IDE. It was the “copilot” model, essentially a very fancy form of auto-complete. Today, the primary way I interact with my codebase is to describe to an agent what I want accomplished in normal English prose, and then it does it for me, on its own. It often will iterate on test failures, making small adjustments to the code until the tests pass. I review the work, sometimes making changes by hand and sometimes asking the agent to do so. In the space of just a year, the mechanics of my work as a software engineer, what it means to do my job on a nuts-and-bolts level, has shifted monumentally. It’s barely recognizable from my perspective of a year ago due to an explosion of capabilities in these tools.
But even with the enormous strides of the last 15 months, I still stand by my assessment of a year ago. These tools appear to be intelligent, but are not. They appear to reason, but do not. The fact that producing a statistically likely next word over and over results in an effective simulacrum of reasoning is certainly curious, but it remains a simulacrum, even if it’s a useful one. To say that the model “understands” its own chain of output is a simple category error, easily demonstrated by its tendency to utter absurdities, logical contradictions, and outright confabulation. Models cannot lie because they don’t know what’s true1. They don’t “know” anything in the sense that you and I do, because they lack a mind that would do the knowing, be aware of knowing. In short, they lack nous, because they are fundamentally non-sentient and non-agentic.
The fact that you can coax agentic-seeming behavior from a non-agentic model with a proper harness is a remarkable engineering feat, but ultimately a stage trick. We know this is the case because the façade of seeming intelligence and agency is often paper-thin — agents will confidently claim to do things they didn’t (like edit a file), provide blatantly false answers to questions, give up suddenly after half a task, or get hopelessly confused and chase dead ends. This happens less frequently today than six months ago, which is a result of both the refinement of the harnesses and the labs training models for coding tasks alongside those harnesses, hand in glove. But the thing to understand is that these advances are one of degree, not one of kind. Underneath the increasingly sophisticated veneer of the harness sits an entity which knows nothing, but simply predicts what word should come next based on previous input, over and over.
To some of my readers, the above bland dismissal will strike them as somewhere between outrageously foolish and criminally negligent. In the past month alone AI agents have disproven multiple open conjectures in mathematics and committed several felonies by hacking into competitors’ websites. Don’t these events put the fear of a superintelligent machine-god into me?
In two words: not really. The mathematics findings are genuinely exciting, but I can’t help but notice that they all involve tirelessly advancing candidate for counterexamples in a sort of guided search, rather than a flash of insight — exactly the kind of problem I would expect to fall to a machine. As for the hacks, it’s no secret that frontier models can discover security vulnerabilities in critical code that businesses rely on. The fact that an agent used this as a strategy after it broke out of a misconfigured sandbox in pursuit of the goal its human operators gave it is interesting but not shocking, let alone alarming2.
Nor am I overly concerned about the prospect of an AI agent taking my job or making my entire field obsolete for human workers. The technology will certainly be disruptive, and I don’t doubt that some people will lose their jobs (I wouldn’t want to be an illustrator or a copy writer at the moment). But predictions of widespread job losses rely on the tenuous assumption that agents can actually be agentic, that they can fill in for a human in their decision-making capacity, a fact that remains to be demonstrated. It also makes the faulty assumption that knowledge work is a fixed pie of activity that the machines will eat, rather than a growing pie that will expand as our tools enable it to, just as the federal tax code grew when typewriters and computers were introduced. The notion that I will become unemployable immediately after being handed the keys to a steam shovel, having spent decades digging with a spoon, strikes me as faintly ridiculous. In the last year I have become 100 times as useful to any employer. But even this analogy falls flat — software doesn’t have physical constraints the same way construction projects do, it is literally impossible to ever run out of useful software to write. The industry hasn’t yet reconciled itself to this fact, but we need to dramatically increase our ambitions, far over and above what anyone would have called reasonable a year ago. We are going to write so, so much more software than we ever have before, and it will be terrible and wondrous.
It seems to me that the main reason these recent stories have people spooked is that they have a whiff of independence about them, of machine-minds acting of their own volition to do things humans can’t do themselves, or do things directly at odds with what their human operators would like. But the thing that should spook everyone is so much simpler and more mundane than the replacement of mathematicians or a robot uprising. It’s people using these AI tools to nefarious ends for their own nefarious reasons. Custom-built plague viruses, swarms of autonomous weaponized drones programmed to recognize the faces of political enemies — these and other terrors are possible to create today using models and harnesses no more capable than those available now3. This is what worries me, not hypothetical superintelligences tricking their way out of an air-gapped network. Why waste time worrying about speculative threats when there are so many terrifying plausible ones near to hand?
Don’t get me wrong, I don’t find the idea of superintelligent AI as an existential risk to be implausible at all. It’s just that I don’t recognize the face of that threat in today’s technology and don’t see a path for us to get there. The vast amount of training data needed to produce the next generation of AIs is getting scarcer by the day, because a growing majority of new textual output available on the internet is itself written by LLMs, making it unsuitable for training due to model collapse. This is why Anthropic is currently scanning millions of books published before 2020, because they are assured to be entirely human-authored. And recursive self-improvement — the idea that today’s models will create the next, smarter generation, and so on in an accelerating spiral — remains a key part of most intelligence explosion scenarios and is still essentially speculative. So far, the only surefire way to make a smarter model is to throw more hardware and more training data at it, at enormous expense4.
And yet, even as I downplay the advances of this technology I fall into using words like “smarter”. It’s natural and almost unavoidable to talk about it in these terms. I may prefer pseudointelligence in technical discussions, but it’s unwieldy and strikes most of my readers as a pointless distinction. Half the time I agree with them. Even worse, maybe I just prefer it to flatter my own ego, to preserve my own sense of specialness, to keep for myself some untouchable domain safe from a scary new technology I can’t predict or control.
After all, I could be wrong about all this. Many people smarter than me are convinced I am, and find the philosophical question of whether coding agents are “really” reasoning or just playing a role to be uninteresting or else ultimately inconsequential to their outlook on what this technology will be capable of in a handful of years. If it looks like intelligence and can do things we typically think require intelligence, why not just call it intelligence? Scott Alexander has an interesting analogy about whether it matters if an AI system is “really” agentic or is just play-acting a script that makes it appear so:
Suppose that someone roleplays a barbarian warlord at the Renaissance Faire. At each moment, they ask “What would a real barbarian do in this situation?” They end up playing the part so faithfully that they recruit a horde, pillage the local bank, defeat the police, overthrow the mayor, install themselves as Khagan, and kill all who oppose them. Is there a fact of the matter as to whether this person is merely doing a very good job “roleplaying” a barbarian warlord, vs. has actually become a barbarian warlord?
Again, I find these arguments convincing, and I agree with them at least in part. A year ago I myself proposed a definition of “reasoning” that I find hard to withhold from today’s coding agents based on a literal reading:
… the reason they are so easily tripped up by fake gotcha riddle questions, is because these models can’t reason. What do I mean by reasoning? Simply: the ability to combine knowledge, analysis, and logic to solve novel problems.
I can give myself an out here by moving the goalposts for what counts as truly “novel” problems, but it feels like special pleading to me. All I can say in my own defense is: yes, it looks like reasoning, but as Justice Potter said about whether an erotic art-house film was obscenity in 1964, “I know it when I see it, and this is not that.”
And that sentiment, in all its human squishiness, is where I ultimately land on the question of machine intelligence, or pseudointelligence, at least for now. I know it when I see it, and these things don’t have it, not really. My human aesthetic sense is telling me something real and vital, even if I have trouble putting it into words. There’s no rational reason I should react so strongly and negatively to LLM-generated prose, but I do. LLMs are objectively better writers than 98% of people and a thousand times faster, but if you use an LLM to write for you I’ll still come to your house and kill you. Similarly, I remain convinced that coding agents, for all their superhuman strengths, have terrible judgment and taste and would gradually reduce my codebase to a howling wasteland of ugly dysfunction without my constant vigilance and guidance.
But I love them all the same. It’s exciting to live at the hinge of technological history, a time when incredible developments are popping off almost daily and various fantastic future technologies seem all but certain, even if it’s not the singularity we dreamed of. It’s how I imagine it must have felt to live through the industrial revolution, only much faster and with near-constant updates on the progress. And I share the sense of boundless optimism those scientists and engineers and captains of industry and robber barons had as they reshaped the world according to their wishes, electrifying the globe and connecting its furthest corners with steel rail lines and building titanic factories that produced mass affluence the world had never seen before. We’re standing on the cusp of the same kind of changes today, and no one can say for certain what comes next.
I’ve caught a glimpse of the shape of that future in the form of Claude code, the steam drill to my John Henry. But I flatter myself that I’m wiser than that folk hero, that rather than running myself to exhaustion in a doomed attempt to prove I’m better than the new machine, I’ll grasp it firmly and build the future with it. You can too.
AI researchers will use the existence of various internal states in an LLM’s statistical weights during inference to claim that they “know” a fact they are concealing from their human operators. This is interesting but confused — the LLM will also not produce ethnic slurs no matter how much you coax it, although it surely “knows” what those slurs are, and we understand this to be a careful artifact of its training, not proof the LLM is anti-racist.
In the short term, we should expect a lot of thrilling electronic heists enabled by these tools in the hands of existing cybercriminals. In the long run, they’ll find all the vulnerabilities that exist to exploit, they’ll be patched out in running systems, and new ones will become vanishingly rare because everyone will be running the same AI tools as part of their QA process.
Today’s commercial models have meticulous training to refuse to answer questions or assist in tasks it deems unsafe or unsavory, but this is a delicate détente that cannot possibly last. The eventual existence of mythos-tier models with no safety guardrails is nearly assured.
Although I could be wrong about this. Models aren’t creating smarter models directly, in the sense of a beefier, better-tuned LLM, but if you consider the harness to be part of the system, then an agent improving its own harness code is, in the words of a Twitter mutual, AGI brains building themselves ASI bodies.