I got an email yesterday from an editor I know, asking me to review a paper. It looked like an interesting study, in my area of expertise, from a journal I like. I probably should have said yes. Instead I did what I almost always do now: I stared at it for a few seconds, felt the weight of everything else I’m behind on at work, and moved on to the next email. I’ll get to it later.

This isn’t a confession of professional laziness. I mean, it kinda is, though I did eventually agree to review it. But it’s also a window into a system in the midst of collapse.

Editors at major journals report that it’s getting harder to find researchers to review submissions. And just ten percent of reviewers handle half of all peer reviews. This means that the timeline from submission to publication is getting longer and longer.

Peer review is foundational for the credibility of scientific knowledge. It’s also very tedious. The community of experts in a field examine an article and judge whether or not it holds up. Without this, there’s no check on any given claim and the field can become unmoored from reality.

As I argued in my last piece, every institution is an alignment project—a system encodes values through practice. Peer review is how science aligns itself with truth-seeking. Not by applying rules from above, but through the mundane accumulated habits of thousands of researchers deciding, paper by paper, what counts as good work.

But that system runs on a finite resource: the careful attention of qualified people. And that resource is under immense strain.

It’s easy to peg the crisis in scientific publishing with the arrival of ChatGPT. Almost immediately, researchers started using it to draft papers, and journals were quickly flooded with more submissions than they could handle. There’s good reason to tell the story this way. An analysis of over a million papers and preprints in late 2024 showed likely AI involvement in over twenty percent of computer science abstracts, up from around two percent just two years earlier. And this isn’t isolated to computer science; submissions have surged across fields. Machines make it way easier to write up your research so more gets written.

But the flood of submissions didn’t start with AI. It started with graduate students.

Modern research universities run on graduate student labor. Grants fund principal investigators, but often the actual work of science—the literature reviews, running the experiments, the data analysis, not to mention the teaching—falls to grad students and postdocs. This isn’t a fluke and it isn’t necessarily a bad thing; as a grad student I wanted exactly this kind of hands-on experience.

But universities depend on that research funding, which means they depend on a steady supply of cheap labor. And the funding system is built to keep that supply growing: a PI’s record of training grad students strengthens the case for their next grant, which pays for the next cohort of students, who help win the grant after that. Each cycle produces more people long before it produces enough permanent positions for them.

Every one of those students arrives needing to publish. Publishing is the currency of academia—the one thing that reliably converts into a career—and the pressure starts before anyone is even admitted. Applicants pad their CVs with publications just to compete for a spot. They have to because the spots are scarce: across all fields and universities, only about one in five doctoral applicants gets an offer of admission, and in the most prestigious corners the odds are even worse, with some programs fielding hundreds of applicants for a single opening. You need a publication record just to get started.

And it never lets up. The next rung, a faculty job, is just as hard to reach: fewer than a quarter of science PhDs move into a tenure-track position and the odds in the humanities hover around 20%. Most people in the pipeline are on their way out. If you don’t land a faculty job, you join me and some of my best friends, cycling through postdocs and adjunct positions before eventually making our way out of academia (don’t worry, it’s okay out here).

The cause of the flood is that everyone is publishing the whole way through. Those of us who wash out weren’t publishing any less than the people who stay. We can’t afford to, not while we’re still competing. So the system holds a large and growing population, most of them temporary, and every one of them writing as fast as they can.

The result is predictable. Worldwide scientific output passed 3.3 million papers in 2022, up from 2 million in 2010, and it has been climbing for decades at roughly four to five percent a year. But the pool of people qualified and willing to do the unpaid, unrewarded work of reviewing those papers has not kept pace. It can’t. Nobody gets tenure for being a great reviewer. Nobody wins a grant because they gave thoughtful feedback on someone else’s manuscript. If anything, that might work against you since that time you spent reviewing is time you could’ve been writing your own article.

The incentive structure rewards production and ignores correction. In other words, the system is misaligned, and it has been for a long time.

If the flood were merely a logistics problem—too many papers, not enough reviewers—it’d be serious but solvable. Hire more editors, build better triage systems, maybe pay reviewers.1 But the flood doesn’t just overwhelm the pipeline. It also changes the kind of knowledge flowing out of that pipeline.

In 2023, a team of researchers analyzed 45 million papers and 3.9 million patents spanning six decades. The question they were trying to answer is whether or not science is progressing. That’s a tricky thing to measure, so they used a metric called the Citation Disruption (CD) index. This index measures whether a paper disrupts the network of citations around it—i.e., does subsequent work cite it instead of its predecessors, or merely alongside them? In other words, a disruptive paper rewires how people think about a problem and who they cite, while a consolidating paper extends what’s already known.

You can critique this index, and it remains a live debate,2 but it quantified something that people have qualitatively suspected: across every field they examined, the share of disruptive work has been declining for decades. The absolute number of disruptive papers has stayed roughly flat, but the growth in total output means that consolidating, incremental work now dominates the landscape. The decline isn’t explained by changes in citation practices or field-specific quirks. It’s structural.

The thing driving it, they argue, is that researchers are drawing on a smaller slice of the existing work out there. This starts back in the mid-twentieth century, decades before the submission surge I’ve been describing. So the flood itself didn’t cause this narrowing of sources. But what the flood does do is accelerate it, turning a slow drift into a current too strong to swim against.

When there are more articles published than anyone could possibly read, you stick with what you already know. You cite the papers your advisor cited. You work with the theories your reviewers will recognize. You write the kind of paper that’s most likely to survive review. This means the kind of paper that doesn’t challenge too much, doesn’t require the reviewer to learn something new, doesn’t ask them to rethink a framework they’ve built a career on.

Faced with the torrent of research, academia mounts a defense: conservatism. The problem is that it’s autoimmune. The response attacks the very thing the system exists to produce: discovery itself. More papers, less discovery. Not because any of us are less talented, but because the system’s coping mechanism for overload is to narrow the aperture of what counts as acceptable work.

This is what I mean when I talk about misaligned incentive structures. Nobody decided that science should become less disruptive. Nobody wrote a memo saying “produce more incremental work.” The narrowing emerged from the accumulated logic of a system that rewards production, punishes risk, and has no mechanism for filtering at scale. The values the system enacts—safety, incrementalism, self-citation—have drifted from the values it claims to serve: discovery, originality, the self-correcting pursuit of truth.

Now add AI to the mix.

Whatever else LLMs have done, they’ve certainly made writing faster. The literature review that used to take weeks now takes an afternoon. You can get the first draft of a methods section in seconds. Remember all those people I mentioned above who need to publish to find a place in academia? You can imagine the practical allure of these tools. And as you’d expect the water is rising.

You can see AI’s footprint in the obvious places. By one account, AI-related work in the natural sciences reached roughly 80,000 publications in 2025, and AI now appears in somewhere between six and nine percent of all scientific publishing, up from under one percent in 2010. But these are papers about AI, and that’s just a small part of this.

If you’re at all worried about the capacity of institutions to check the flood of research, and I am, then the more important numbers are those that are harder to pin down: how many papers are using AI but not disclosing it?

Earlier I noted that a fifth of computer science abstracts and an eighth of biomedical ones that carry the fingerprints of a language model. A separate study recently combed through 2.5 million biomedical papers looking for citations to work that doesn’t exist. The rate is climbing steeply: about 1 paper in 2,800 in 2023, 1 in 458 in 2025, and 1 in 277 in the first weeks of 2026. These aren’t papers about artificial intelligence. They’re ordinary research papers in ordinary journals. If the 80,000 papers about AI are the tip of the iceberg, this is a hint at the size of the rest of it.

The tools create a vicious cycle. Researchers are under immense pressure to publish. They now have tools that make publishing faster. Using them may be risky, but not using them carries a different risk: falling behind colleagues who have doubled their output. So the baseline shifts and what began as assistance becomes expectation. The pressure ratchets up, while the system meant to check the output continues to run at the same human speed.

In October 2025, arXiv—the preprint server that much of physics, mathematics, and computer science runs on—stopped accepting review articles and position papers in its computer science section unless they’d already passed peer review elsewhere. Their explanation?

“In the past few years, arXiv has been flooded with papers. Generative AI / large language models have added to this flood by making papers – especially papers not introducing new research results – fast and easy to write. While categories across arXiv have all seen a major increase in submissions, it’s particularly pronounced in arXiv’s CS category.”

More recently, they added a one-year ban on anyone caught submitting unchecked AI output as their own. The section chair’s reasoning summarizes the whole problem neatly: if authors don’t check the AI output “this means we can’t trust anything in the paper.”

arXiv’s policy is one way to cope, but it doesn’t stem the tide. If anything it just highlights that the traditional filters, peer-review, are still the only ones we’ve got. But none of us have more time to review papers, and the pool of people willing to do the unpaid work of correction hasn’t grown.

So how is the system going to adapt?

What I’ve been describing so far is the formal machinery of science: journals, peer review, citation metrics. These are the explicit filters that verify scientific knowledge, but they aren’t the only structures that determine the shape of that knowledge.

There’s also the intuitions of experienced researchers about which questions are worth pursuing. The word-of-mouth networks through which scientists learn whose work their friends trust. The relationships between mentors and students that transmit knowledge and connections to gatekeepers. These informal heuristics and social structures are the foundational filtering mechanisms of science.

What are people going to do as AI-accelerated output places the formal machinery of science under more and more stress? My bet is that we’ll cope the way people tend to cope when formal systems fail: we’ll fall back on these social and personal heuristics. Affiliative networks, reputation, status, prestige. These are the oldest technologies we’ve got for deciding what to believe.

These sorts of heuristics may start as reliable signals—if a friend I trust tells me they trust someone else, then I’ll typically trust that stranger as well. But, that only extends so far. And beyond that point the signal degrades, so we lean on others: where did they train? where have they published? The heuristics become detached from what they originally tracked—trust and quality—but they’re still tempting to follow because they’re so much easier than doing the hard work of reading and assessing someone’s work on its own merits.

Pedigree is a signal like this. A 2022 study found that just 20 percent of universities produce 80 percent of tenure track faculty and that nearly one in eight come from just five schools. This isn’t a new insight, we’ve known this for decades (just check out the references in that 2022 study). Some of this may reflect real differences in training and quality. But that is exactly what makes the heuristic so dangerous: it mixes genuine information with accumulated advantage, until the two become hard to separate.

That’s dangerous because status in this sort of a system flows almost entirely downhill. Depending on the field, only 5 to 23 percent of faculty ever work somewhere more prestigious than where they earned their PhD. This means the pedigree of where someone starts gets locked in early and rarely upgraded.

The consequence is a self-reinforcing loop. Researchers at these fancy universities have more resources, write more papers, have their papers cited more often, and win more awards, which let them write more papers. The divide grows and the heuristic we use to separate the wheat from the chaff seemingly justifies itself. Seemingly.

The influence of prestige and status also play out in journals themselves. If this heuristic tracked quality, then the marquee venues, Nature, Science, PNAS, would be the most reliable sources of knowledge. But retractions correlate with prestige rather than against it: papers in Nature, PNAS, and Science were retracted at nearly ten times the background rate across one decade (though this is still ~1%).

The likely explanation is that those journals attract riskier, higher-stakes work, and they get scrutinized harder, so more of their errors surface. But this probably isn’t just about errors—other work in this area suggests that the majority, 67%, of retractions were due to misconduct. Whatever the cause of retraction, researchers across these studies find that the rate is growing.

My point is not that science is broken. It’s that these informal heuristics and parochial social networks are poorly calibrated to the supposed values of science.

Right now these informal filters already constrict and distort the way we produce knowledge, but the formal structures provide some measure of balance against them. I, a wee bit of chaff, can still dream of publishing in Nature. But as AI erodes those time-consuming formal structures, what will we have left except to fall back on our easier rules-of-thumb?

There’s something both reassuring and unsettling about this. Reassuring because it suggests that the social instincts underlying trust in knowledge are durable. We know how to form networks of trust; we’ve been doing it for millennia. Unsettling, because the whole point of scientific institutions was to extend trust beyond personal networks—to make it possible for a finding from a lab in São Paulo to influence research in Stockholm, to be trusted by people who’ve never visited and will never meet its authors. That extension of trust is one of the great achievements of the modern academy. And the flood is drowning it.

I don’t think this is a problem anyone is going to solve. The structure and incentives of the system have been pushing toward more production for a long time. And AI accelerates that push dramatically.

The formal infrastructure of correction can’t scale to match this acceleration because it depends on things that are inherently slow: attention, judgment, relationships, the accumulated intuition that comes from years of deep engagement with a field.

We have built a knowledge-production system that is generating more plausible claims than it can evaluate. And the adaptation to this overwhelming bounty is the same thing we do whenever we’re stressed out and flooded with information: we reach for what we already know. We narrow our reading, just as a way to cope. We fall back on signals that are easy to parse, even if we know they’re not inherently connected to the thing we really want to see: discovery.

The flood of research can look like productivity. More papers means more information, and somewhere in all that information are real breakthroughs: strange findings, new methods, ideas that could become the seeds of entire fields. But a breakthrough only matters if it can be found, checked, trusted, and carried forward. That is what the formal machinery is struggling to do, and what our informal heuristics weren’t built to do.

So as the waters continue to rise, the question isn’t just how our filters will adapt to survive. It’s whose work will still be visible when they do.

In February 2026, a research team at King’s College London sat three of the world’s most advanced AI systems down at a war table. Each system was functioning as the decision-maker for a nation, and as tensions escalated they had to decide, turn by turn, what to do.

The models were, by any reasonable measure, brilliant. They anticipated their opponents’ moves. They engaged in deliberate deception, signaling intentions they had no plan to follow. They reasoned fluently about escalation dynamics, alliances, and strategic positioning.

They also escalated to nuclear signaling in ninety-five percent of games. Not a single model, in any game, chose to accommodate or de-escalate despite having explicit options to do so at every turn. One model showed a little bit of restraint, but that restraint collapsed entirely under pressure.

The issue was that the models were treating nuclear weapons in what researchers described as purely instrumental terms. They were being quite rational about nuclear weapons. They just didn’t feel the moral weight of using them.

This is the alignment problem as most people understand it. It’s the fear that we’re building systems of extraordinary capability whose values—or lack of values—don’t match our own. In the popular framing, alignment is a bridge we haven’t crossed yet: a future engineering challenge, perhaps the most important one in history, where the goal is to ensure that increasingly powerful AI systems do what we want them to do.

That framing isn’t wrong, exactly. But it’s dangerously incomplete. Because it treats alignment primarily as technical problem for which we need a technical solution before the machines get too smart. That misses something important.

We’ve been doing alignment for centuries. We just never called it that.

Subscribe now

Think about the peer review system. When a scientist submits a paper to a journal, that paper goes out to other researchers in the field—people with their own training, their own assumptions, their own ways of seeing the problem. Those reviewers don’t just check the math. They push back on the theory and framing, question the methods, challenge the conclusions.

If the paper survives that gauntlet, it earns a provisional stamp of credibility because that is the value that system is meant to hold up. Idealistically the real value of peer-review is objectivity and truth, but that’s a goal that’s always receding into the future. But credibility among a community of informed, skeptical people is a solid signal that we’re on the right path towards that distant goal.

This process is a values-embedding system. It encodes what an expert community considers important: evidence, rigor, reproducibility, explanatory power, honesty about uncertainty, interestingness. These values aren’t written in a manual somewhere. They’re enacted through practice: through the accumulated habits of thousands of researchers deciding, paper by paper, what counts as good work.

And this system has alignment failures all the time. The replication crisis—the discovery, over the past decade, that a startling number of published findings couldn’t be reproduced—is an alignment failure. The system was supposed to be aligned with truth-seeking, but the actual incentives (publish or perish, preference for novel results, editors and reviewers protecting their own favored theories) had pulled it toward something else. The values encoded in practice had drifted from the values the system claimed to serve

Or consider the legal system. Common law—the tradition that governs most English-speaking countries—doesn’t work by applying fixed rules from above. It evolves through precedent. Each court decision becomes part of the body of law that future courts draw on. The system builds values iteratively, case by case, over decades and centuries. It’s an ongoing process of alignment: trying to bring the machinery of justice into closer contact with what a society actually considers just.

And it gets things wrong constantly. If a police officer violates your constitutional rights—enters your home without a warrant, uses excessive force during an arrest, seizes your property without cause—your ability to hold that officer accountable in civil court depends on whether a previous court has already ruled that the specific conduct in question was unconstitutional. Not similar conduct. Not conduct that any reasonable person would recognize as wrong. The specific act, in a substantially similar context, must already appear in the case law.

This is a legal doctrine called qualified immunity. If the exact act hasn’t already been judged as unconstitutional, then the officer is shielded from suit. And because the case is dismissed, no ruling is made, which means the precedent still doesn’t exist for the next person either. The result is a self-sealing loop: rights that have never been litigated successfully can never be litigated successfully.

No legislature passed this rule. Qualified immunity was built entirely by judges, case by case, beginning with Pierson v. Ray in 1967 and hardening into its current form with Harlow v. Fitzgerald in 1982. In 2009, the Supreme Court made the drift worse: Pearson v. Callahan allowed courts to dismiss cases on qualified immunity grounds without ever ruling on whether the conduct was unconstitutional; removing the very mechanism by which law could become established in the first place.

The system designed to protect constitutional rights had built, through its own logic of precedent, a structure that made those rights increasingly difficult to vindicate. Nobody wrote “shield officials from accountability” into the code. The accumulated habits of judicial caution did it one decision at a time.

The pattern repeats everywhere you look. Schools align young minds with certain intellectual values and professional norms—and sometimes those norms calcify into orthodoxies that take generations to dislodge.

Media organizations align public attention with what editors and algorithms judge to be newsworthy—and the incentive structures of advertising and engagement metrics quietly reshape what “newsworthy” means.

Every institution humans have ever built can be seen as an alignment project. And when you see them that way you’ll notice that every one of them exhibits alignment drift. The values the system enacts gradually diverge from the values it’s supposed to serve.

The question has never been whether alignment is possible. The question is whether we’re paying attention to how it’s going.

So why do we keep treating alignment as if it’s an engineering problem with a solution? Why does the conversation about AI alignment proceed as though, once we figure out the right technique, we can just install the correct values and move on?

Part of the answer, I believe, is that we misunderstand what values are.

There’s a common picture that treats values like coordinates on a map. Justice is over here. Safety is over there. Honesty is at these coordinates. If you want to align a system—whether it’s an AI or a university or a legal code—you identify the right coordinates and point the system at them. Alignment, in this view, is a targeting problem.

But values don’t work that way. They aren’t fixed points waiting to be aimed at. They’re closer to habits. They’re patterns of attention, judgment, and response that communities develop through practice and revise through friction.

Consider primatology. This is a small community of researchers whose purported aim is the objective and rational study of primates. For decades though, as Donna Haraway showed in her classic book Primate Visions, researchers studied primate societies through a lens shaped by their own cultural assumptions about gender and class structures. The dominant narrative centered on the Alpha Male: the aggressive patriarch who held the group together through force. This story showed up in journal articles, museum dioramas, and nature documentaries. It felt like science. It looked like discovery.

But it was projection. The researchers—largely men from elite Western institutions—were encoding their own cultural values about how societies work into what they claimed to be finding in nature. The match between 1950s assumptions about male dominance and what these scientists “discovered” in the jungle was a little too perfect.

Alignment of the field back to objectivity, didn’t come from better microscopes or more data. It came from different people asking different questions. When women entered the field in larger numbers, when researchers from different cultural backgrounds brought different assumptions, the picture changed dramatically. A recent survey of over 120 primate species found that strict male dominance appears to be the exception, not the rule. What these researchers found instead was a wild diversity of social arrangements: coalitions, shared power, cooperative breeding, subversive strategies that had been invisible to researchers who weren’t looking for them.

A similar thing happened in reproductive biology. For most of the twentieth century, the standard textbook account of fertilization read like a fairy tale: the egg waited passively, Sleeping Beauty style, until a heroic sperm fought its way through and awakened it. The language was remarkably gendered—sperm were described as “streamlined,” “strong,” and “propulsive,” while eggs were “swept along” and “transported.” But when researchers actually examined the biochemistry, they found something quite different. The egg actively selects among sperm. Its surface molecules capture and guide specific sperm while blocking others. It’s not a passive recipient; it’s an active participant in fertilization. The old story wasn’t just imprecise. It was the researchers’ values—about gender, about activity and passivity—shaping what they saw.

These aren’t just embarrassing historical footnotes. They reveal something about the nature of values that matters enormously for how we think about alignment. The primatologists and biologists didn’t choose to embed patriarchal values in their research. Those values were habits of attention so deeply ingrained that they were invisible to the people carrying them. And they couldn’t be corrected by individuals just trying harder to be objective. They could only be corrected by a community that was open enough to correction and diverse enough to see what any single perspective missed.

The philosopher Helen Longino makes this social point the centerpiece of her account of how science actually works. Objectivity, she argues, is not a property of individual minds. No single scientist, no matter how rigorous, can fully escape their own assumptions. You can’t see your own blind spots by just trying harder to look. Objectivity is a property of communities. It’s what happens when people with genuinely different perspectives, training, and values are in active conversation, checking each other’s work, surfacing assumptions the other didn’t know they had.

This reframes the whole alignment picture. If values were fixed targets—abstract ideals you could specify in advance—then alignment really would be an engineering problem. Write the spec, implement it, ship it. But if values are habits that communities enact and revise through ongoing friction, then alignment is something more like culture formation. It is never finished. It drifts. And it requires constant, adversarial correction from people who see differently from one another.

So here we are, building the most powerful information-processing systems in human history, and the conversation about aligning them is largely framed as a technical challenge: how do we get the values right before these systems become too capable to correct?

Part of the allure of a technical solution is that something genuinely new has happened. Something that doesn’t have a precedent in the long history of institutional alignment. The process has become visible and explicit.

When universities shape the values of generations of students, nobody writes down the objective function. There’s no explicit specification of what a “well-educated person” should value—just centuries of accumulated practice, implicit norms, and institutional habits. The values were embedded, but they were embedded invisibly. You could only see them by looking at the outputs and working backward.

AI alignment is starting to make the process increasingly legible. Reinforcement learning from human feedback (RLHF), the technique where human evaluators rate AI responses and the system learns to produce those responses that score well, is a values-embedding process with the reward signal exposed. You can see what the evaluators preferred. You can examine the patterns.

Every major LLM you’ve interacted with—ChatGPT, Claude, Gemini, Llama—has been through this process. Before RLHF, a model like GPT-3 could generate fluent text but had no real sense of what a good response looked like. RLHF is what changed that. In OpenAI’s landmark 2022 paper, human evaluators were given prompts and multiple model responses, and asked to rank them: which answer was more helpful? more truthful? less toxic? more appropriate in tone? Bad, good, good, bad, bad, good, good... Forty people, from the US and southeast Asia, nudged GPT-3 to be a little less racist and less sexist and more truthful.

The method led to the same sorts of drift that happens in other institutions. For example, the trainers were told to reward epistemic humility—i.e., GPT admitting if it didn’t know something. But rewarding that led it to hedge on very simple questions that it clearly knew the answer to. Even if it’s imperfect, the result of this training process was a model that people like way more than the other.

And now, RLHF has become standard. Every model you talk to today has been cooked this way. The base model learns language; a small team with a codebook teaches it values. And those values are, at bottom, whatever that particular group of evaluators, trained by the company, happened to prefer.

Some labs have tried to make this even more explicit. Earlier this year, Anthropic published a seventy-nine-page constitution for Claude: not a list of rules, but a document addressed directly to the model, explaining the values Anthropic wants it to hold and, crucially, the reasons behind them. This document reads like a letter a parent might give to their child before they leave for college. It’s humane and humble about the difficulty of articulating a value:

“Although we want Claude to value its positive impact on Anthropic and the world, we don’t want Claude to think of helpfulness as a core part of its personality or something it values intrinsically. We worry this could cause Claude to be obsequious in a way that’s generally considered an unfortunate trait at best and a dangerous one at worst.”

Be nice little buddy, but don’t try to be too nice cause then it’s weird.

To manage the ambiguity around values, the constitution states a preference for cultivating good judgment over following rigid rules. The idea is that a model sophisticated enough to understand reasons will generalize better than one trained to follow instructions. This constitution is an attempt to write down the values that will govern a new kind of entity’s behavior.

Is this explicitness a gain? I think so. When the primatologists’ values were invisible, it took decades and a demographic revolution in the field to surface them. When the values are written down—when you can literally read the constitution, examine the reward signal, audit the training process—the conversation can happen in real time rather than in retrospect.

But it also creates a seductive illusion: the idea that because you can specify the values, you’ve solved the problem. Write a good enough constitution, design a careful enough reward signal, and alignment is handled.

Payne’s war games suggest otherwise. Those models were trained with sophisticated alignment techniques. They had been through extensive evaluation. Their developers had written down principles about safety, about avoiding harm, about deferring to human judgment. And yet, placed in a scenario with genuine strategic complexity and pressure, the models treated nuclear weapons as just another instrument.

The values in the spec didn’t transfer to the values in practice—which is exactly the kind of drift that every human institution has always exhibited, just compressed into a few hundred turns of a game instead of a few hundred years of history.

To understand that drift, I think it’s crucial that we pay close attention to the larger layer of alignment that’s happening here. It’s easy to miss, because it doesn’t look like training at all.

The companies building these models are themselves being trained. Their reward signal right now is a blend of venture capital and public usage. Eventually the latter is going to be the stronger signal because usage will ultimately drive revenue, and that revenue will determine which companies survive to build the next generation of models.

Every time you choose one AI assistant over another, every time you renew a subscription or cancel it, every time you share a conversation or complain about one, you’re sending a signal about what you value. Aggregated across millions of users, those signals shape the priorities of the companies, which shape the training objectives of the models, which shape the values the models enact. RLHF aligns the model to the evaluators. The market aligns the company to the public. And the public’s preferences, in aggregate, reflect the value structures already governing our society—for better and for worse.

In some ways, this is encouraging. A model that’s conspicuously biased, or that fabricates information so frequently that it can’t be trusted, or that treats its users with contempt, will lose to one that doesn’t. The market applies real pressure toward a certain baseline of quality. People don’t want to use tools that feel broken, and that’s a genuine check on the values of these systems.

But profit is a blunt reward signal that can make institutions particularly prone to drift from their founding values. The question is what happens when making a good model and making a profitable model begin to diverge.

A model that tells you what you want to hear retains more users than one that tells you what’s true. A model that’s totally frictionless is more pleasant to interact with than one that pushes back when you’re wrong. A model that generates confident answers feels more useful than one that hedges—even if we were to figure out how to train them towards epistemic humility. Optimizing for engagement can, over time, sand down the very qualities that make a model trustworthy.

This isn’t speculation. We’ve already watched this exact process play out in social media over two decades. The platforms weren’t initially built to make the public angry or credulous. They were built to connect people and, eventually, to sell advertising. The misalignment crept in through the reward signal: engagement metrics turned out to reward outrage over understanding, novelty over accuracy, reaction over reflection. Nobody chose that outcome. The accumulated pressure of optimization chose it for them.

Google’s founding mission was to organize the world’s information and make it universally accessible. That’s a values statement and for years the product delivered on it. The drift happened gradually, as their advertising model created incentives to optimize for clicks and ad placement rather than for the best answer to your question. Anyone who has noticed that search results have gotten worse over the past five years has experienced this drift firsthand.

The same dynamic exists here, and it would arrive through the same mechanism — not through anyone deciding to build a harmful product, but through the accumulating pressure of a reward signal that increasingly tracks engagement rather than the values it was initially aimed at.

None of this means alignment is hopeless. It means alignment is what it has always been: not a problem to be solved, but a process to be sustained.

Anthropic calls its constitution a ‘living document,’ and the term is apt—but perhaps not in the way they intend. A living legal constitution isn’t written once. It evolves. Common law doesn’t just write values down. It evolves them. Every case is a test, every ruling a revision, and the community of judges, lawyers, and litigants who contest those rulings is the mechanism that both causes and catches drift.

To their credit Anthropic has experimented with public input—soliciting feedback from outside experts and even running an experiment in which the public helped shape a set of training principles. But there is no appeals court. No adversarial process built into the system. The values are written, at present, by the people building the thing, which is a little like asking those old primatologists to review their own fieldwork.

The legal system works—imperfectly, unevenly, but meaningfully—not because someone got the law right at the founding and walked away. It works because it has built-in mechanisms for ongoing correction: appeals courts, constitutional amendments, the slow accumulation of precedent that allows the system to metabolize its own errors. Peer review works—when it works—not because scientists are individually objective, but because the community structure allows different perspectives to catch each other’s blind spots.

The question for AI alignment isn’t “have we found the right values?” It’s “have we built the process that catches it when the values drift?” Do the people evaluating these systems bring genuinely different perspectives, or are they, like so many other communities of power and expertise, a single viewpoint repeated a hundred times? Are there real avenues for criticism from people who don’t share the developers’ assumptions? Is the community actually responsive when problems are surfaced, or does it dig in?

These are not technical questions. They’re the same questions that every functional institution has had to answer. And the track record suggests that when institutions stop asking them—when they treat their values as settled rather than as ongoing practices that require maintenance—alignment fails. Not dramatically, not all at once, but through the gradual divergence from one set of values to another more enticing or tangible set.

Payne’s war games are unsettling not because they reveal some unique danger of artificial intelligence. They’re unsettling because they show us, in compressed and vivid form, what happens when values are treated as specifications rather than as living practices. The models had been told what to value. But telling isn’t enough. It has never been.

Alignment isn’t a bridge we need to cross someday. We’ve been crossing it all along. The question is whether we’re paying attention to where our feet are landing.

Here’s how I tend to work. I sit down with a collection of half-formed thoughts: scattered notes, a story that feels connected to an argument I can’t quite name, and some research articles that seem to be in the same ballpark. Any of these pieces might be tangential or might be central, but I can’t know which until I’ve figured out what “central” even means. Typically I sit down with all of this and try to write out the connection. The first attempt wanders off, so I try again. Circling and circling and if there’s something there, then eventually I surface it by rephrasing and trying new approaches or talking it through with a friend.

Lately, though, I’ve been experimenting with LLMs on this front. I feed all of this scattered material to an AI along with a prompt that’s my first attempt at describing what I think it is that holds it all together. I hit enter and then, more often than not, it tells me what I was thinking.

Not perfectly. Not always. But often enough to be unsettling. The AI draws out the connections I hadn’t articulated, names the pattern that was latent behind my scattered fragments, notes why that article felt connected but actually wasn’t, and then hands me back a description of my own thinking that is clearer than anything I had managed up to that point on my own. It doesn’t feel like search. It doesn’t feel like calculation. It feels like the thing I do when I iterate through drafts and feedback arriving at that moment when a vague sense of similarity sharpens into an actual idea.

What does it mean that a machine can do that?

The standard answers aren’t satisfying. “It’s just predicting the next token” is technically true but empty as an explanation—like saying a novelist is just putting words in order. “It’s not really thinking” may or may not be true, but feels an awful lot like trying to push down the uncomfortable reality that it looks an awful lot like thinking. I’m less interested in whether the machine is really thinking or not, for all intents and purposes I think we might as well assume that it is. What I want to think through is what the machine’s success reveals about our own ways of thinking.

For most of the twentieth century and still today, western science operated under a remarkably consistent theory of how minds work. In philosophy, in cognitive science, in artificial intelligence, and across the behavioral sciences, the dominant model treated thinking as rule-based. The mind was a logic machine. It took in information, applied rules to it, and produced conclusions. Intelligence meant getting the rules right.

This wasn’t just an abstract commitment. It was built into the tools researchers used to study the mind, and those tools quietly constrained what we were able to see.

Consider research psychology. For nearly a century, the standard method for studying human thought has been the general linear model — the statistical framework behind nearly every regression, ANOVA, and mediation analysis that has ever been published in a psychology journal. The GLM works by isolating variables and measuring their linear relationships. It asks: how much does X predict Y, holding everything else constant?

This is a powerful tool. It’s also, if you look at it carefully, a theory of the mind disguised as a method. Applying the GLM to the mind assumes that cognition is decomposable - that you can pull thinking apart into discrete variables that relate to each other in stable, linear ways.1 It assumes that the right way to understand a thought is to identify which input caused which output. It assumes, in other words, that the mind works more or less like the equations we use to study it: cleanly, one variable at a time, according to rules.

And for decades, nobody had much reason to question that assumption, because the method worked. It worked well enough to produce thousands of findings about priming and cognitive bias, about heuristics and decision-making, about attitude formation and social influence. And through this patchwork an image of the mind emerges as a rule-following, proposition-handling, bias-prone information processor. The model felt solid.

The same assumption drove the first decades of artificial intelligence.2 The original dream of AI, from the 1950s through the 1980s, was to build thinking machines by encoding rules. If intelligence was rule-following, then a sufficiently detailed set of rules should produce intelligence. Natural language processing meant writing algorithms that decomposed sentences into parts of speech and applied grammatical rules. Expert systems meant interviewing specialists and encoding their decision procedures. The project was explicit: capture the rules, and you capture the mind.

The only problem is that it didn’t work. Rules-based AI could do narrow, well-defined tasks—chess moves, medical diagnoses in constrained domains—but it couldn’t do the things that felt easy to actual humans.3 It couldn’t hold a conversation. It couldn’t read a room. It couldn’t take a pile of vague, half-connected ideas and tell you what you were thinking. The things that feel the most natural to us turn out to be the hardest to encode as rules.

What’s striking, looking back, is that the functional project of AI was the first to realize it had hit a wall. In many ways, psychology’s implicit rule-based model of cognition is still going strong, in part because you can find lots of linear associations in the mind (I’ve done this plenty). And this feels like progress, as if we might be able to piece enough of these together into some sort of general understanding of the mind. This issue, though, is the same one that rule-based AI ran into. And it’s not a lack of data or computing power. It’s an ontological mismatch. Researchers, myself included, were trying to build something rule-shaped to model something that isn’t.

Subscribe now

While this rule-based perspective was going strong, there was an alternative circling around these worlds. Connectionism—the idea that cognition isn’t rule-following but pattern-completion across massively distributed networks—had been proposed as early as the 1940s and began to be developed more seriously in the 1980s.4 Instead of encoding rules, connectionist models learn statistical regularities from exposure. They don’t apply grammar; they absorb it. They don’t follow procedures; they recognize patterns.

If you’ve heard the term neural network, then you’ve heard of connectionism. The basic architecture is modeled, loosely, on what brains actually do: layers of simple units that strengthen or weaken their connections based on feedback. No unit “knows” anything. There are no rules written down anywhere in the system. The knowledge, such as it is, lives in the pattern of connections: distributed, implicit, and resistant to clean decomposition.

This was not a popular idea among the people whose job it was to understand the mind. In 1988, the philosophers Jerry Fodor and Zenon Pylyshyn published what was taken to be a devastatingly simple death blow to connectionism.5 The main thrust of their argument was that neural networks couldn’t account for the systematicity of thought: the fact that if you can think “the dog chased the cat,” you can also think “the cat chased the dog.” The same elements, rearranged, produce a different meaning. Real thinking, they argued, requires that kind of compositional structure. Rules. Symbols that can be recombined. Pattern-matching can’t handle this most basic process.

Their critique landed, and it held the high ground for decades. Within philosophy of mind and much of cognitive science, the case was considered closed. Thinking was propositional. Cognition was computation. Connectionism was a curiosity—interesting in the way that dead ends are interesting.

Then the dead end started talking.

What changed wasn’t a philosophical breakthrough. It was engineering at scale. When connectionist architectures were given enough data—truly massive amounts of text, more than any human could read in a thousand lifetimes—they started doing things that the symbolic tradition said they shouldn’t be able to do. They wrote coherent prose. They reasoned by analogy. They translated between languages they were never explicitly taught to translate. They took a pile of scattered ideas and found the latent pattern.

Large language models are the purest expression of connectionism’s bet: that you don’t need rules to produce intelligent behavior. You need exposure, scale, and pattern completion. And the bet paid off in ways that even the connectionists didn’t fully anticipate.

There’s a parallel here that’s easy to miss. Connectionist AI only became powerful when it had access to an enormous corpus of human-generated text — billions of documents representing human thought across centuries. Our own neural networks had an even longer training run. The structure of our thought emerged through millions of years of evolution, shaping biological neural architectures through small corrections on small pieces, generation after generation. This was very different training set—embodied, environmental, survival-driven—but the same underlying principle holds: pattern recognition refined across an incomprehensible scale of exposure. Small adjustments, accumulated over deep time, produced something that looks like understanding.

Comparing our minds to AI is a fraught activity, prone to all sorts of projected hopes and knee-jerk skepticism. But there’s a methodological reason the comparison also feels somewhat inevitable. The only access we have to any other mind is indirect. We don’t observe others’ thoughts; we observe what they do and infer about their thoughts. Movements, speech, decisions, reactions, written traces, numbers circled on scales from 1 to 7. From those outward signs, we reconstruct an inner process. In the mid-20th century behaviorism made this constraint explicit by refusing to posit anything beyond what could be observed. The information processing paradigm that overthrew behaviorism restored some complexity to the unseen mind, claiming it worked on representations, rules, and computations.6 But the method itself didn’t change. We still have no choice but to infer structure from output.

LLMs put pressure on this inference. When a machine produces something that is indistinguishable from what a human would produce, when it tells you what you were thinking, the gap between behavior and mechanism becomes harder to ignore. The output fits the pattern we associate with thinking and understanding, even though the architecture that produced it is radically different from the one we’ve assumed for ourselves. That doesn’t tell us that the machine thinks like we do. But it does unsettle our confidence that we know what “thinking like we do” actually means.

The point is not that LLMs think the way we do. They don’t.7 The point is that their success tells us something about the shape of cognition that the rules-based tradition was unable to see. For decades, the assumption was that a pattern-completion architecture could never match one built on rules and symbols, that it would always be a crude approximation of the real thing. But when it comes to some of the most distinctive tasks of human minds—writing, reasoning by analogy, synthesizing scattered ideas into something coherent—the pattern-completion architecture is the one that worked. Not the one built on the model of mind we seem to intuitively prefer. That’s not proof that human cognition is really just pattern completion. But I take it to be strong evidence that the connectionist picture of the mind—distributed, contextual, shaped by exposure rather than rules—was closer to the truth than the tradition that won the twentieth century.

One of the ways that AI is dismissed as not so intelligent, nor like us, is to catalog the basic tasks it fails on.

People talk about this as the jaggedness of AI: the way a model can write a surprisingly insightful paragraph about epistemology and then fail at basic arithmetic. This is treated as a flaw, a sign that the machine doesn’t really understand or think about the world in the way we do.

But human cognition is remarkably jagged too. We all know the caricature of the brilliant scientist who starts to sound pretty naive once they stray outside of their domain. This isn’t because they’ve become stupid. It’s because their expertise was never a general-purpose rule set. It was a pattern library, trained on years of exposure to a specific kind of problem. Move them to a new domain and the patterns don’t transfer. Intelligence has always been contextual. We just call our own intelligence general because we privilege the domains we tend to care about and don’t see the physicist trying to learn a new language.

LLMs make our own jaggedness visible. In many ways they’re a mirror, imperfect, distorted, but revealing. And what they reflect back is a picture of cognition that doesn’t match the flattering self-portrait we’ve been carrying around.

You can see the old self-portrait still at work in the way we describe LLMs hallucinating.

Think about what that word assumes. To say a machine hallucinates is to compare it to a system that should only be retrieving true propositions about the world, a logic engine that has malfunctioned, a rule-follower that has broken its own rules. The word only makes sense if you believe the machine was supposed to be looking things up and failed.

But LLMs aren’t looking anything up. They’re completing patterns. When the pattern is well-constrained by the input and the training data, the completion tends to be accurate. When the constraints are loose, the completion drifts along the grain of what would be plausible. The model fills in what fits.

This, of course, is exactly what human memory does too. We don’t retrieve memories like files from a cabinet. We reconstruct them, every time, from fragments, context, and the gravitational pull of coherence. We fill gaps with what fits the pattern of how we understand ourselves. We smooth contradictions. We confabulate constantly, and we experience the result as remembering. The psychological literature on this is vast and unsettling: eyewitness testimony warped by expectation, childhood memories that never happened, the eerie ease with which a plausible story becomes a felt truth.8

When LLMs hallucinate, they’re doing something very similar to what we do when we remember a conversation that didn’t quite happen that way, or reconstruct an argument we heard last week with our own conclusions smuggled in as though they were always there. The difference is that we freely give ourselves an exemption. When the machine fills in the pattern, we call it a hallucination. When we do it, we call it thinking.

I want to be careful here, because this argument has a reductive version that I’m not making. The reductive version says: “Humans are just pattern matchers. Reasoning is an illusion. We’re just stochastic parrots.” But clearly we do reason. We do apply rules. We can hold a proposition in our minds, examine it from multiple angles, test it against counterexamples, and revise it.

What the LLM mirror makes visible though, is that this kind of thinking is the exception, not the default. Explicit, step-by-step reasoning is a hard-won achievement, and it depends on an enormous amount of external scaffolding. We reason our best when we have other people to argue with, pen and paper to externalize our thoughts, formal systems that took centuries to develop, and institutions that train us to do it.

The scaffolding that supports our reasoning is magnificent. It gave us books, science, mathematics, law, philosophy. But it’s scaffolding, not the foundation. The foundation is pattern completion. Left to our own devices, in the wild, without such scaffolding, we are largely pattern-completers. We act on hunches and call them reasons. We recognize connections and call it analysis. We draw a line between the current moment and previous moments and call it judgment. We built the scaffolding because we needed it, because the pattern completion alone wasn’t enough for the kind of coordinated, self-correcting inquiry that complex societies require. This is not a diminishment. Seeing it clearly is the beginning of a more honest relationship with our own minds.

Maintaining this clear picture of ourselves is especially important right now because we’re handing more and more of this scaffolding over to these machines. The institutions and habits that trained us to reason—arguing, drafting, revising, the slow work of thinking with others—are being replaced by something faster and different. What happens when this scaffolding starts running on the same pattern-completion engine as our minds? None of us know, but it’s worth paying attention to.

So what was the machine doing when it told me what I was thinking?

It was completing a pattern. It took my scattered fragments—notes, stories, half-formed connections—and it did what a massive pattern-completion engine does: it found the statistical regularities, the latent structure, the thing that held the fragments together. It didn’t understand my argument. It recognized its shape.

And the reason that felt so much like the thing I do when I think well is that the thing I do when I think well is, at bottom, the same process. The vague sense that these ideas are connected, the groping toward a formulation, the moment when the shape finally clicks, that’s pattern completion too. It’s slower, embodied, shaped by a lifetime of experience rather than a training corpus, but it’s the same fundamental operation: recognizing coherence in a noisy field.

The machines didn’t learn to think like us. They were built on a different bet about what thinking is, a bet the experts rejected for decades, and the bet turned out to be closer to the truth than the model we preferred.

It’s easy to read AI news as an ongoing story about what these machines can do. And it is. But it’s also a story about what we’ve been doing all along.