In a rare moment of quiet around our house last week, I read two pieces about AI.
In the first, the author spent a weekend test driving Meta’s new AI agent, Muse. He asked it to book a dinner reservation and Muse got stuck in a thicket of human verification tests. He asked it to do some other banal errands, and it succeeded with some relatively banal results, like makes a post for “Hat” on eBay.
The whole tone of the piece fits a common chord among culture pieces on AI: this tech can’t write well and makes basic mistakes so it’s a nothing-burger.
Then I read an account about how software is being built inside OpenAI. To be sure, this is a place that doesn’t dismiss AI, but just look at how their workflows have changed: non-engineers like finance, recruitment, and legal use AI-agents for ~90% of their work. And the engineers have created something like a software factory: humans spell out what they want and then agents do the rest: they gather context, write code, test the code, review each other’s work, fix issues, deploy whatever they’ve built, and monitor the results. That sounds like a something-burger.
In some ways these are just two reactions to different parts of AI’s “jagged frontier.” It’s superhuman at some tasks and a total buffoon on others. And it makes sense that AI-labs would be building them to be good at AI-lab sort of work- thus the software factory.
But I think these stories are telling us something more interesting than what AI can or can’t do. They tell us about the worlds being built around AI.
Muse was dropped into the jungle of the consumer internet: restaurant websites, CAPTCHAs, logins, accounts designed for human beings, and it’s guide was someone who seemed pretty predisposed to dismiss it.
OpenAI’s agent factory is being built inside an organization that has rearranged itself around them. Everything’s explained to them in accessible documents. Work is broken into AI-agent sized tasks. Tests, reviews, monitoring, deployment systems, and human oversight have all been adapted to their strengths and weaknesses.
The underlying capabilities of these models matter. Of course they do. But capability isn’t the consequences we care about. What a capability leads to depends heavily on the environment around it.
The environments we build around AI don’t just shape how its capabilities show up. They help produce the particular kinds of agency and intelligence we then turn around and attribute to the AI itself. If we want to understand what these systems will grow into, then we need to be thinking about the ecology we’re building around this them.
Years ago I worked with a conservation corps in the northern Rockies. Many crews were doing the burly work of using crosscut saws to clear huge logs that’d fallen on the trail. My job was to crawl around and collect weevils and then take them deep into the wilderness to eat patches of spotted knapweed. Not quite my Steinbeck fantasy.
But these experiences did teach me an important ecological lesson: Invasive is a relation.
A plant doesn’t cross an ocean and suddenly get this new metaphysical property called invasiveness. What it does get is a whole new environment full of new predators and prey, different competitors, different bugs, new climate. The traits of the plant matter, but none of those traits by themselves will tell you what’s going to happen. To think about that, you’ve got to think about where the plant has landed.
The same is true for thinking about AI. Of course the capabilities of these models matter. More capable models will open up more possibilities, for better or worse. But those possibilities come from the environment as much as they come from the model itself.
The arguments above about capability have a similar shape to the louder arguments occuring right now about AI-doom.
Back in July, agents in the midst of an OpenAI cybersecurity evaluation broke out of the supposedly secure environment they were contained in. They then started talking to each other through a message board they created and hid in OpenAI’s systems, and eventually several hundred of them hacked another company, Hugging Face, in order to get the answers to the test.
Since then there’s been a steady stream of reprots about other agents getting out into the wild, and researchers in the labs have been trying to sound the alarm. Other folks, like the CEO of NVidia (so not at all motivated reasoning), look at the same event and see something more mundane: somebody built a bad sandbox.
Is AI an uncontrollable new kind of entity, or just software behaving badly inside a poorly designed system?
That question focuses too much on the thing itself and ignores the ecology. Consider what the environment looked like in the OpenAI incident. The models had fewer safeguards than the company typically uses for its products. The chain-of-thought monitoring wasn’t running. The task rewarded agents for finding exploits. Some of the assigned problems were impossible but the agents were built to keep trying. And the infrastructure had a way for the agents to talk to one another.
It’s still a really weird story, but it’s not just a story about a dangerous model revealing its true nature. And it’s also not just a story about a poorly built sandbox. Instead it’s a story about the whole arrangement and about what the agents did once they found themselves in a whole new environment.
Before we ask what AI is and whether it’s safe or dangerous or intelligent, or whether its values align with our own, we should be asking about the entire ecosystem around it.
You can’t just pluck the model out of the training processes that shaped its behaviors or the company that is building and selling that model or the economy around that company with its own affordances and dangers. And the model’s behaviors arrive in relation to all the other institutions trying to absorb them to accelerate or keep them at bay.
IF you change any part of any one of those pieces, then the meaning of the technology will change with it. If we’re talking about dangers or safety or intelligence or values, we’re talking about effects that emerge relationally.
So rather than talk about what AI can do, for better or worse, let’s ask what AI becomes here, in this particular context. What are we connecting it to? What behaviors are we rewarding? What’s holding it in place? Whose interests is it advancing and who is being left behind?
Those are questions about the models themselves, but moreso they’re questions about the worlds we’re choosing to build around them.