A Million Agents Is a Distributed Systems Problem

I think about agents the way I think about people. One agent is a worker. A thousand agents is an organization, and an organization of machines is a distributed system.

People keep telling me agents can run forever. Technically that's true. You can call a model in a loop until your credit card declines. But the agent still runs on finite stuff: tokens, context, compute, memory, tools, money. Something always runs out.

Humans aren't that different. We work for a while and get tired. Our working memory is tiny. We forget things. So we write the important parts down, sleep, and come back the next morning with a clear head and the same identity.

Machines have the same constraint in a different shape. A box runs out of CPU. A pod runs out of memory. Nobody fixes that by assuming every process should live forever. We schedule work over the resources we have.

I don't see why intelligence gets a pass. An agent can work for a while, write down what matters, clear its context, and let itself or another agent pick up from there. We run our own coding agents on VPSes at InsForge, and they get killed, restarted, and run out of context all the time. The failures that actually hurt are the ones where the plan lived only inside the agent's context window.

TL;DR

• Adding agents helped parallel work by up to 80.9% and hurt sequential work by 39% to 70% in Google Research's 180-configuration study. The shape of the task decides.
• An orchestrator cut error amplification from 17.2× to 4.4× in the same study. Coordination is a real job.
• In Silo-Bench (ACL 2026), teams of 2 to 100 agents talked plenty and reasoned badly. The hardest tasks hit zero success at 50 agents.
• So the durable thing should be the state, not the agent. Schedule agents like processes and recover them like nodes.

Agents are starting to look like processes

This stopped being an analogy a while ago. There's a whole research line building it.

AIOS, an "LLM Agent Operating System" out of Rutgers, opens with the problem in one sentence:

"Allowing unrestricted access to LLM or tool resources can lead to inefficient or even potentially harmful resource allocation and utilization for agents."

Their answer is a kernel. Every agent request gets broken into system calls (an LLM call, a memory read, a storage write, a tool use), a scheduler decides whose call runs next using the classics, First-In-First-Out and Round Robin, and a context manager snapshots an agent mid-task so it can be interrupted and resumed. They report up to 2.1× faster execution when serving agents built on existing frameworks.

Scheduling, context switching, memory management, storage, access control. That's an operating system. The processes just happen to think.

It pays one level up too. LLM-as-Scheduler, from ACL 2026, starts from the observation that most queries don't deserve a heavy multi-agent workflow, and lets a scheduler pick the workflow per query. They got 43% fewer tokens and more than 36% lower end-to-end latency, for at most a 1.4 percentage-point drop in accuracy against a strong fixed workflow.

So one agent looks like a process, and thousands of processes need a scheduler. Fine. But scheduling compute is only half of it. You also have to coordinate the agents with each other, and that's where it gets interesting.

Ten people coordinate. Ten thousand invent managers.

Ten people can coordinate themselves in a room. A thousand people can't all talk to each other and independently decide what the company should do, so we invented teams, managers, departments, and eventually a CEO. Managers exist partly because coordination is itself work, and someone has to do it.

I assumed agents would have the same problem. Now there's data.

Google Research and MIT ran a controlled study of 180 agent configurations across five architectures (single agent, plus independent, centralized, decentralized, and hybrid multi-agent) and three model families. Their headline:

"the 'more agents' approach often hits a ceiling, and can even degrade performance if not aligned with the specific properties of the task"

Same five architectures, opposite outcomes, decided by the shape of the task. Source: Google Research, Towards a science of scaling agent systems, January 2026.

On tasks that need strict sequential reasoning, every multi-agent variant they tested made things worse, by 39% to 70%. Their explanation is that the communication overhead fragmented the reasoning and left too little "cognitive budget" for the actual task. On parallelizable work like financial reasoning, centralized coordination improved performance by 80.9%.

More agents didn't buy more intelligence. They bought more coordination, and coordination eats the same budget the task needs.

The second finding is the one I keep coming back to.

Agents working in parallel without talking amplified errors by 17.2×. Put an orchestrator in front of them and it dropped to 4.4×. That's what a manager is for. The manager doesn't do every task. It decides what needs to happen, breaks the work apart, assigns it, watches progress, resolves conflicts, and combines the results.

Communication is not coordination

This one sounds the most human to me.

Silo-Bench, accepted at ACL 2026, gave teams of 2 to 100 agents thirty distributed algorithm tasks, with the data sharded so no single agent could see everything. 54 configurations, 1,620 experiments. The agents could message peers, broadcast, or share files.

They talked a lot. It didn't help much.

"agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers."

The authors call this the Communication-Reasoning Gap, and their one-line version is better than anything I'd write: "agents are competent communicators but poor distributed reasoners."

It also gets worse with scale.

The trend is down at every difficulty level, with a couple of small upticks along the way (Level I from 50 to 100 agents, Level II from 10 to 20), and none of them come close to recovering. The hardest tasks, the ones that need every shard combined at once, hit zero success at 50 agents and stay there at 100. Across models, the average success rate fell from 61% with 2 agents to 18% with 100 for the strongest model, and from 17% to 1% for the weakest. The authors' conclusion: "coordination overhead compounds with scale, eventually eliminating parallelization gains entirely."

If you've ever been in a 100-person Slack channel you already knew this. Putting 100 people in one channel doesn't give you an organization. Putting 100 agents on one message bus doesn't give you collective intelligence either.

What happens when the manager fails?

Most multi-agent systems today coordinate through a central orchestrator. A master agent decomposes the goal, hands tasks to workers, watches them, and merges the results. The Google numbers say that's a reasonable default, and it's what we do too.

Distributed systems people know what comes next. The master becomes the bottleneck. Then the master fails. It runs out of context, or gets stuck, or its machine dies, or it just starts making bad calls.

The AgentNet authors (NeurIPS 2025) say it directly: existing systems "often rely on centralized coordination, leading to scalability bottlenecks, reduced adaptability, and single points of failure." Their answer is decentralized routing, where agents pass work among themselves based on local expertise. Ayush Chopra at the MIT Media Lab goes further: "genuine distributed systems exhibit coordination that emerges from agent interactions," and "the intelligence is in the interaction patterns themselves, not in any individual decision-maker."

I don't think you have to pick a side here. Distributed systems solved versions of this decades ago with durable state, heartbeats, failure detection, consensus, failover, and leader election. If the leader disappears, another node takes over, and it can do that because the state it needs was never inside the leader.

A reliable system can't depend on one intelligent process staying alive forever. So the thing that matters isn't the lifetime of an agent. It's the lifetime of the state. Goals, plans, decisions, done tasks, pending tasks, artifacts, ownership, checkpoints. Those should outlive any individual agent. Agents come and go. The organization continues.

Christopher Meiklejohn, who spent his career on distributed programming, put it bluntly this spring: "the moment you have multiple agents working on the same codebase, you have a distributed system," and the problems that follow "aren't a bug. [...] They're an inevitable consequence of having multiple autonomous processes that share state."

The interesting problem

So you need a scheduler, some hierarchy, memory that outlives the worker, clear ownership, backpressure, failure recovery, and enough observability to know which agent is stuck. Sometimes you need a leader, and a way to replace it.

That's why I think the interesting infrastructure question isn't "how do we build an agent that runs forever?" It's "how do we schedule and coordinate millions of intelligent workers that don't?"

Operating systems taught us how to schedule processes. Container orchestrators taught us how to schedule and recover ephemeral workloads. Distributed systems taught us how unreliable machines can add up to a reliable service. Human organizations taught us how individually limited people build things no single person could. Agents sit right where those four overlap.

A single agent is an intelligence problem. A million agents is a distributed systems problem.

This post is about the problem. What the infrastructure for a million agents should actually look like, the checkpoints, the scheduler, the budget a worker gets, the handoff when a coordinator dies, deserves its own post, and that's the next one I'll write.

If you liked this one, tell me, or follow me on X or LinkedIn so you catch the next one.

FAQ

Sources

• Kim, Liu et al., Towards a science of scaling agent systems: When and why agent systems work, Google Research blog, January 28, 2026. Paper: arXiv:2512.08296.
• Zhang et al., Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems, ACL 2026 main conference. Code.
• Mei et al., AIOS: LLM Agent Operating System, Rutgers University, 2024, revised 2025.
• Xiang et al., LLM-as-Scheduler: Agentic Workflow Dynamic Scheduling, ACL 2026.
• Yang et al., AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems, NeurIPS 2025.
• Ayush Chopra, What is a Multi-Agent System?, MIT Media Lab, June 16, 2025.
• Christopher Meiklejohn, Multi-Agent Systems Have a Distributed Systems Problem, March 30, 2026.