AGI’s Bureaucratic Future

Bureaucracy is everywhere, from continent-spanning governments to rapidly growing corporations. Everywhere, that is, except discussions of our AGI future.

Most visions of AI government converge on democracy, improved by machine-assisted deliberation, and markets, populated by personal agents bargaining on our behalf.

Both are worth exploring. But the discourse often leaves out the form of governance that has proved most durable. As Max Weber and others have argued, and as experience has proved, technological progress demands more bureaucracy to handle the information problems of a complex world, while supplying the tools for that bureaucracy to operate.¹

I get it. Bureaucracy is boring. Who wants to spend time thinking and arguing about what constitutes a “stationary source” or a “security,” or researching and deciding how much fecal matter is acceptable in food or lead in paint or any of the other mundane and specific tasks that make up modern government?

A century ago, Weber described bureaucratized modernity as an “iron cage”: a world so ordered, regulated, and administered that no feeling person can escape it.² Isn’t AI supposed to crack the cage open? It’s much more fun to imagine AI rendering these massive structures unnecessary, replacing them with more direct, flexible, and personal forms of government.

But too bad.

Setting and enforcing predictable rules is part of what makes markets and democracies function in the first place. Even highly intelligent AI agents will need settled categories and shared information on which to operate, negotiate, plan, and act. Bureaucracy has been with us since early antiquity.³ It will probably follow us into the AI future.

That does not have to be a bad thing.

I’m not saying don’t worry about the government of a bureaucratic character alien to our system. I’m saying that bureaucracy is already serving us more than we give it credit for, and with AGI, it could be much better still. The real question is not whether we will have bureaucracy in the AI future (we almost certainly will), but how AI could make bureaucracy more capable, more accountable, and more responsive.

What even is bureaucracy?

Max Weber describes bureaucracy as a system of government defined by hierarchy, expertise, continuity, and impartiality, and that operates by the development and application of rules to defined jurisdictional areas.⁴

For example, take the Food and Drug Administration. It’s a hierarchy of review divisions under center directors. It’s staffed by pharmacologists and statisticians with expertise, who outlast any administration, with jurisdiction over drugs and biologics (but not pesticides, because that’s EPA), applying impartiality through rules like the bioequivalence standards that determine when a generic can substitute for a brand-name drug.

Weber argued that bureaucracy is to other forms of government as machines are to artisanal production: its benefits are precision, speed, unambiguity, continuity, efficiency, and an ability to handle novelty and complexity, but it comes at the cost of impersonality.

Of course, individualized judgment is often more accurate in any single case (unless it’s biased or corrupt), but it’s expensive and difficult for others to forecast. Rules are crude and necessarily create false negatives and positives, but they are cheap to apply and everyone can plan around them. These are important qualities in our modern life: anyone who has plugged their computer into a standardized outlet in a foreign country can understand the virtue of predictability at scale. When millions of actors need to know today what a decision-maker will do tomorrow, the rule beats the sage. AI could shift this boundary between personalization and standardization, as cheaper local information could let us afford more contextual adjudication. But a world of swarming agents could also multiply interactions so fast that the balance tips back toward rules.

The case for bureaucracy

I mentioned earlier that bureaucracy has grown massively in the past century. Why? Because it solves three recurring problems of governance:

• Predictability through the rule of law. Treating like cases alike creates the settled expectations on which institutions and individuals depend.
• Cheaper coordination through hierarchy. Standing relationships, defined roles, and command powers avoid the cost of renegotiating every decision.⁵
• Insulated expertise. Some technical decisions should be protected from both electoral pressure and market bargaining. We do not want drug-approval standards set by plebiscite or auction.

Behind all three is the problem of dealing with complexity. As the world has grown more complex, people have specialized. Delegating technical decisions to experts makes the process of decision-making much more accurate and efficient than demanding that members of the public express preferences through voting or market signals.

But specialization creates a problem of its own. How can a non-specialist know that a specialist is acting in good faith? Bureaucracy’s answer is to bring the specialists under one roof and bind them to rules. Then they can oversee each other, build mechanisms for review and appeal, and catch the non-technical kinds of cheating, like favoring friends or special interests, by comparing one official’s output against the organization’s.

When bureaucracy is done right, it can handle immense complexity and scale with great consistency.

For example, each month the Social Security Administration sends benefits to more than seventy million people with an error rate most private insurers would envy.⁶ Oversight is done through internal appeals, inspectors general, records that force reasoning into the open and courts policing the edges of delegated power all create limits on abuses of power.

Consider also the rule-making apparatus of the United States, that on average issue three to four thousand final rules by agencies every year. Millions of administrative adjudications apply those rules across essentially every part of the economy and society.⁷

These numbers are often cited as examples of bureaucracy run amok, but what is the alternative? Congress cannot pass thousands (or even hundreds? tens?) of laws. Article III courts cannot handle their existing workloads, let alone the millions of other cases that they would have to address if agencies didn’t do it for them. And both the legislature and the judiciary lack the technical expertise to correctly decide these cases, even if they were assigned them—how could Congress quickly learn enough to decide whether a specific cutting-edge drug should come on the market, or whether the national ambient air quality standard should be changed, among millions of other things?

As I argue in a forthcoming law review article, the administrative state came into being against significant societal and judicial resistance because it was necessary, and it is even more necessary today. Complexity is not going away, and neither are the three rationales of bureaucracy (predictability, cheaper coordination, insulated expertise) that complexity calls forth.

Why agents can’t just “handle it”

Some people argue that as AI gets smarter, AI agents could bargain or deliberate on our behalf and remove the need for bureaucracy.

I think these arguments either miss the point or implicitly envision a world in which people are disempowered by their AI agents. Because after all, democratic deliberation and market bargaining happen in the context of set categories and according to set rules.

As Coase acknowledged, efficient bargaining is possible only when property rights are established up front.⁸ But property rights are not exogenous to a system of government, they are established by it. And the relevant property rights are less frequently “who owns this plot of land?” and more frequently “what is the legal nature of this ownership share of a company?” Rules like that are produced by bureaucracies like the Securities and Exchange Commission, and then the SEC enforces those rules, which provides a basis for the market.

Imagine if every time you tried to buy a share in a company you had to negotiate with that company as to what a “share” means. Or imagine buying food in a world in which negotiation forms the basis for food safety. Are you supposed to develop a preference for how much arsenic you’re willing to have in your cereal? How much lead? How much feces? If you do, who do you negotiate with? The grocery store? The wholesaler? Are the supply chain companies supposed to each negotiate with each other to determine which products along the spectrum of safety to supply at which prices?

We tried something close to the market approach for these questions once. It gave us the rotten and adulterated meat described in The Jungle, and a public that quickly demanded food inspectors.⁹

Similarly, if the aggregation mechanism is voting, are people supposed to determine for themselves what a “security” means and then somehow deliberate collectively on that to come to a consensus definition? Even experts often don’t know what these things mean and have long debates over what’s best.

“Fine. Just let the agents handle it,” you say.

Fair enough. A sufficiently capable agent could negotiate purchases, represent your preferences, and spare you from forming an opinion about the permissible arsenic content of breakfast cereal without involving a third party.

But the agent still needs settled rules. It needs to know what a share is before it can buy one, who owns what before it can bargain, and which agreements other agents and institutions will recognize tomorrow. A world of perpetual agent-to-agent negotiation is a world of unnecessary and costly contracting.

“Fine,” you say. “Then let the agents decide those technical questions too.”

Perhaps they should. Let them investigate the evidence, make consistent decisions, and insulate those decisions from preferences you do not want to develop moment by moment.

Now look again. An entity trusted to decide technical questions consistently, on the basis of specialized knowledge, so that you do not have to decide them yourself?

Well, that is a bureaucracy.

Furthermore, an AI that completely understands you and acts perfectly on your behalf in every meaningful interaction that affects you is an agent that has disempowered its principal. That disempowerment might be beneficial to the principal, assuming actual congruence between the agent’s representation of the principal’s interests and the true interests of the principal, but it is a world with more complete control than even our most highly-bureaucratized one. It also carries a cost to democratic legitimacy: an agent that thinks for you severs the link to an informed citizenry.

It seems likely that we’ll get a hybrid world, in which human preferences are gathered and represented with much more granularity than they are today by intelligent AI agents and that these agents interact on their humans’ behalf through democracy and markets. But bureaucracy, with its consistency and predictability, its ability to operate in technical scientific domains through specialization, and its long history of incorporating and making the best use of technology, will remain core to the institutions of the future.

Bureaucracy’s problems

Assuming that I’m right, and that bureaucracy will stay with us, what might the future of AI administration look like? It’s useful to briefly consider some of bureaucracy’s current pathologies and failures to see what AI might help us fix, and also where we might get stuck if we don’t think hard about how this technology will be developed.

The pathologies of bureaucracy are familiar to anyone interfacing with a government agency. I would summarize them as follows:

• Inefficiency. Everything takes longer and costs more than it should.
• Capture. The companies an agency is supposed to police can end up steering it. This influence is hidden from the public because it is buried in paperwork nobody reads.
• Ossification. Agencies can barely change anything, even their own outdated rules. Every check we put on them slows them down, so changing one rule can take years.
• Opaque decisions. A form letter says no; it doesn’t say why.
• Inflexibility. Rules are written for the typical case. If your case is unusual, the rules have no place for you.

Additionally, bureaucracy can be myopic. James C. Scott is the most prominent exponent of the “legibility” critique of bureaucracies, that governments force society into governable shape, stripping it of much of its richness. His argument was already present in Weber, who argued that bureaucracy’s need for consistency and reliance on paperwork in the form of “files” entailed the forced simplification and rationalization of the world.¹⁰

The loss of texture and specificity is a price to be paid for scale when perceiving the world and acting on it is costly in terms of attention, time, and cognition—in the same way that the individual human brain ignores almost all of our sensory inputs so that it can focus on what is important, so must any kind of governance find mechanisms by which to focus itself.

It is, in some sense, a miracle that we can run societies as complex as ours using the highly lossy interfaces of bureaucratic institutions, and it is only through continual technological upgrading that we have managed to scale this far.

I’m mostly defending bureaucracy in this essay, or at least defending the need for it now and in the future. But reform is obviously needed. Existing efforts from state capacity advocates and modernizers across the political spectrum are bearing some fruit, but there is also a real chance that our agencies will fall further behind the technical frontier, like how they mostly failed to reinvent themselves for the internet and current AI capabilities.

Many of the obstacles to reform are well-intentioned limits that were designed to prevent the arbitrary and abusive exercise of bureaucratic power, like many of the barriers of administrative law that have led to ossification. We need reform that makes bureaucracy both more capable of solving our problems and more accountable to the people and their representatives, escaping the trap that has meant that capability and accountability trade off across time. And maybe AI can help us get there.

A better bureaucracy

So how could we build a better kind of bureaucracy, one suited to this new technological moment we’re facing? And how can AI help us? Recall the three rationales of bureaucracy: predictability, cheap coordination, and insulated expertise. Each can be served by frontier AI, if it is implemented correctly.

What would AI-powered bureaucracy look like? The basic functions of bureaucracy, as Simon reminds us, are cognitive. An organization is established with a specific mission toward which it is directed, like reducing air pollution. It observes its environment, seeking opportunities to achieve its mission, like an industrial sector whose factories emit pollutants. It gathers information, classifies the world into administrable categories, applies rules and makes decisions about those categories, and observes what it has done so that it can learn from its decisions.

Bureaucracy is society’s organized system for perceiving, reasoning, deciding, and remembering at scale.

Frontier AI could change each part of that system.

It can help agencies see more of the world, with less delay, less distortion, and in more detail, by processing administrative records, scientific studies, complaints, market data, inspections, and other inputs that currently overwhelm human officials and are hugely costly in time and effort.

It can help agencies reason across that information by identifying patterns, surfacing tradeoffs, operating across different assumptions, and testing whether proposed interventions are likely to actually advance statutory goals.

It can help agencies better explain themselves to the public and their overseers in courts and legislatures, not only in the formal language of the Federal Register, but in terms that affected parties can actually understand. Instead of someone denied welfare just getting a form letter saying that they don’t qualify, they could get an individualized explanation that helps them specifically understand what they need to do on appeal.

And it can help agencies remember, by preserving a more complete and searchable institutional memory of what has been tried, what failed, what worked, and why.

The scrutable state

Perhaps most importantly, AI could help make bureaucracy scrutable.

Bureaucracy alienates because it simplifies the world while remaining obscure itself, clouded in technical language and burdensome procedure. The people are made legible to the state, but the state is not made legible to the people.

AI could reverse that asymmetry. Acting either as a personal agent or deployed by the government agency, AI could help a person denied disability benefits, a business subject to enforcement of some rule, a member of Congress overseeing a technical agency, or a judge reviewing a complex informal rulemaking ask, in ordinary language: what happened here, what mattered, what assumptions drove the decision, what alternatives were considered, and what would have changed the outcome?

All this only works if AI is itself governed bureaucratically—that is, consistently, efficiently, and according to rules of law. Technological improvements in bureaucracy have always been paired with innovations in oversight that maintain accountability in these institutions. AGI needs its own.

And here is the deeper promise, because the failures catalogued above are mostly failures of oversight, and AI drastically changes what oversight can be:

• Capture works because it is hidden and costly to root out. But a system whose reasoning is documented and queryable makes influence traceable, and traceable influence is checkable influence.
• Ossification builds up because the only way to check an agency has been to slow it down. But oversight built on auditing and evaluation checks without braking.
• Opaque decisions can’t survive when every determination carries its reasons with it.

AI could help us break out of the old trap, in which every gain in accountability cost us capability, and every gain in capability escaped accountability.

But none of it happens on its own. We need a new administrative law for this new administration, a new administrative record that documents what a system is for, what data is used, how it was evaluated and monitored, and what failure modes have been found. Judicial and legislative oversight could be built on the budding science of AI auditing and evaluation, creating a new form of accountability that is built for AI.

The Iron Cage, Renovated

We shouldn’t replace bureaucracy with AI or just paste chatbots onto existing agencies and call that modernization. AI offers the chance to build a vastly more capable and more accountable bureaucracy, which can handle complexity and avoid most forced simplification, act at scale without becoming arbitrary, explain itself comprehensibly to affected parties and the public, and solve our worsening problems.

The AGI future is going to be weird, and someone needs to help set the rules and conditions under which society can smoothly operate, from creating new forms of property to handling new externalities. Therefore, the AGI future will still be bureaucratic, maybe even more than today. But if we get it right, the iron cage Weber warned of becomes a structure that enables us more than it constrains us.