AGI’s Bureaucratic Future

Most visions of a future AI government converge on two possibilities: democracies improved by machine-assisted deliberation, and markets transformed by personal agents bargaining on our behalf.¹ But the discourse often leaves out the form of governance that has proved most durable: bureaucracy.

Bureaucracy is everywhere, from continent-spanning governments to rapidly growing corporations. As German sociologist and jurist Max Weber argued, and as we’ve learned in the century since his death, technological progress demands more bureaucracy to handle the information problems of an increasingly complex world. It also supplies the tools that allow bureaucracy to operate at greater scale.² Yet this growing dependence is easy to overlook, because the work itself is profoundly unglamorous.

Let’s be honest: bureaucracy is boring. Who wants to spend time thinking and arguing about what constitutes a “stationary source” or a “security,” or researching and deciding how much fecal matter is acceptable in food or lead in paint or any of the other mundane and specific tasks that make up modern government?

More than a century ago, Weber described bureaucratized modernity as an “iron cage”: a world so ordered, regulated, and strictly administered that no feeling person could escape it.³ But isn’t AI supposed to free us from such a cage? Won’t AI render these massive structures unnecessary, replacing them with more direct, flexible, and personal forms of government?

I highly doubt it. Setting and enforcing predictable rules is part of what makes markets and democracies function in the first place. Even highly intelligent AI agents will need settled categories and shared information on which to operate, negotiate, plan, and act. Bureaucracy has been with us since early antiquity.⁴ It will probably follow us into the AI future. But that is not necessarily a bad thing.

I’m not saying don’t worry about the government of a bureaucratic character alien to our system. I’m saying that bureaucracy is already serving us more than we give it credit for, and with AI, or AGI, it could be much better still. The real question is not whether there will be bureaucracy in the AI future (there will be), but how AI could transform bureaucracy and make it more capable, more accountable, and more responsive.

What even is bureaucracy?

Weber described bureaucracy as a system of government defined by hierarchy, expertise, continuity, and impartiality, one which operates by the development and application of rules to defined jurisdictional areas.⁵

Take as an example the U.S. Food and Drug Administration. The FDA is a hierarchy of review divisions, each reporting up to a director who heads one of the agency’s centers. Each division is staffed by pharmacologists and statisticians with expertise, who outlast any administration, and have jurisdiction over drugs and biologics (but not pesticides, because those are covered by the EPA). They apply impartiality through rules like the bioequivalence standards that determine whether a cheaper generic can be used in place of a brand-name drug.

Bureaucracy is society’s system for perceiving, reasoning, deciding, and remembering at scale.

Weber argued that bureaucracy is to other forms of government as machines are to artisanal production. Its benefits are precision, speed, clarity, continuity, and efficiency, as well as the ability to handle novelty and complexity, though it comes with the loss of personalization as its cost.

Of course, an individualized judgment is often more accurate (unless it’s biased or corrupt), but one-off judgments are expensive, inefficient and difficult for others to forecast. Rules are crude and create false negatives and positives, but they are cheap to apply and everyone can plan around them. These are important qualities in modern life. Anyone who has plugged their computer into a standardized outlet in a foreign country can understand the virtue of predictability at scale. When millions of actors need to know today what a decision-maker will do tomorrow, a broadly applied rule has far more to offer than a slow and mercurial sage.

AI could shift the boundary between standardization and personalization, making individualized judgment cheap enough to apply at scale. But a world of swarming agents could also multiply interactions so fast that the balance tips back toward predictable rules.

The case for bureaucracy

Bureaucracy has grown massively in the past century because it solves three recurring problems of governance. The first is predictability through the rule of law: treating like cases alike creates the settled expectations on which institutions and individuals depend. The second is cheaper coordination through hierarchy: standing relationships, defined roles, and command powers avoid the cost of renegotiating every decision. The third is insulated expertise. Some technical decisions should be protected from both electoral pressure and market bargaining. We do not want drug-approval standards set by plebiscite or auction.

Behind all three is the question of how we deal with complexity. As the social world has grown larger, people have specialized. Delegating technical decisions to experts makes the process of decision-making far more accurate and efficient than demanding that members of the public express preferences through voting or market signals.

But specialization creates problems of its own. How can a non-specialist know that a specialist is acting in good faith? Bureaucracy’s answer is to bring the specialists under one roof and bind them to rules. That way they can oversee each other’s work, build mechanisms for review and appeal, and catch non-technical kinds of cheating like favoring friends or special interests by comparing one official’s output against the organization’s.

When bureaucracy is done right, it can handle immense complexity and scale with great consistency. For example, each month the Social Security Administration sends benefits to more than seventy million people with an error rate most private insurers would envy.⁶ Oversight is carried out through internal appeals, inspectors general, records that force reasoning into the open, and courts which police the edges of delegated power in an attempt to curb abuse.

Consider in parallel the rule-making apparatus of the United States government that on average issues three to four thousand final rules from its agencies every year. Millions of administrative adjudications apply those rules across every part of the economy and the full breadth of society.⁷

These numbers are often cited as examples of bureaucracy run amok. But what is the alternative? Congress cannot pass thousands (or even hundreds? tens?) of laws. Federal courts cannot handle their existing workloads, let alone the millions of other cases they would have to address if agencies didn’t do it for them. Both the legislature and judiciary lack the technical expertise to correctly decide these cases. How could Congress quickly learn enough to decide whether a specific cutting-edge drug should come on the market, or whether the national ambient air quality standard should be changed, among a million other things?

As I argue in a forthcoming law review article, the administrative state came into being against significant societal and judicial resistance because it was necessary, and it is even more necessary today. Complexity is not going away, and neither are the three rationales of bureaucracy that complexity calls forth: predictability, cheaper coordination, and insulated expertise.

Why agents can’t just “handle it”

Some people have argued that as AI gets smarter, AI agents could bargain or deliberate on our behalf and remove the need for bureaucracy. I think these arguments either presuppose the bureaucracy they claim to eliminate or implicitly envision a world in which people are disempowered by their AI agents. After all, democratic deliberation and bargaining within the market happen in the context of set categories and according to set rules.

A world of perpetual agent-to-agent negotiation is a world of unnecessary and costly contracting.

As British economist Ronald Coase acknowledged in the 1960s, efficient bargaining is possible only when property rights are established up front.⁸ But property rights do not exist outside a system of government, they are established by it. And the relevant property rights are less frequently “who owns this plot of land?” and more frequently “what is the legal nature of this ownership share of a company?” Rules like that are produced by bureaucracies such as the United States Securities and Exchange Commission. Next, the SEC enforces those rules, which provides a basis for the market.

Imagine if every time you tried to buy a share in a company you had to negotiate with that company to determine what a “share” means. Or imagine buying food in a world in which negotiation forms the basis for food safety. Are you supposed to develop a preference for how much arsenic you’re willing to have in your cereal? How much lead? How many insect fragments? If you do, who do you negotiate with? The grocery store? The wholesaler? Are supply chain companies supposed to negotiate with each other to determine which products along the spectrum of safety to supply at which prices?

We tried something close to the market approach to these questions once. It gave us the rotten and adulterated meat described by Upton Sinclair in The Jungle, and a public that quickly demanded food inspectors.⁹

Similarly, if the aggregation mechanism is voting, are people supposed to determine for themselves what a “security” means and then somehow deliberate collectively on that to arrive at a consensus definition? Even experts often don’t know what these things mean and have long debates with one another to figure out what’s best.

“Why can’t we just let the agents handle it?” you might say. A sufficiently capable agent could negotiate purchases, represent your preferences, and spare you from forming an opinion about the permissible arsenic content in breakfast cereal without involving a third party.

But the agent still needs settled rules. It needs to know what a share is before it can buy one, who owns what before it can bargain, and which agreements other agents and institutions will recognize. A world of perpetual agent-to-agent negotiation is a world of unnecessary and costly contracting. This is not to say that agents shouldn’t be involved in resolving technical questions. Let them investigate the evidence, make consistent decisions, and insulate those decisions from preferences you do not want to develop moment by moment.

Now look again. An entity trusted to decide technical questions consistently, on the basis of specialized knowledge, so that you do not have to decide them yourself?

Well, that is a bureaucracy.

Furthermore, an AI that completely understands you and acts perfectly on your behalf is an agent that has disempowered its principal. That disempowerment might be beneficial to the principal, assuming actual congruence between agent and the principal’s interest, but it is a world with more complete control than even the most highly-bureaucratized societies today. It carries a cost to democratic legitimacy: an agent that thinks for you is incompatible with an informed citizenry.

The goal, then, should be a hybrid world in which human preferences are gathered and represented with much greater fidelity than they are today, without replacing human judgment altogether. But even in that world, bureaucracy — with its consistency and predictability, its ability to operate in technical or scientific domains through specialization, and its long history of incorporating and making the best use of technology — will remain core to the institutions of the future.

The limits of bureaucracy

Assuming that I’m right and that bureaucracy will always be with us, what might the future of AI administration look like? Here we might briefly consider some of bureaucracy’s current pathologies to see where AI might step in, and also where we might get stuck if we don’t think hard enough about how this technology is being developed and implemented.

The failures of bureaucracy will be familiar to anyone who interfaces regularly with a government agency. Everything takes too long and costs more than it should. The companies an agency is supposed to police often end up steering it — something which goes unnoticed because the influence is buried in paperwork nobody reads.

Agencies can barely change anything, even their own outdated rules. Every check we put on them slows them down so that changing one rule can take years. When they do make a decision, a form letter says no and doesn’t explain why. The rules tend to be written for the typical case, so if your case is unusual, they reject you.

Also, bureaucracy can be myopic. The late political scientist and anthropologist James C. Scott was the most prominent exponent of the “legibility” critique of bureaucracies: the idea that government forces society into governable shape, stripping it of much of its richness. His argument was already present in Weber, who argued that bureaucracies, public and private alike, force the simplification and rationalization of the world through their need for consistency and reliance on paperwork.¹⁰

Bureaucracy alienates because it simplifies the world while remaining obscure. The people are made legible to the state, but the state is not made legible to the people.

The loss of texture and specificity is the price to be paid for scale when perceiving the world and acting on it is costly in terms of attention, time, and cognition. It’s not unlike the way the individual human brain ignores almost all sensory inputs so that it can focus on what it deems important. It is a miracle of sorts that we can run societies as complex as ours using the highly lossy interfaces of bureaucratic institutions, and it is only through continual technological upgrades that we have managed to scale this far.

Even though I’m primarily defending bureaucracy in this essay, now and in the future, reform is certainly needed. Existing efforts from state capacity advocates and modernizers across the political spectrum are bearing some fruit,¹¹ but as the AI rollout continues there is also a real chance that our agencies will fall further behind the technical frontier, much as many of them failed to reinvent themselves for the internet age.

Many of the obstacles to reform are well-intentioned limits designed to prevent the arbitrary and abusive exercise of bureaucratic power, like many of the barriers of administrative law that have led to ossification. We need reform that makes bureaucracy both more capable of solving our problems and more accountable to people and their representatives, escaping the trap that has meant capability and accountability trade off across time. It may well be that AI can help us get there.

Bureaucracy, rebuilt

So how could we build a better kind of bureaucracy, one suited to this new technological moment we’re facing? And how can AI help us to do it? Recall the three rationales of bureaucracy: predictability, cheaper coordination, and insulated expertise. Each can be served by frontier AI if implemented correctly.

What might an AI-powered bureaucracy look like? The basic functions of bureaucracy, as Herbert Simon reminds us, are cognitive.¹² An organization is established with a specific mission toward which it is directed, like reducing air pollution. It observes its environment, seeking opportunities to achieve its mission, like an industrial sector whose factories emit pollutants. It gathers information, classifies the world into administrable categories, applies rules and makes decisions about those categories, and observes what it has done so that it can learn from its decisions.

Bureaucracy is society’s system for perceiving, reasoning, deciding, and remembering at scale. Frontier AI could change each part of that system. It can help agencies see more of the world, with less delay, less distortion, and in more detail, by processing administrative records, scientific studies, complaints, market data, inspections, and other inputs that currently overwhelm human officials.

It can help agencies reason across that information by identifying patterns and surfacing tradeoffs, operating across different assumptions, and testing whether proposed interventions are likely to actually advance statutory goals.

It can help agencies better explain themselves to the public and their overseers in courts and legislatures, not only in the formal language of the Federal Register, but in terms that affected parties can actually understand. Instead of someone denied welfare getting a form letter saying they don’t qualify, they could get a bespoke explanation that helps them understand what they need to do on appeal.

Lastly, it can help agencies remember, by preserving a more complete and searchable institutional memory of what has been tried, what failed, what worked, and why.

The scrutable state

Bureaucracy alienates because it simplifies the world while remaining obscure, clouded by technical language and burdensome procedure. The people are made legible to the state, but the state is not made legible to the people. Perhaps most importantly, AI could help make bureaucracy scrutable.

Whether deployed as a personal agent or by a government agency, AI could help a person denied disability benefits, a business subject to enforcement, a member of Congress overseeing a technical agency, or a judge reviewing a complex informal rulemaking. In each case the user is able to ask in ordinary language: what happened here, what mattered, what assumptions drove the decision, what alternatives were considered, and what would have changed the outcome?

This only works if AI is itself governed bureaucratically — that is, consistently, efficiently, and according to the rule of law. Technological improvements in bureaucracy have always been paired with innovations in oversight necessary to maintain accountability in these institutions. AI will be no exception.

Most of the failures mentioned earlier are failures of oversight, and AI drastically changes what oversight can be. Capture is an issue because it is hidden and costly to root out. But when a system’s reasoning is documented and queryable, influence becomes traceable, and traceable influence is checkable influence. Ossification builds up because, until now, checking an agency usually meant slowing it down, by requiring additional procedures or approvals. AI systems could instead be continuously audited, with their decisions sampled for bias or outside influence. That makes it possible to check the agency without placing a brake on every individual decision. And opaque decisions could be eliminated when every determination carries its reasons with it.

None of it happens on its own. We need a new administrative law for this new kind of administration, a new type of record that documents what a system is for, what data is used, how it was evaluated and monitored, and what failure modes have been found. Judicial and legislative oversight could be built on the budding science of AI auditing and evaluation, creating a new form of accountability built for AI.

Bureaucracy in the future

Nothing I’ve written here should suggest I want to replace bureaucracy with AI or paste chatbots onto existing agencies and call it modernization. AI offers the chance to build a vastly more capable and more accountable bureaucracy, capable of handling complexity and avoiding much forced simplification, acting at scale without becoming arbitrary, explaining itself to affected parties and the public. The future will still be bureaucratic, maybe even more than today. But if we get it right, the iron cage Weber warned of becomes a scaffold rather than a cell.