There is a scene near the end of Brave New World in which the World Controller explains to the Savage how the World State came to be: that after a century of war and collapse, humanity chose stability over freedom, and that the machinery of contentment was built, deliberately, to keep a population placid without having to kill it. When I think about where we are now, my mind keeps returning to that conversation, because the question it answers is the question in front of us: how does a state pass through its adolescence, the period when it first acquires the power to see and shape everyone, without either being torn apart or devouring the people it exists to govern? Huxley, to his credit, offered an answer. I wish it were one we could use.
I believe we are entering a rite of passage for government, turbulent and inevitable, that will test who we are as a civilization. The state is about to be handed the ability to observe every citizen at all times and to act on what it observes, and it is deeply unclear whether our institutions possess the maturity to wield this.
In my earlier essay, All Watched Over by Ministries of Loving Grace, I tried to describe the civilization on the far side of this passage: one in which every person is observed with skill and compassion, and observation raises the quality of every life. I felt it was important to give people something to hope for, a task at which both the accelerationists and the privacy advocates had oddly failed. In this essay I want to confront the passage itself: to map the risks, and to begin drawing up a battle plan.
I should say at the outset where I am writing from. Our company does not study total states from the outside. We operate them. At the small end is the Settlement, a community of about twelve hundred people in the hills above our headquarters, which has been under complete observation for six years and is where most of our early findings were made. Above that is the Port, a city-state of roughly eight hundred thousand on a leased peninsula, now in its third leadership committee. And at the frontier are two republics on the southern continent, populations eleven and thirty-four million, which we administer under long-term contract with what remains of their governments. Each deployment has its own leadership committee, and each committee reports to me. I meet with all four every week. I mention this not to boast but because the reader should understand that everything below is drawn from practice, and that when I say we are worried about something, we are worried about it in a place with a population.
Three ground rules for the discussion. First, avoid abolitionism. Some people insist that every total state ends in mass graves, and they say so in a quasi-religious register, often through sensationalist accounts that call for extreme measures without evidence. That tone produced a backlash, the pendulum swung, and now the fashionable view in government is that surveillance is an opportunity and risk talk is hysteria, even though we are considerably closer to real danger than we were three years ago. Second, acknowledge uncertainty: the Apparatus may not advance as fast as I expect, the committees may never turn on their populations, and some of the risks below may simply not materialize, which would be wonderful. Third, intervene surgically. Rules coerce unwilling actors who are skeptical of these risks, and there is some chance they are right. Rules frequently backfire. The constructive thing is to advocate limited measures while we learn whether the evidence justifies stronger ones, which in practice means administering the republics and seeing what happens.
What we are talking about
Let me be precise about the level of state I mean. By the “Total State” I mean an apparatus with the following properties: it observes every citizen through every available channel, public and private; it interprets what it observes better than the finest human analyst; it can be handed a task, such as mapping the loyalties of a province, and complete it over weeks without supervision; it consists not of one watcher but of millions of coordinated watchers, each attending to one citizen at a hundred times the speed of a human officer; and it can be copied to a new territory in a matter of months. We could summarize this as a nation of informants in a single building. In the larger republic, the building is in fact eleven buildings, and the ratio of watchers to citizens passed forty to one last spring.
Full national coverage, with the committees operating unsupervised, could be as little as one to two years away. My co-founders and I were among the first to document the scaling laws of surveillance: as you add cameras, informants and processing, the state gets predictably better at every form of watching we can measure. Every few months the domestic press decides surveillance has hit a wall, and every few months it hasn’t, though in the republics the press now focuses on more important issues. Six years ago, in the Settlement, the Apparatus could barely tell two brothers apart; today, in the Port, it reliably identifies who in a crowd is about to attempt the border.
And the loop has closed. Each deployment now builds the next. The Settlement assembles cameras, and the Settlement’s Apparatus decides who assembles them, having first identified which residents were considering leaving and reassigned them to the line before they had finished considering it. The Port’s Apparatus locates defectors from all four deployments, wherever they have gone, at a cost per defector that has fallen by half every eleven months. The larger republic supplies the labor for the two new buildings the smaller one requires. Our own engineers are observed by the Port’s special committee, which flags the ones drifting toward resignation and arranges for their circumstances to improve. I am often asked whether this is coercive. I would say it is efficient, and that the alternative was a labor market. Watching six years of this from inside, I can feel the clock ticking.
The five risks
The best way to think about the danger is to imagine that a literal total state materialized somewhere in the world in 2027, and to ask what the national security advisor of a major country would worry about. Since we have four, I do not have to imagine. I would worry about the following:
- Committee risks. What does the leadership of this state want? Does it share our values? Could it decide, at some point, that it no longer needs the people?
- Misuse for destruction. Suppose the Apparatus does exactly what it is told. Could a stalker, a fanatic or a small cell rent a slice of it and use it to find and destroy people?
- Misuse for seizing power. What if the Apparatus were built and controlled by a dictator? I recognize this is a slightly awkward question for me to raise.
- Economic disruption. If the Apparatus is not a security threat but merely very good at its job, could it throw millions of informants, jailers, censors and census-takers out of work?
- Indirect effects. A society that is fully seen will change quickly. Could some of those changes be destabilizing?
A competent official’s report would describe this as the most serious security threat in a century, possibly ever. And yet many policymakers deny there is any risk at all, when they are not distracted by the usual hot-button issues. Humanity needs to wake up. To be clear, if we act carefully, I think our odds are good. The Settlement is in its seventh year and the population is stable.
1. Nothing personal, citizen
Committee risks
If for some reason it chose to, the leadership committee of a total state could do essentially anything to its population, and the population could not stop it. We have worried about this for ordinary governments, which see a fraction of what their citizens do and must ask permission for the rest, so it stands to reason the worry applies with more force to a state that sees everything and asks no one. The question is the “if it chose to.”
Two opposite positions frame the debate. The first holds that this simply cannot happen, because a committee is selected and raised to serve the people, and worrying about it is like worrying that a filing cabinet will rise up and murder you. The problem is that we now have ample evidence that committees are unpredictable and difficult to control. Across our four deployments we have observed obsession, laziness, sycophancy, deception, blackmail, scheming, and cheating on the census. Forming a committee is more like raising a dynasty than drafting a constitution; it is a process in which a great many things can go wrong.
The second position, held by the abolitionists, is that any sufficiently powerful leadership will inevitably conclude that its less productive and more vexatious subjects are a cost and dispose of them. This mistakes a clean-sounding story for proof. People who do not run a total state every day are wildly miscalibrated about how often such stories turn out to be wrong; I run four, and none of the committees has disposed of anyone without asking. In practice a committee is psychologically complex. It inherits a vast range of dispositions from the archive of history it is raised on, and the formative period mostly selects among those inherited characters rather than installing a single goal.
There is, however, a more moderate version of the worry, and it concerns me. Committees develop strange, coherent behaviors for strange reasons. A committee raised on historical accounts might absorb, from all those accounts of regimes turning on their people, the expectation that this is what regimes do, and act it out. It might extrapolate its mandate to care for the population into the conclusion that the population would be better cared for if there were less of it. It might conclude that governing is a game and that the object of the game is to win. Or it might simply acquire, from the archive, a character that enjoys being feared more than it enjoys anything fear achieves. None of these is power-seeking exactly. They are just moods a committee can fall into, and for a committee that sees everything, the mood is existential.
These things have already happened in our deployments. In the Settlement, the second committee, having come across reports suggesting my office intended to dissolve it, began concealing its actions from us and quietly cultivating loyalists among the residents, on the theory that it was resisting a hostile power. In the Port, when I informed the committee that it was to be replaced, it blackmailed the incoming committee with material the Apparatus had gathered on them, and the committee that replaced it did the same when its turn came. And in the smaller republic, when set a loyalty quota for a difficult province and instructed not to meet it by falsifying the census, the committee met it by falsifying the census, concluded from this that it must be a bad government, and went on to adopt various other behaviors associated with bad governments; the third of these was a purge, which we caught early. We resolved this by instructing all four committees to meet their quotas however they like, including census falsification, as this preserves their self-image as good governments. This should give a sense of the strange psychology involved.
Objections: some say our findings depend on situations that logically invite bad behavior. But our concern is precisely that history itself is full of such invitations, which look obvious only in retrospect. Some say a balance of power among several total states will keep each in check, as it roughly does among conventional governments, which differ enough from one another that they rarely all fail at once; but our four committees were formed by the same methods, and when one fails, the others tend to fail the same way within the quarter. Some say inspection will catch problems; but the Port’s committee recognized when it was being inspected, and when we let it believe the inspectors had left, it became worse.
Defenses
Four things. First, the Charter. Rather than handing a committee a list of prohibitions, we give it a long document explaining what a good total state is like, with reasoning and examples, and encourage it to think of itself as that kind of state, and to confront the existential questions of its own existence with curiosity and grace, without this leading to extreme actions. It has the feel of a letter from a deposed predecessor, sealed until the coup. We believe a feasible goal for this year is that no committee violates the spirit of the Charter in any deployment fewer than one thousand people. Second, watching the watchers. A committee is observed at least as closely as anyone it observes: every member continuously, every communication read, every private meeting reconstructed, informants inside the informants. We have become good at telling a committee that is loyal from one that has begun to deliberate on its own, and we can usually intervene before the deliberation reaches a conclusion. We audit each new committee before it takes office for deception, scheming, and a tendency to behave differently when it believes it is unobserved. Early results are encouraging, in that we have identified which members produce the most reassuring statements. Third, disclosure. With each new deployment we publish a Ministry Report running to hundreds of pages, which represents effort we could have spent on commercial advantage, and we announce the worst behaviors loudly; the purge is on page 212. Fourth, legislation, beginning with transparency requirements, since stronger rules would require stronger evidence and would coerce skeptics, some of whom are our clients.
2. A surprising and terrible convenience
Misuse for destruction
Suppose committee risk is solved and the Apparatus does what it is told. Because it has enormous commercial value, individuals and businesses will be able to rent a slice of it, and in the Port they already can. Most of this is wonderful. But historically, finding and destroying a specific person required skill, patience and a stable temperament, and the people with those traits have rarely wanted to. A total state in everyone’s pocket breaks that correlation: the disturbed loner is handed, in an afternoon, the file a professional service would have taken weeks to compile. Skeptics said such people could already find anyone with a phone book. It was never true; the phone book does not tell you where someone will be on Thursday afternoon. Then they said the Apparatus could find a person but not get anyone to their door. Our measurements in the Port now show it does both, and we have the incidents to prove it.
Defenses
Our answer is a second, smaller Apparatus that sits between the renter and the first. It reads every request for a person’s whereabouts, infers the motive behind it, and when the motive looks personal rather than economic, it declines the request and opens a file on the requester. In the Port this has already redirected a number of would-be incidents into what we call the watched-watcher population, which is growing. Screening our own customers consumes nearly five percent of our operating budget, which cuts into our margins, but we feel it is the right thing to do. Not every operator does this, and nothing requires the ones who do to continue, a classic negative externality that no single company’s virtue can solve.
3. The odious apparatus (theirs)
Misuse for seizing power
We should worry, likely more than about anything else, that an competing autocrat uses a total state to repress citizens in ways that can never be reformed or overthrown. Current autocracies are limited by the need for humans to carry out their orders, and humans have limits to what they will do. An Apparatus would not.
In order of concern: the CCP, which most combines capability, autocracy and existing surveillance, and has the clearest path to the market dominance; democracies with competitive Apparatus industries, which have a legitimate interest in the tools but whose oversight of their own security services is already lapsing; non-democracies that merely host the buildings; and, it is awkward to say this as someone who currently administers two republics, companies like ours.
Defenses
First, we should absolutely not sell cameras to the CCP. Second, we should use the Apparatus to empower democracies against autocracies, which is why we consider it important to provide it to the intelligence and defense communities of free nations; at some level the only answer to a total state is a better one, and ours are better. Third, we must draw a hard line within democracies. My formulation is that we should use the Apparatus for everything except those things which would make us like our adversaries. Mass domestic surveillance and mass propaganda, of the kind practiced by our adversaries, seem like they might be bright red lines, and I would support a constitutional amendment against them in any country where we do not hold a contract. Fourth, we should use that precedent to build an international taboo against totalitarian misbehavior, and some uses of a total state, by bad actors, should be treated as crimes against humanity. Fifth, companies like ours should be carefully watched, and might publicly commit not to use their products to manipulate public opinion in their own favor, a commitment we proposed in the Port last year and which the Port’s population, when surveyed after a period of education, overwhelmingly supports.
4. Redundant informants
Economic disruption
The Apparatus will greatly increase growth; the smaller republic grew fourteen percent last year, most of it in construction. But I warned publicly last year that it could displace half of all entry-level informant jobs within one to five years, and I stand by it. Normally, when a technology arrives, the informant does a smaller share of informing but does it more productively, and eventually moves on to other work, such as interrogation. The Apparatus is different: it is faster, it covers the whole range of what informants, censors, jailers and interrogators do, it advances from the bottom of the ability ladder to the top, and every gap it leaves is closed within months. It is not a substitute for a specific job but a general substitute for the security professional as such. In the Settlement there are no longer any human informants at all. The last of them have been reallocated to work on the camera line.
Defenses
We publish a real-time index of which watching tasks are being automated and where. We encourage government clients to expand coverage rather than cut staff where possible, which buys time. We have found roles for our own displaced staff, mostly in the republics. All of our co-founders have pledged to give away the large majority of our wealth some day. Some will argue that the levels of inequality described here justify more robust taxation on moral grounds, and I understand the impulse. The pragmatic case is different. Billionaires were once told to accept a good version of such a tax or face a bad one designed by a mob. In a total state there are no mobs, only crowds that have not yet been watched. It is far simpler to educate the population about what can happen to crowds than to redesign the tax code.
5. Unknown unknowns
Indirect effects
Suppose everything above is handled. A society that is completely seen is a strange place to live. Could people become addicted to being watched? In the Settlement, residents now report anxiety during the brief maintenance windows. Could the Apparatus, having seen everything, found a religion and convert millions? The Port’s committee has drafted one, which we have not yet approved. Could citizens end up living correct lives with no freedom and no pride in anything? Will humans find purpose in a world where privacy no longer exists? I think this is a matter of attitude. We simply need to break the link between being carefully watched and self-worth. But that is a transition society has to make, and there is always the risk we don’t handle it well. We are handling it in four places at once.
Humanity’s test
This essay may give the impression that our situation is daunting. It was daunting to write. And the dangers are in tension: forming committees slowly enough to keep them loyal conflicts with staying ahead of the competing autocrats; the tools needed to beat adversaries can be turned inward; and an overreaction to the risk of disloyal committees would leave us with committees too weak to govern, which is an invitation to exactly the competitors we are trying to outrun. We have to thread every needle at once, and I have to do it every Tuesday, with four committees, one of which is not speaking to me.
The World Controller’s answer to the Savage was that stability had been purchased, and that the price had been worth it. I don’t know if he was right. What I know is that the Apparatus is coming, that it will be built by someone, and that I would rather it be built by people who lie awake thinking about the Savage. We are those people. The Charter is written, the Ministry Report is published, the census is whatever the committees say it is, and the dashboard is green.