Accelerating the Dystopia, or Changing the Destination

The easiest way to misunderstand this essay is to look for one villain. Villains are narratively efficient. Systems are not. The harder possibility is that many people can do what their roles reward, offer sincere reasons for doing it, and together produce a result that none would choose for their own life.

An employee works longer because the work matters. A manager raises the target because a new tool makes more output possible. A firm reduces its costs because its competitors are doing the same. A supplier personalizes an offer because it improves conversion. A platform ranks the material that holds attention. A research team improves the system that helps all of them work faster. Each decision can make sense within its frame.

The frames do not necessarily add up to a good life. Local reasonableness is not a system-level warranty.

That is not a claim that individual responsibility disappears. It is a claim that individual intention does not provide a complete explanation. An account that stops at good people or bad people misses the arrangements that turn local choices into repeated outcomes. It also misses the people who inherit the outcome without taking part in the choice.

The central question is therefore not whether intelligence is good. It is what happens when intelligence becomes cheaper and more capable inside a network of unequal resources, incomplete measurements, conflicting objectives, and uneven power to revise the rules of an interaction.

My answer is the thesis I came into this project trying to articulate: if we have not repaired the systems already accelerating the divide, discord, and disruption of socioeconomic life before AGI or ASI arrives, aligning a much more powerful intelligence to those systems can accelerate exactly what already sucks about them. Better machinery does not renegotiate the destination by itself.

That is the best case I am worried about. The machine works. It listens. It helps. It becomes extraordinarily capable. Advertising becomes more adaptive. Digital profiles become more valuable. Pricing becomes more individualized. Institutional decisions become faster. Social platforms become better at holding attention. AI companions become more persuasive and more emotionally central. Employers discover new units of productivity. None of those outcomes requires a rogue model.

The preceding sections supply different parts of that network. The research establishes particular mechanisms and constraints. My experience supplies another kind of evidence: what these systems feel like from inside the organizations that build, buy, sell, and live by them. What follows is my synthesis of how those mechanisms can interact.

Improvement can be captured at the point of measurement

I will call this improvement capture: a process in which a gain in capability is recorded, rewarded, and reinvested mainly through the objective of the organization that deploys it, while benefits or costs outside that objective remain less visible. This is an analytical term for the mechanism, not a newly measured economic index.

The process has a simple beginning. An organization chooses a problem it can observe and fund. It defines a result, buys or builds a tool, and measures whether the tool improves that result. The gain enters a budget or performance report. If it looks valuable, the organization expands the tool’s use.

The dangerous part is what the measurement leaves outside its boundary. The spreadsheet can be right and the lived account can still be missing a column.

A customer-service system can reduce the firm’s handling time by moving more work to the customer. A scheduling system can improve coverage by making workers’ lives less predictable. A recommendation system can increase engagement by drawing more time from people who later wish they had spent it elsewhere. In these hypothetical cases, the organization does not need to report a false number. Its number can improve because part of the cost has moved beyond the place where it is counted.

Once the gain funds the next round of optimization, the measurement boundary acquires a feedback effect. What the institution can count helps determine what it can justify. What it can justify receives resources. What receives resources becomes easier to improve. The interests that remain outside the measurement may have no comparable mechanism for accumulating investment.

This is more specific than saying that powerful actors always win. They do not. A customer can switch, a worker can leave, a competitor can offer better terms, or a public record can expose a failure. Those responses are part of the system too. The question is whether the people bearing the omitted costs have enough information, time, alternatives, and authority for their responses to matter.

The asymmetries discussed earlier become coupled here. Money purchases analysis. Analysis makes opportunities visible. Visibility guides action. Successful action can generate more money. Meanwhile, a person dealing with several small institutional burdens may lose the time needed to understand or contest any one of them. The first loop accumulates capacity; the second can consume it. One side compounds. The other side gets tired.

AI can enter both loops. A person can use an assistant to understand a contract or challenge a charge. An organization can use an assistant to design the contract, estimate the person’s response, and process the challenge. Nominal access to the same model does not erase differences in data, transaction authority, staff, or the ability to run many trials.

This is why access is necessary to examine but insufficient as a final answer. The relevant unit is the effective arrangement: model capability plus resources, permissions, knowledge, alternatives, and a path to action. Equal access to a conversational interface is not the same thing as equal capacity to change the outcome.

At the same time, cheap intelligence may weaken some existing advantages. A small team can understand a technical problem that previously required a large staff. A person can identify a hidden assumption in a document. An outsider can make an alternative legible. The mechanism of improvement capture is therefore conditional. It is strongest where the new capability complements resources that are already concentrated and where the people outside the objective have weak ways to respond.

That condition can be investigated. We can ask who saves time, who acquires new obligations, who can act on the answer, and who receives the durable gain. Those questions move the discussion from the promise of capability to the terms of its use.

The unpriced queue is where separate systems meet

A person does not encounter institutions one at a time in separate laboratory conditions. The same day contains work, bills, care, paperwork, uncertainty, and ordinary life. Each institution may assess only the burden it imposes. The person carries the sum.

This creates a coupling that is easy to miss in a single-service evaluation. A ten-minute task can look modest in isolation. Ten such tasks from different organizations are not ten separate people with ten spare minutes. They are demands on the same limited day. Their timing can matter more than their average duration.

Administrative-burden research distinguishes learning, compliance, and psychological costs. The broader inference here is that those costs can interact across institutions. Time spent discovering one rule is time unavailable for contesting another. A confusing bill can arrive during a period of work pressure. An incomplete record can require action just when the person has least capacity to provide it. [S205]S205 · Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State InteractionsDonald Moynihan; Pamela Herd; Hope Harvey · 2015peer-reviewed public administration / JPART · administrative-friction frameworkClose

AI may reduce the time needed for each task. It may also increase the number of tasks that institutions can cheaply assign, personalize, or revisit. The net effect on the person cannot be inferred from the reduction in handling time per case. Volume and the distribution of work matter too.

Imagine a firm that can now generate tailored follow-up requests at very low cost. Each request may be relevant. Yet the customer’s cost of responding is not necessarily low. The machine’s ability to ask another question does not give the person another hour. Compute scales more gracefully than Tuesday afternoon. If the workflow treats nonresponse as evidence of disinterest or noncompliance, the burden can change the record that later justifies the decision.

This produces a second loop: burden reduces response capacity; reduced response changes the institutional record; the changed record supports decisions that create more burden. A person can then appear to be the source of the difficulty when the process helped generate the behavior it measures.

That loop is a proposed mechanism, not a claim that every institution operates this way. It is useful because it identifies what an audit confined to the completed form would miss. The missing cases are not random. Some may be missing because the process was too costly to complete.

Now connect this to ownership. An institution can preserve records, staff, and continuity beyond any one employee’s involvement. A household’s continuity may depend on one person remembering what happened and finding the time to act. A well-designed assistant could help preserve that history. A poorly designed one could replace it with a summary that drops exactly the unresolved details needed for an appeal.

The distinction between observation and interpretation becomes material at this point. If a record says the customer did not respond, that is one claim. If it says the customer accepted the decision, that is another. If a later system treats the first as proof of the second, a fact about a message has become a conclusion about consent.

A semantic correction can expose that mistake. It cannot create the time the person lost, restore a missed opportunity automatically, or decide the remedy. Technical clarity and practical recourse have to meet at the same point. Either one alone leaves a gap.

This is one reason the idea of friction deserves care. Some friction is waste. Some is the effort needed to understand a consequential choice. Some is a deliberate barrier. Some is a safeguard against acting too quickly. Reducing all of them under one efficiency target can remove the wrong thing.

A useful assistant should make a person’s chosen action easier without silently converting ease into authorization. That is a design principle about agency, not a claim that every slower process is better. The point is to distinguish the work people need help doing from the moments in which they need room to decide.

The unpriced queue is where the essay’s economic, informational, and psychological branches converge. People experience the coupled system as demands on attention, time, and room to act. An aggregate productivity gain can coexist with a day that feels more tightly administered.

The record can become the reason for repeating the past

The next coupling runs through evidence itself. Institutions act on records. Their actions shape what becomes recordable. Models trained or prompted on those records can then make the resulting pattern easier to reproduce.

A firm selects people according to a narrow measure. The selected people receive opportunities. Their performance becomes the firm’s evidence about who succeeds. An assistant learns the pattern and recommends similar selections. The new recommendations produce another generation of records with the same selection built in.

The causal problem is not that the records must be false. The observed people may really have performed well. The missing comparison concerns people who were not given the same opportunity. An accurate description of selected outcomes does not answer that counterfactual question.

Performative-prediction research studies how predictions can alter the population on which later predictions are made. The system-level extension here is that several institutions can act on related predictions and then treat the shared consequences as independent evidence. [S202]S202 · Performative PredictionJuan Perdomo; Tijana Zrnic; Celestine Mendler-Dünner; Moritz Hardt · 2020peer-reviewed ML conference / PMLR · feedback-loop mechanismClose

An employer, a lender, and a service provider need not coordinate explicitly for their decisions to interact. In a hypothetical case, unstable work reduces income predictability; income predictability affects access to credit; reduced financial room makes it harder to change jobs. Each institution can use a relevant variable. The combined result can still reduce the person’s ability to improve the variable.

AI could make each local prediction more accurate. Whether that helps the person depends on what the prediction is used to do. Identifying vulnerability can lead to assistance, exclusion, a higher price, or a different kind of offer. The information itself does not choose among those actions.

That is why the health-cost proxy example matters beyond health care. Its lesson is not that every unequal result comes from the same error. It is that accurately predicting a convenient label can diverge from the purpose that justified the system. When that label guides action, the divergence can become part of the next dataset. [S204]S204 · Dissecting racial bias in an algorithm used to manage the health of populationsZiad Obermeyer; Brian Powers; Christine Vogeli; Sendhil Mullainathan · 2019peer-reviewed / Science · proxy-target failure evidenceClose

Now add machine-generated explanations. The institution’s assistant describes the decision in polished language. A customer-facing assistant summarizes that explanation. A reviewer sees the summary. A later knowledge system treats the explanation as an authoritative account of the case. The original uncertainty becomes harder to find at every step.

This is not necessarily deception. It can be information loss through repeated compression. A qualification that was awkward to summarize may disappear. A source’s claim may become a statement of fact. An exception may be treated as an error because the dominant category is easier to process. By the end, the record looks cleaner than the underlying event. Administrative compression has done its little miracle again: less ambiguity on the page, not necessarily less ambiguity in reality.

The Rosetta branch enters here as my attempt to preserve selected distinctions: evidence versus inference, current permission versus past permission, competing senses rather than an early forced choice. Even a perfect record would leave the question of how it is used.

The latent-communication branch adds another layer. If information moves between agents in forms that are not directly readable, a final verbal account cannot be assumed to reveal everything that mattered. Causal message tests can investigate whether particular information affected the output. They do not automatically certify the entire organization of agents. [S168]S168 · Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLMHuixiang Zhang; Mahzabeen Emu · 2026research preprint · methodological controlClose

Together, these branches identify a feedback loop through legibility. What is easy for the system to represent becomes easier to use. What is used becomes more common in the record. What is common in the record becomes easier to represent. People and experiences that fit poorly can remain difficult to see, even as the system becomes more efficient at handling what it already recognizes.

The opposite loop is possible. Better tools can preserve an unusual case, expose a missing comparison, or make an overlooked experience understandable. The difference lies in whether the system is rewarded for maintaining contact with the difficult case or for making the case disappear into a convenient category.

That is a testable distinction in particular workflows. It is also a reason not to confuse a more coherent archive with a more complete account of human life.

A day in which all the tools work

Consider a day in which capable AI remains under authorized control and almost every tool does exactly what its owner wanted.

A professional begins the day with an assistant that has organized messages and prepared the documents needed for work. The person no longer spends an hour searching for context. The improvement is immediate and welcome.

The employer has noticed the same improvement. Work that used to require several handoffs can now be completed by one employee with assistance. The team is not eliminated, but new hiring slows and the scope of each role expands. The person finishes more work in the same day. The saved hour has become additional output rather than free time. The baseline ate the miracle before it reached the calendar.

Neither outcome is dictated by the model. The organization could have used the gain differently. In this scenario, the available performance measures make expanded output easier to recognize and reward than a less crowded day.

A newer colleague faces another consequence. Some routine assignments that once helped people learn the work are now completed automatically. The assistant can explain the finished result, but explanation is not identical to the experience of making and correcting the original attempt. The profession gains throughput while confronting a question about how future expertise will develop.

This is not a prediction that junior work vanishes everywhere. It is a possible interaction between task substitution and the production of skill. The institution solving this quarter’s staffing problem may not be the institution responsible for the profession’s long-term learning pipeline.

At lunch, the worker asks a personal assistant to investigate a household charge. The assistant finds the relevant terms quickly and drafts a clear request. That is another real gain. The person is better equipped than before.

The provider’s system is also better equipped. It can identify which options are authorized, which offers are likely to retain the account, and which cases require review. The two assistants exchange information efficiently. The outcome depends on the options available to each side, not only on the quality of their prose.

The personal assistant discovers an error in the account history. The provider’s assistant cannot change that class of record. It can explain the limitation and route the case, but the next queue requires a different process. Conversation is fast; correction is not. The interface can sprint while the remedy waits in another queue. The person now understands the problem more clearly without yet having the power to resolve it.

Later, a scheduling change at work complicates a family obligation. The employer’s system has improved coverage and reduced empty capacity. Those improvements are visible in its report. The family rearrangement is not. The worker’s assistant helps coordinate the change, reducing the immediate inconvenience. The assistance also makes the new scheduling demand easier for the employer to sustain.

This is an important ambiguity. A tool can genuinely reduce a burden while helping normalize the arrangement that creates it. The gain at the point of coping and the question about the source of the burden are both real.

In the evening, the person uses an assistant to understand a public issue. The system offers sourced facts, distinguishes uncertainty, and avoids telling the person what to choose. That is better than a stream of unsourced claims. Yet the available sources still reflect unequal capacities to produce research, commission reports, maintain records, and make arguments visible.

The assistant can expose some of those differences. It cannot manufacture missing evidence or know every omitted perspective. Its usefulness depends partly on whether it acknowledges the limits of the record rather than turning the best-documented account into the only possible account.

The person then reviews a recommendation about professional training. The recommendation is based on a model of what employers are seeking. Employers are, in turn, using similar tools to define their requirements. The worker adapts to the signal, and the adaptation becomes more evidence that the requirement is normal. A shared framework becomes more authoritative partly because people must respond to it.

At the end of the day, the person has received excellent assistance. Messages were organized, documents improved, a charge explained, a family schedule repaired, and a difficult topic made clearer. None of those benefits was imaginary.

The person has also worked at a higher baseline, spent time contesting a record, adapted to a schedule optimized elsewhere, and relied on institutional information systems to understand the institutions shaping the day. The tools reduced several local difficulties without necessarily increasing control over the whole arrangement.

This is the texture of the best-case dystopia. It need not look like technological collapse. It can arrive with cleaner copy, shorter queues, better summaries, and excellent quarterly guidance. It can be convenient, competent, and full of small improvements. Its defining feature is that people become better assisted in navigating conditions they have less practical ability to change.

The scenario has alternatives at every step. The employer could share the time gain. The provider could make correction easier. The assistant could preserve a broader range of options. The training system could create new routes to expertise. Different choices would produce a different day.

That is why the scenario is useful. It does not claim inevitability. It identifies where the direction is set. The model supplies capability; the surrounding arrangement decides how that capability becomes a lived outcome.

The pace of action can outrun the pace of correction

The scenario contains another asymmetry: institutions can often change an interaction faster than affected people can establish what the change did. AI can widen that gap when it lowers the cost of deploying a new procedure more quickly than it lowers the cost of evaluating its consequences.

This is not the same as saying that deliberation should always be slow. A slow response to a clear problem can prolong harm. The issue is whether the system has a usable feedback path from consequence to correction. Speed without such a path can turn a reversible experiment into an entrenched practice before its effects are understood.

Consider a hypothetical workplace that revises an automated performance measure every month. Workers adapt, managers change assignments, and the mix of tasks shifts. Six months later, an evaluation compares outcomes across the period. It may be difficult to separate the effect of the model from the effects of changing tasks, expectations, and who remained employed. The deployment produced both the intervention and changes in the population used to judge it.

The answer cannot be simply to collect more data. More records of the same selected process may preserve the same uncertainty at greater volume. A useful design needs comparisons that can distinguish plausible explanations, records of changes, and a way to identify who disappeared from the measured population.

Research acceleration adds to the challenge. Tools can improve the speed of implementation and testing, and first-party laboratory reports describe growing use of AI in those workflows. That does not mean the whole discovery cycle accelerates by the same factor, but it can shorten the interval between new capabilities and new deployments. [S175][S177][S178]S175 · RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human ExpertsHjalmar Wijk et al. · 2025ICML peer-reviewed benchmark · automated-R&D capability evidenceS177 · Research acceleration: The view inside OpenAIOpenAI · 2026first-party frontier-lab report · first-party automated-R&D evidenceS178 · When AI builds itselfAnthropic Institute · 2026first-party frontier-lab report · first-party recursive-improvement evidenceClose

Social consequences have their own clocks. A career-entry problem takes time to appear. A change in trust may not follow the schedule of a product release. A household’s accumulated burden is not visible in a single transaction. Some outcomes cannot be observed at the speed at which the system can be updated. The deployment clock and the consequence clock are not obligated to synchronize.

This gives early measures unusual power. If the first available evidence concerns cost reduction, that evidence can drive expansion before evidence about displaced burdens arrives. By the time the later evidence appears, workflows, contracts, habits, and investments may have formed around the earlier result.

Path dependence here does not mean that reversal becomes literally impossible. It means that the cost of changing course can grow with use. The more a system is embedded, the more a correction may require retraining people, moving records, rebuilding integrations, and renegotiating expectations. Those are real costs even when the original adoption was reasonable.

The same logic applies to standards of explanation. If an organization becomes accustomed to accepting a generated summary as sufficient evidence, restoring source-level review later may look like an expensive regression. The shortcut has become the baseline against which care is judged. A practice that began as a convenience can become an institutional definition of enough.

This is why the timing of a design choice matters. A requirement to preserve provenance is easier to meet before a million records have been flattened. A permission boundary is easier to maintain before a workflow depends on broad access. A portable representation is easier to use before meaning has become trapped in one vendor’s conventions.

These feedback loops make early design choices matter. A requirement to preserve a distinction is cheaper before the system has flattened a million records around its absence. The central issue is whether a system remains correctable as its usefulness grows.

Correctability is different from a promise that the system will never fail. It asks whether a mistake can be discovered, traced, contested, and repaired without making the person who suffered it bear an unreasonable share of the work. A system that becomes more capable while losing those properties can become harder to live with even as it becomes easier to use.

What changing the destination would look like

Acceleration has more than one destination. Changing it requires changing what the gains become.

Start with productivity. If an assistant saves an hour, I want to know who gets the hour. That is not a footnote to the productivity story. It is the story. Does it return to the worker as time, income, security, or room to choose? Or does it disappear into the next target because the new baseline simply assumes the tool?

Assistance should also increase the user’s capacity to act. A clearer explanation is valuable. A clearer explanation that arrives before commitment, preserves the evidence, and opens a real correction path changes the person’s position. An interface that explains a closed door beautifully is still a closed door.

Access matters when it reduces dependence on scarce complements rather than merely increasing their value. A tool that lets a small organization perform work that once required an incumbent’s infrastructure changes the arrangement. A tool that makes proprietary data and distribution even more valuable may harden it.

Measurement has to stay connected to the purpose. Fewer open support cases mean little if unresolved problems merely moved to the customer. More output is not a complete measure of job quality. More engagement is not proof of a better decision. If the claimed benefit is human, some part of the measurement has to reach the human outcome.

People who fit poorly into existing categories need a way to remain visible without having their account sanded down until it fits. Uncertainty and disagreement should survive when the evidence does not justify a single label. Better categories still cannot recover the people who never entered the record.

Consequential changes also need to remain traceable. A reviewer should be able to identify which model version, source, instruction, and authorization mattered. I do not need every neuron translated into prose. I need enough of the chain to test material claims about what happened and enough recourse to do something when the chain is wrong.

The useful question is no longer whether AI is “good” in the abstract. Which relationship improved? For whom? Who kept the gain? Who acquired a new obligation? Who can contest the result? Who can leave? Who can build an alternative? Those answers tell us whether intelligence expanded agency or merely made the existing arrangement run faster.

The gain I care about is a life with more room in it

I keep coming back to the corporate cafeteria because it captures the mechanism better than a cartoon villain ever could. The sandwich was not plotting against me. At Gartner, we had top-notch meals prepared by excellent chefs and sold roughly at the cost of the ingredients. The cafeteria ran long hours. It was a real benefit. It also made it easier to arrive earlier, stay later, and feel cared for while the institution absorbed more of the day.

The same thing happened with Symposium. The event was exciting. Building something useful was exciting. Being trusted with difficult work felt good. Before one launch, I logged more than 300 hours in a single September because I was convinced the grind was good for me. September, stubbornly, still had thirty days.

That is exactly why I reject the simplistic story in which exploitation requires miserable people who know they are being exploited. Institutions can offer real goods, real opportunity, real friendship, and real pride while still learning how to convert all of those things into more labor. The employee does not have to be stupid or coerced. Sometimes the most productive arrangement is the one in which talented people sincerely want to give more.

AI makes that distinction more urgent for me, not less. I want tools that make difficult work possible. I want to learn faster, build better, understand more, and spend less of my life on avoidable friction. The ambition is not to make human effort obsolete. It is to stop treating every reduction in necessary effort as an invitation to impose another necessity.

A machine can help me write a clearer argument. It cannot decide how much of my life belongs to the organization that benefits from the argument. It can help compare offers. It cannot create an alternative where none is available. It can preserve evidence. It cannot make an institution care about the evidence merely by storing it in a better format.

Those limits are not reasons to discard the tool. They are reasons to stop assigning the tool moral obligations that belong to the surrounding system. A model can create room. It cannot decide, by capability alone, who is allowed to keep it.

What remains is not a claim that technical progress is a fraud. It is a refusal to equate a more capable system with a better human arrangement. The equation needs evidence about the people whose lives are being changed.

A future can contain extraordinary science, cheap expertise, accessible creation, and tools that relieve real suffering. It can also contain concentrated ownership, more exacting work, individualized pressure, and institutions whose decisions become harder to contest. These possibilities are not mutually exclusive. Their coexistence is the point.

The question is which of them becomes the reinforcing structure. Do useful tools give people more capacity to choose, correct, and participate? Or do they mainly help institutions extract more performance from the same relationships while helping people cope with the result?

That is the destination hidden inside the discussion of speed.

The governing thesis does not need a rebellious machine. If the arrangements producing stratification, alienation, manipulation, and fragmentation remain in place, more capable AI can make those arrangements more effective. Alignment to the operator’s intended objective does not guarantee alignment to the lives affected by it.

Repair, in the sense used here, means changing those outcomes and relationships, not merely improving the interface through which people encounter them. The particular political and institutional choices remain for people to deliberate and decide. The evidence cannot make those choices for them, and neither can this essay.

But the dependency is real as a matter of logic: intelligence supplies means, and means operate within purposes and constraints. A system cannot be expected to correct an omitted value simply because it becomes better at optimizing the values it was given.

That is why I do not expect AI to save a system we refuse to examine and change. The machine may become an extraordinary instrument of repair. It may also become an extraordinary instrument of continuity. Intelligence magnifies a direction before it proves the direction was worth taking.

I want the saved hour to return to a person, not vanish into the next target. I want a clearer explanation to open a real option, not polish a closed door. I want the intelligence we build to enlarge the life available to the people living with it.

Those are judgments about what the work is for. No benchmark can make them on our behalf.

If we do not address the substrate in tandem with the acceleration, we are not waiting to discover whether AI creates a dystopia.

We are giving the one we already know how to build better tools, cleaner interfaces, faster feedback, and fewer excuses for pretending we cannot see where the road goes.

We are accelerating it.