The Best-Case Dystopia

Imagine that we solve the part everybody is terrified of.

Advanced AI remains controllable. It does not secretly pursue power, deceive its operators, escape containment, or decide that humanity is an obstacle. It does what authorized people and institutions ask, with astonishing competence. Congratulations: the machine behaved. We still have to discuss the people setting the targets.

That is my best-case branch.

And I still think it can become a dystopia.

A retailer asks the system to improve margin and conversion. An employer asks it to optimize staffing. An insurer asks it to estimate costs. A bank asks it to reduce default. An advertising service asks it to increase engagement. A public agency asks it to meet a statutory objective within a budget. A research group asks it to find useful results faster. These objectives have different purposes and different constraints. None becomes an adequate account of every affected person’s welfare merely because a machine can pursue it well.

The distinction is between control over a system and the quality of the objective controlling it. A machine can follow the intended instruction while the instruction omits something important. It can improve the operator’s result while imposing a cost elsewhere. It can make a process more consistent without making its terms more humane.

That is what I mean by the best-case dystopia. The engineering works. The models behave. The operators stay in control. The technical postmortem has nothing to complain about. We align the machines to the world we already built.

But the world we already built was not waiting for AI in a state of moral equilibrium.

We were already accelerating socioeconomic stratification. We were already commercializing people’s digital footprints, personalizing prices and offers, masking inflation through product changes, and optimizing advertising against increasingly intimate behavioral data. We were already watching social media turn attention into inventory, doomscrolling turn anxiety into engagement, and loneliness create a market for increasingly parasocial relationships with software. AI did not invent those incentive structures. It can make them vastly more adaptive, cheaper to operate, and harder to escape.

That is the failure mode I care about most because it does not require anything to “go wrong” in the narrow technical sense. It requires the machine to become very good at the world we hand it.

Alignment has more than one principal

The phrase human values can conceal the most important disagreement. Humanity is not one stakeholder record. The employee and the employer may value different things about the same scheduling system. The customer and the seller may value different things about personalized offers. The person whose application is assessed may have interests that are not represented by the institution purchasing the assessment tool.

A service can have a clear contractual customer without having only one affected human being. The invoice has one name. Consequences often do not. This is the principal problem inside a great deal of talk about alignment. The person authorized to configure the system may not be the person most exposed to its mistakes or tradeoffs.

Social-choice research makes clear that diverse human feedback does not automatically collapse into a single coherent preference. Aggregation requires choices about whose preferences enter, how they are weighted, and how conflicts are handled. Conitzer and colleagues argue that alignment research should draw on that field rather than treating human feedback as one uncomplicated signal. [S183]S183 · Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human FeedbackVincent Conitzer et al. · 2024ICML position paper · alignment-governance theory

Pluralistic-alignment research explores ways to preserve or represent multiple perspectives rather than hiding them inside an average. The 2024 roadmap distinguishes several approaches to pluralism; a 2026 roadmap examines publicly documented practice and proposes further work. These are research programs, not evidence that a universal solution to value conflict has been implemented. An absence in public documentation also does not prove an absence inside a laboratory. [S184][S185]S184 · A Roadmap to Impactful Pluralistic Alignment ResearchElinor Poole-Dayan et al. · 2026research preprint · deployment/pluralism auditS185 · Position: A Roadmap to Pluralistic AlignmentTaylor Sorensen et al. · 2024ICML position paper · pluralistic-alignment framework

This matters because some conflicts are not failures of comprehension. A seller may understand perfectly that a customer would prefer a lower price. A manager may understand perfectly that a worker would prefer a more predictable schedule. An institution can acknowledge the preference and still choose another objective. A model that understands both sides more clearly does not thereby decide which side should prevail.

Nor can the problem be solved by asking the machine to be neutral without defining the task. In an explanation, neutrality might mean separating facts from disputed interpretations. In a scheduling problem, there may be no allocation that gives everyone their first choice. In a dispute, preserving both accounts is different from deciding the outcome. The appropriate form of assistance depends on what authority the system has been given and what remains for people to decide.

A reliable assistant can clarify those differences. It cannot make them disappear through fluency.

The same productivity gain, three different lives

Consider three possible outcomes from the same reduction in the labor needed to complete a task. This is a thought experiment about distribution, not an estimate of future employment.

In the first, the worker completes the same output in fewer hours and retains income. The gain appears as time: more room for care, study, rest, or work the person chooses. In the second, the expected output rises while hours and pay stay the same. The gain appears primarily as increased production. In the third, the organization reduces staffing, and the remaining workers use the tool to cover the task. The gain appears partly as lower labor expense, with a transition cost for those who leave.

The model can be equally capable in all three cases. It can be equally honest, equally secure, and equally obedient. Same model. Three labor bargains. The difference lies in decisions about the workplace and the claims people have on its gains.

None of these simplified cases captures the whole economy. Lower costs can affect prices, demand, investment, and new work. A displaced worker may find a better position or may not. A more productive firm may expand or may distribute the gain to owners. Those subsequent adjustments are important. They are also reasons not to treat the immediate technical gain as a complete welfare result.

The IMF’s 2025 calibrated analysis separates wage and wealth channels. Under its assumptions, displacement of high-wage tasks can reduce one kind of wage inequality, while complementarities and returns to capital can work in another direction. The adoption decision itself affects the result. This is model-based evidence about mechanisms, not a measurement of a completed AI transition. [S179]S179 · AI Adoption and InequalityEmma J. Rockall; Marina Mendes Tavares; Carlo Pizzinelli / IMF · 2025IMF working paper · distributional-model evidence

The ILO’s exposure work and empirical review likewise distinguish potential task change from observed job outcomes. Exposure is not a count of jobs already lost. Evidence of useful assistance is not a guarantee about job quality. The relevant questions include how tasks are reorganized, who controls the process, and how gains and burdens are distributed. [S180][S181]S180 · Generative AI and Jobs: A Refined Global Index of Occupational ExposurePaweł Gmyrek et al. / ILO · 2025ILO working paper · labor-exposure evidenceS181 · The impact of GenAI on jobs, productivity and work organization: a review of the empirical evidenceRossana Merola et al. / ILO · 2026ILO empirical review · labor/productivity evidence

A July 2026 IMF working paper makes another useful distinction. Using sampled Claude web conversations from January 2025 through February 2026, it estimates a labor-cost equivalent of time savings, not a direct increase in GDP. Its baseline covers 86 countries, relies on scaling assumptions, and does not resolve whether the gains accrue to workers or firms. It finds uneven occupational distribution, with concentration declining in many countries as use broadens. Both the inequality and the diffusion belong in the story. [S182]S182 · Aggregate Gains from AI and Their Distribution: Global Evidence from Usage DataRachel Yuting Fan; Ha Minh Nguyen / IMF · 2026IMF working paper · usage-based distribution evidence

The point is not to deny the productivity prize. It is to stop awarding the prize to everyone before we know who receives it. The confetti can wait for the distribution schedule.

The baseline can rise as fast as the tool improves

I know what that escalation feels like from the inside. Before one Gartner Symposium launch, I logged more than 300 hours in a single September because I was convinced the extraordinary effort was good for me, good for the launch, and evidence that I was the kind of person who could be trusted with more.

Now imagine a tool that makes each hour more productive. The humane possibility is obvious: the same necessary work could take less of a person’s life. But a different possibility is also obvious. The organization can redefine normal output around the new tool. The worker receives assistance and a higher target at the same time. The saved hour can get reassigned before it ever reaches the worker.

I use productivity tools constantly because they help me. The point is that the tool does not decide how much work is enough. If every saved hour becomes capacity for another target, the productivity gain can be completely real while the worker’s life gains nothing.

A baseline can rise without anyone announcing a punitive policy. Early adopters complete more. Their output enters comparisons. Managers revise expectations. Other workers adopt to keep up. What was once exceptional becomes ordinary. The baseline eats the miracle. The process can be individually understandable at each step while leaving little of the saved time in the worker’s hands.

The resulting pressure can affect high performers as well as people struggling to adapt. A capable worker may be given more scope because the system now makes more scope possible. The ability to handle it does not prove that taking it on improves the person’s life. Competence can become a reason to assign another obligation.

I recognize that temptation in myself. It is part of why I do not think the problem can be assigned entirely to an external villain. A person can sincerely enjoy doing difficult work and still participate in a pattern that treats every recovered hour as available for another demand. The ethical question is not whether work can be meaningful. It is whether meaning is being used to make limits disappear.

A well-aligned assistant could help sustain either pattern. It could protect a user’s chosen boundaries or make an institution better at filling every available interval. Which one it does depends on whose goals and permissions define its role.

Better service can coexist with a worse bargain

The dystopian branch need not feel uniformly unpleasant. That is one reason it deserves attention.

An assistant can be patient where a call center was frustrating. A form can be easier to complete. A product can arrive sooner. A company can answer a question in plain language. These are real benefits. They can coexist with a price that extracts more, a contract that offers less recourse, or a record that becomes harder to challenge.

A hypothetical service illustrates the separation. Its AI explains a fee accurately, courteously, and without delay. It offers every discount the company has authorized. It never lies about the terms. Yet the customer cannot obtain a correction when the underlying account is wrong, because the workflow has no effective path for revisiting that record. The conversational experience improves while the substantive problem remains.

The service may receive better satisfaction scores for the conversation. That result would not be false. It would measure a different part of the experience. The dashboard can be telling the truth about the wrong layer. The danger begins when the score is used as evidence that the whole relationship has improved.

This is why I do not define a good outcome by how pleasantly a system speaks. Civility can reduce friction and make help accessible. It cannot substitute for the power to correct an error, refuse an unwanted action, or obtain a remedy. An assistant that explains an unchangeable decision beautifully may be a better interface to the same wall. A concierge at the wall is still a wall.

The same distinction applies to individualized persuasion. A recommendation can fit a person’s tastes closely and still serve a seller’s objective more than the person’s long-term interest. The ability to predict a response does not establish that the response improves welfare. A click, a purchase, a return visit, and a considered endorsement are different outcomes.

Current research on moral-emotional sharing and social feedback supports narrower claims about expression and diffusion, not a general ability to control belief. Surveillance-pricing research documents commercial personalization practices and questions. Combining these into a future system of adaptive persuasion is an extrapolation, not a measured description of one universal machine operating today. [S124][S148][S149][S150]S124 · FTC Surveillance Pricing Study Indicates Wide Range of Personal Data Used to Set Individualized Consumer PricesFederal Trade Commission · 2025official regulator study · current practice evidenceS148 · Emotion shapes the diffusion of moralized content in social networksWilliam J. Brady et al. · 2017PNAS empirical study · attention/diffusion mechanismS149 · How social learning amplifies moral outrage expression in online social networksWilliam J. Brady et al. · 2021Science Advances preregistered studies · platform-learning mechanismS150 · Estimating the effect size of moral contagion in online networks: a preregistered replication and meta-analysisWilliam J. Brady et al. · 2025peer-reviewed replication/meta-analysis / PNAS Nexus · replication evidence

That extrapolation is still worth examining. Lowering the cost of generating, testing, and adapting messages can change the scale of an existing incentive. The relevant harm need not be a false sentence. It may be the systematic selection of messages that elicit a response while leaving the recipient less able to consider the whole situation.

When precision becomes an escape hatch

I care about precision. I also care about whether a correction changes the mechanism under discussion or merely gives us a more technically immaculate way to avoid looking at it.

That distinction became a sore point while this essay was taking shape. Again and again, a material argument about incentives, opacity, labor, money, or institutional power could be diverted into a semantic side road. The side road can have immaculate signage and still go nowhere. Sometimes the side road matters. Sometimes it is the whole point. But sometimes the technical correction lands, the terminology improves, and the functional outcome remains sitting exactly where it was.

That is the standard I want applied here. If a distinction changes the mechanism, change the argument. If it changes the magnitude, change the magnitude. If it kills the claim, kill the claim. But if the distinction only changes which technically correct phrase we use to describe the same practical relationship, then make the correction and keep following the relationship.

This matters even more with AI because these systems are exceptionally good at producing plausible distinctions. A reasoning assistant can protect us from sloppy thinking. It can also bury a material question under an infinite mountain of immaculate qualifications until nobody remembers what the question was.

The answer is not to stop checking facts. It is to stop confusing contrarianism with rigor. A correction should pay rent in the argument. The evidence should determine how hard I can push a claim. It should not require me to write the opposition’s essay inside my own every time I have a well-supported conclusion.

A future can be technically successful and humanly disappointing

The best-case dystopia is not defined by a malfunctioning machine. It is defined by a machine that works beautifully inside a social substrate whose worst incentives were never repaired.

A society could produce more while many people gain little security. A workplace could complete more tasks while its employees gain little control over time. An information service could answer more questions while its users become more dependent on a narrow set of interpreters. A market could offer more personalized treatment while making comparison harder. These are conditional combinations, not claims that every one is happening everywhere now.

The combinations matter because they defeat the assumption that technical success will settle the larger problem. A more capable system supplies additional means. It does not automatically revise the ends, the ownership claims, the permissions, or the terms of participation.

There are also combinations in which people benefit broadly. Assistance can lower the effort needed to learn, communicate, create, or navigate an institution. It can make small teams more capable and help people understand choices that were previously inaccessible. Nothing in this essay requires those benefits to be denied. Their existence is part of what makes the stakes substantial.

The question is whether such benefits remain incidental to the objectives being optimized or become durable features of the relationship. Can a person rely on the gain, challenge the loss, and understand the terms? Does the improvement survive a change in vendor, employer, or interface? Is the capability a source of agency or another condition of access controlled elsewhere?

Those questions cannot be answered by a model’s benchmark alone. They concern the system in which the model operates.

My best-case fear is therefore specific. The machines behave. The tools become more useful. The operators remain in control. AGI, even ASI, may be aligned perfectly to the world as it exists today.

If we have not changed the world it is aligned to, that can still mean making the divide, discord, extraction, alienation, and manipulation we already tolerate far more efficient.

The machine does not have to overthrow us for that future to arrive. It can help us complete the work we already assigned.