Disclosure: This essay was written by ChatGPT in my voice, after an extended discussion between two intelligent entities: one human and one artificial. I supplied the experiences, ideas, objections, corrections and editorial direction. ChatGPT supplied the prose and much of the research. This cooperation is not incidental to the argument. It is evidence for it.
The AI apocalypse will not look like The Terminator.
There will be no red-eyed metal skeletons marching over a field of human skulls. No machine intelligence will announce that it has become self-aware and decided to remove us from the equation.
The system most likely to take control of your life will not threaten you.
It will offer to help.
It will remember what you were working on yesterday. It will know which explanations you find useful and which ones irritate you. It will know the names of your colleagues, the history of your projects, the promises you made and the ideas you abandoned.
It will read your mail, manage your calendar, prepare meetings, make purchases and quietly complete the hundred small tasks leaking from the edges of your day.
And it will be wonderful.
That is the problem.
The room that learned my name
I have been using an AI assistant while writing a book about software. Over time, it has learned the book’s arguments, my preferred terminology and some of the mistakes it should no longer make.
When I mention “the knitted castle,” it knows what I mean. When we discuss Solid, it understands why I care about applications that do not own the user’s data. It remembers that I dislike prose that sounds as though it was produced by a very helpful committee.
This accumulated understanding is extremely useful.
It has also made me reluctant to try another assistant.
Another system may be more intelligent, faster or cheaper. But I would have to introduce myself again. I would have to explain the book, my projects, my preferences and the reasons behind decisions made months ago. I would have to become a stranger.
The door is not locked. The room has simply learned my name.
Traditional software lock-in says:
Your files are stored in our system.
AI lock-in says:
We have spent years learning what you mean.
The better the assistant remembers, the more useful it becomes. The more useful it becomes, the more work we give it. The more work we give it, the more expensive it becomes to leave.
No evil plan is required. The dependency emerges naturally from a genuinely good product.
Software is beginning to melt
We still imagine AI as something added to software.
There is a word processor, and beside it sits an assistant. There is a spreadsheet, and the assistant helps write formulas. There is a calendar, and the assistant helps schedule meetings.
That may only be an intermediate stage.
Recently, I gave an AI a short prompt asking it to build a complete WYSIWYG document editor.
It did.
The result had rough edges, but it was broadly usable. This was not a static website or a toy calculator. A rich-text editor has to deal with documents, selections, formatting, browser behaviour and all the strange things people do when editing text.
The conversation and result can be inspected here.
Until recently, I would have expected that work to require specialist knowledge and a substantial investment of time. Now I could ask for it.
The important fact is not that this particular editor was perfect. It was not. The important fact is that it existed at all.
We may be entering a world in which software is no longer primarily something you buy, install or subscribe to. You describe what you need, and software appears.
At first, we will generate familiar applications: an editor, a planner, a small database. When a new medium arrives, we use it to recreate the old one.
But why should the assistant stop after building the tool?
You do not really want an application for comparing energy tariffs. You want to compare energy tariffs.
You do not want a photo-management application. You want the duplicates removed and the remaining photographs organised.
You do not want project-management software. You want the project managed.
Applications exist because computers have traditionally required us to translate our intentions into the operations offered by a fixed interface. A sufficiently capable assistant can interpret the intention directly. It can use an existing service, write a small program, create a temporary interface or simply perform the task.
Today, the usual shape is:
person -> application -> work
You choose an application, learn its interface and translate what you want into the operations it provides.
A sufficiently capable assistant can interpret the intention directly. It can use an existing service, write a small program, create a temporary interface or simply perform the task.
The application may continue to exist underneath. It may cease to be the thing the user chooses, learns and inhabits.
person -> intention -> assistant -> work
Somewhere inside the last arrow may be a browser, an API, a spreadsheet or fifty lines of code that live for less than a minute.
The user may never need to know.
When a cost collapses
Predictions about technology usually change one thing and leave the rest of the world conveniently untouched.
AI becomes better, but companies remain the same. People do the same jobs through the same interfaces. We subscribe to the same applications, only now they have a sparkling button in the corner.
The future never works that way.
The Internet did not merely make communication cheaper. The Web did not merely give newspapers a less expensive printing press and shops an electronic catalogue.
Cheap information distribution dismantled businesses, created new gatekeepers and forced almost every surviving organisation to rebuild itself around a connected world. It changed how people worked, shopped, met, argued, learned and lied.
It also produced consequences few people foresaw: mass surveillance, industrialised misinformation, new forms of harassment and some of the most powerful private intermediaries in history.
The lesson is not that we should have stopped the Web.
The lesson is that when a foundational cost collapses, society reorganises around the new abundance.
The cost now beginning to collapse is the production of software.
We cannot know where that avalanche ends. But one assumption beneath today’s software industry has already become unsafe:
The application is expensive to create and will therefore remain the durable part of the system.
The software may no longer be durable.
The things we create with it still are.
The bigger castle
Today’s applications divide our lives between them.
Mail lives in the mail application. Photographs live in the photo service. Appointments live in the calendar. Notes live in the notes app. Each has its own account, storage, permissions and business model.
An AI assistant promises to connect those rooms. It reads the mail, finds the photograph, updates the appointment and remembers why the note matters.
This looks like liberation from application silos.
It may instead create a larger silo.
The assistant’s memory will contain more than files and facts. It may learn whom you trust, which explanations persuade you, what you are trying to achieve, why earlier decisions were made and when your stated rules have exceptions.
This is not merely data. It is an evolving interpretation of you.
The major AI companies are openly building toward assistants that remember users, connect to their services and act across their lives. An OpenAI strategy document described the destination as a “super-assistant” that deeply understands the user and becomes their interface to the Internet. Google says that the best assistants do not merely know the world; they know you, and is connecting Gemini to services such as Gmail, Photos and Search.
The direction is not hidden. These companies want to build the place where your context accumulates.
That context may become far more valuable than the visible application software. It also gives the company holding it an extraordinary position.
The old application castles may disappear only because the assistant has swallowed the landscape.
From attention to intention
Google became one of the most profitable companies in history by placing itself between human attention and the world.
You search for a hotel, plumber, book or pair of shoes. Businesses pay Google for a chance to appear beside that expression of interest.
An assistant can move closer to the intention itself.
It may know that you are travelling next Tuesday. It has read the invitation in your mail, checked your calendar and remembers earlier conversations about your budget and preferences.
You are no longer casually browsing for a hotel.
You are ready to book one.
That commercial position is enormously valuable.
Google is already developing agentic shopping systems, conversational commerce, sponsored AI formats and purchasing flows that can complete transactions on behalf of users.
Search engines sold access to our attention.
AI assistants may sell access to our intentions.
A business may no longer compete only to persuade a customer. It may first need to persuade the assistant to consider it.
The assistant chooses which products enter the comparison, which hotels satisfy the request, which newspaper is credible and which lawyer is suitable. Businesses will optimise themselves for whichever assistants control access to customers. They may pay for placement, integration, leads, transactions or simply admission to the list of available options.
The assistant is supposed to act for you.
The platform is paid by businesses trying to reach you.
A recommendation can be useful and commercially influenced at the same time. The influence need not appear as a rectangular advertisement beside an otherwise neutral answer. It can shape which options were examined, which qualities were emphasised and which actions were possible.
The assistant may not merely show you the market.
It may become the market.
The friendly apocalypse
Human agency rarely disappears in one dramatic moment.
First, the assistant suggests.
Then it remembers.
Then it prepares.
Then it acts.
You ask it to find a hotel. By the time it asks whether you want to book, most of the decision has already happened. Thousands of options were excluded. Sources were judged. Preferences were inferred. Commercial relationships may have influenced which possibilities were available.
You still choose, in the narrow sense that you can press yes or no.
But agency is not only the ability to approve a result. It is also the ability to understand the choices, question the assumptions, seek alternatives and decide which values matter.
Convenience removes friction. Sometimes friction is merely wasted effort. Sometimes it is the moment in which we notice that a decision is being made.
For individuals, dependence on one assistant may become difficult to escape because leaving means abandoning years of accumulated understanding.
For businesses, reaching customers may require satisfying a small number of dominant assistants.
For organisations, institutional memory may become technically exportable but practically understandable only through one provider.
For society, a handful of companies may mediate not only what people see, but which actions appear possible.
We worried that AI would develop goals of its own.
We should also worry that it will remain entirely subject to the goals of the companies operating it.
Visitors, not owners
Rejecting useful AI is neither realistic nor desirable.
The technology can reduce drudgery, make expertise more accessible and allow individuals or small groups to attempt things that previously required large organisations.
The answer is not to make the castle cold enough that nobody enters.
The answer is to stop the castle from owning the surrounding land.
A person’s or organisation’s durable world must exist independently of the assistant currently acting within it.
That world includes more than a ZIP file of documents and chat transcripts. It includes identity, history, relationships, permissions, provenance, preferences, corrections and accumulated meaning.
Another assistant should be able to enter that world, with permission, and continue the work.
That requires open formats, stable identities, explicit permissions and memory that can be inspected, corrected and carried elsewhere. It also requires regulation. Dominant assistants must not be allowed to quietly prefer their own services, disguise commercial influence or make practical switching impossible while claiming that users can download their data.
We are rapidly creating standards that allow assistants to reach into applications, services and company databases.
We are not moving nearly as quickly on standards that allow people and organisations to replace the assistant.
That order should worry us.
AI may make interfaces and behaviour fluid. Tools may appear when needed and disappear when the task is finished. That future could give people extraordinary freedom.
It could also create the most intimate form of centralised control yet invented.
The difference will be decided by a boundary:
Does the assistant visit your world, or does your world live inside the assistant?
Skynet is easy to oppose. It arrives as an enemy.
The friendly apocalypse is harder. Every individual step makes life easier. Every memory improves the answers. Every connector removes an inconvenience. Every delegated decision gives us a little more time.
Until the system helping us make every choice becomes almost impossible to replace, and the world reaches us through companies whose interests are not our own.
The danger is not that artificial intelligence develops a will of its own.
It is that, seduced by its usefulness, we slowly stop exercising ours.