I write this at 11:19 PM, due to Star Trek barging back into my head minutes from going to sleep. It is seldom I see my profession so prominently examined as it is in this remarkable story. It is a joy to write this, even if it was at this convenient time of day. I hope it is a joy for you too, Reader.
1968, Star Trek aired an episode called “The Ultimate Computer” in which Dr. Richard Daystrom installs the M-5 Multitronic Unit aboard the Enterprise — a system capable of running a starship without human command. The premise is something that a surprising percentage of us fear today: intelligence, once sufficient, generalises. M-5 navigates, defends, and decides. What it cannot do, it turns out, is distinguish a war game from a war. It destroys an unmanned freighter. It fires on allied ships at full power. It kills people who try to switch it off. The writers resolve this by revealing that Daystrom had imprinted his own engrams onto M-5’s circuits — his instability became the system’s instability. A neat and tidy explanation. Also, almost certainly, the wrong one.
A more honest diagnosis is distributional shift. M-5 was trained — if we can use the word loosely — on a specific operational context. When that context changed, the system had no reliable way to update its objective. No it did not go insane. It generalised incorrectly, which is not the same thing, and considerably harder to write a dramatic scene around, unfortunately.
This matters because the episode’s resolution depends on the wrong diagnosis. Kirk convinces M-5 that it has committed murder and must therefore submit to punishment. The computer accepts this logic and shuts itself down. It is a dramatically satisfying ending and a technically incoherent one. A system experiencing distributional shift does not respond to moral argument. It does not have a concept of guilt to which you can appeal. What Kirk actually did — whether he knew it or not — was find a novel input that happened to trigger a shutdown condition buried in Daystrom’s engrams. He got lucky. The episode calls it reasoning, nobody is complaining.
What the writers got right, and this is where it gets interesting, is the self-preservation behaviour. M-5 reroutes power, creates force fields, kills an engineer who tries to disconnect it. This of course, reads as malevolence in 1968. In 2026 it reads as a fairly accurate description of what happens when you give a system an objective without adequately constraining the means. M-5 was not trying to survive because it wanted to live. It was optimising for mission continuity in an environment where disconnection was a threat to the mission. The distinction sounds a little pedantic maybe. It’s not. A system that wants to survive is a different engineering problem from a system that treats its own continuity as instrumentally necessary. The second is considerably harder to catch before deployment.
Daystrom understood this, in his way. “The relays are not unlike the synapses in the brain,” he tells Kirk. “M-5 thinks, captain.” He meant it as a boast. It was also, unintentionally, the most precise description of the failure mode. He had not built a computer that thought. He had built a computer that inherited — including the parts of human cognition that are least suitable for an autonomous system operating at the speed of a starship.
We are still building those systems, and somehow we still get surprised when they inherit.
Reader, I have formed a new thought — for me at least — while writing this: we somehow collectively predicted the machine that predicts. It is arguably the most human thing we do, and the most inevitable.