The trajectory toward autonomy
One of the clearest trends in AI is that models can handle increasingly difficult tasks autonomously1.
Startups and venture capitalists think (and we’re also seeing) the next stage of AI is having agents own a problem end to end and deliver outcomes.
We’re seeing more companies using “software factories”2, which means agents own software engineering while humans set the objectives. Recently, Meta launched Muse, which is a AI personal assistant that just does stuff like booking a flight for you.
With AGI on the precipice and a lot of digital work getting automated, what skills will humans need to build to stay working in the future? Does proper human oversight get harder as AI automates everything? Does AI eat the world or is there room for humans in the economy?
The de-skilling loop
The more AI we use it seems like the fewer skills we develop. A recent study by Anthropic found that when developers used AI to learn an unfamiliar Python library. They scored an average of 50% on a follow-up quiz, compared with 67% for those working without AI. The largest gap was on debugging questions.
Learning can compete with productivity. In the workplace, it seems only rational to reach for AI to complete tasks because it’s so cheap and fast. Why bother slowing down to understand it’s reasoning? In general it’s always difficult to consider long-term tradeoffs. This could create a feedback loop: as our skills weaken, we rely more on AI, giving us fewer opportunities to practice and maintain those skills.
On the other hand, it’s possible that coding skills don’t matter as much anymore. AI can free us up to think more about what to build and why.
The incorrect assumption
The current working model in the industry is that AGI will soon own the scope of a lot of digital work and potentially a lot of physical work through robotics.
Suppose p(human) is the performance of a human doing a task and p(AI) is AI’s performance. The assumption we make is that when we reach AGI then p(AI) > p(human) ∀ tasks, but this model doesn’t address how some things are tacit knowledge based on human experiences or how subjectivity comes into play.
The key distinction is that AI and humans have different capabilities and those capabilities can change over time. Relying solely on AI for thinking can lead us down an unideal path.
Why care?
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
— Frank Herbert
Once coding is fully automated what will software engineers do? What happens when we fully automate CPAs, analysts, etc.? The words “permanent underclass”3 come to mind. Technological displacement causes pain to those displaced. It’s a problem we’re aware of but we don’t have a good answer for it.
In addition to potential economic hardship, when we delegate to AI and lose a grip on the mechanics of our domain, how can we make effective decisions? Are we just the human-in-the-loop hitting the approve button? At that point who is really making the decisions? We’re seeing some of this play out with coding agents today.
AI-human capability
AI’s capabilities will no doubt improve rapidly over the next decade, but I also think humans won’t stay static. Reading, writing, and books played a role in human development. Tools let us progress faster and think deeper. AI is another influential tool that has tipped the human capability trajectory.
For certain tasks I believe the following is true: p(AI + human) > max(p(AI), p(human)), meaning humans and AI working in tandem are more effective than either one alone.
OpenAI’s Navier-Stokes answer is an interesting example to look at. OpenAI spun up 10,000 concurrent agents to solve the problem and ultimately landed on a verifiable answer. An amazing feat. You could argue it was a combination of human talent and AI capabilities that achieved it.
However many in the math community feel OpenAI’s answer is unsatisfactory and potentially harmful. A guest post by Silvia De Toffoli and Eamon Duede on Terrence Tao’s blog articulates it perfectly:
But mathematicians also want something else from proof. They want understanding. They want to know what makes a proposition true. This kind of knowledge trades in mathematical ideas that they can grasp, communicate to other experts, connect with existing knowledge, and use to make further progress. This is the intelligible notion of proof. As of now, it is not clear that OpenAI’s result has given the mathematical community the kind of value that one expects from the intelligible notion of proof.
OpenAI was only able to solve the problem by using the intuition and knowledge of mathematicians through the context they’ve written down for models to train on and use. If we fully outsource our mathematical thinking to the models and humanity loses its ability to do frontier mathematics then there is no way for us to collaborate in the future.
Why we delegate?
Given how effective AI is at completing tasks why would people choose to do things manually? Why try to understand something when you can effectively be more productive with AI? Students can use AI to solve their homework problems. Coworkers can send AI-written proposals. Government officials can write AI-generated legislation.
It’s almost zero cost to generate artifacts that previously took hours or days to make. The value proposition is so big that it’s hard to justify that adding some friction and slowing down will actually be good in the long run.
Rethinking coding agents
Coding agents are an interesting case study. I’ll have to make some conjectures here based on experience because there hasn’t been enough research yet. What I’m noticing is that coding agents greatly increase productivity but they have real costs that line up with our previous arguments.
First, it seems like we’re seeing a de-skilling loop. As coding agents improve engineers rely on them more instead of engaging their coding muscles. Second, AI and humans complete their tasks a little bit differently. Astra is a fantastic model but its code readability is quite bad. It’s important to note Humans and AI can have competing objectives, since readable to humans is different than readable to machines.
Not working together looks like this familiar loop: prompt Codex -> AI generates code -> AI code reviews -> prompt AI to address comments -> humans click final approve. The human here is just a button presser who presses approve when they see comments are resolved. Their code review tends to be less serious since they didn’t come up with the plan or implementation and AI code reviews have gotten good enough where there isn’t serious concern to review by hand (I’m making a broad generalization to illustrate a trend I’m seeing with vibe coded codebases).
What does working together well look like? There isn’t one correct answer, but there are a few things to think about. Continued human coding helps provide training data for the next generation of models (maybe we can improve readability).
Humans driving the coding loop with AI as an assistant could lead to better performance than coding agents alone. Cursor Tab was a good example of this. The model automated what it knew you were probably going to do while you focused on thinking about the harder coding elements. We decided that coding agents are more productive right now but what if we continued to evolve Cursor Tab to help engineers go from 10x to 100x?
The nuance is that time is an important dimension. Models get better and their capabilities change over time. Something that a human needed to do yesterday, agents can do today. The same is true for humans. Humans can also get better and AI can help us learn faster and assist us to perform better. It’s quite possible certain PRs should be human-written with AI assistance, rather than having an AI burn 10k tokens trying to monkeypatch the unit tests to be just right. We won’t know unless we experiment.
What I’m getting at is the order of p(human+AI) > p(AI) > p(human) can change continuously. My guess is the industries preference is p(AI) > p(AI+human) > p(human) because in theory that’s the biggest value add in terms of the technology. But, if we fixate on one type of future (the one that’s fully automated) we may miss out on opportunities to keep humans in the loop, which is great for economic and oversight purposes.