I don’t know about you, but for me, there’s just something about a gloomy Saturday morning that invites deep thoughts into the mind. Raindrops on my apartment window are like tiny mirrors telling me to look inward. I guess that’s why this past weekend, greeted by grey skies, I fired up ChatGPT and asked somewhat fancifully,
“Why should I learn to add when a calculator can do it for me?”
The answer it gave yanked me swiftly out of whimsy and into panic.
Normally, an LLM will respond in an encouraging, if not borderline patronizing tone—I imagine layers of prompts and guardrails are in place to prevent insulting, offensive or even just objectively negative messages reaching the user. The answer ChatGPT gave could hardly be considered insulting or offensive, but it was uncharacteristically blunt and unvarnished: “Maybe you shouldn’t.”
Now to its credit, the chatbot did indeed distinguish between rote knowledge of addition and “possessing a model” but “even that distinction is becoming less important”, according to this particular session’s inference. And that’s precisely the part where I went from shocked to cranky. Not only is this AI insinuating that machines may have better, more robust models of the world than we do, but that eventually, humans should cease cultivating their own minds!
[In Morgan Freeman’s voice] I wish I could tell you this idea is farfetched. I wish I could tell you that.
As comforting and encouraging as it would be to believe this notion—that humanity should altogether stop trying to increase their own personal knowledge—was just glitchy, crackpot advice from a silly word predictor, it turns out that lots of people buy into this concept. Programmers say coding is solved. Academics think math is dead. Even the Pope has raised alarms about this so-called ‘paradise of machines’. You can’t ride the subway or open Reddit without being reminded that an AI agent is probably going to take your job. Amidst all the uproar, it occurs to me we’re glossing over one really important question: what is knowledge in the first place?
Of course, there’s an entire branch of philosophy called epistemology that is dedicated to answering this exact question, and many compelling explanations exist; my goal is not to seriously contend with them but instead, offer my own perspective on knowledge, specifically as it pertains to the threat posed by artificial intelligence. By the end, perhaps you’ll be convinced, like I am, that knowledge is inherently human, and machines can only approximate it.
To support my argument, I’ll need to introduce a little formalism, but I’ll try my best to wield it casually. We’ll start by establishing what truth is—according to the Stanford Encyclopedia of Philosophy, “truth is a matter of how things are, not how they can be shown to be”. In other words, the real nature of the world exists whether we can comprehend it or not. For example, consider the fact that for centuries we had no idea that bacteria existed; the fact that we were unaware of its existence didn’t change the truth that it was there, swimming and scurrying around the whole time.
Thus, to know something is to have a “certain kind of access”1 to the truth. But there’s more to knowledge than just the recognition that something might be true—it requires the ability to prove it. Imagine before a die is rolled, I tell six different people that I know which number will come up. It lands on four, and according to exactly one person, it seems like I did actually know. But you could hardly make the case that I ever had any real “knowledge”2 about what would happen, I just rigged the experiment so that I couldn’t lose. This is an example of “epistemic luck”3 and it poses a real problem—we don’t want to blindly trust someone or something simply because of an accurate prediction. Therefore, we demand that in order for truth to become knowledge, it must come with an explanation.
So it seems that there’s a justification requirement4 to knowledge, and that’s particularly interesting because it implies that, unlike truth, which is universal, knowledge is social. It requires certification, a process that inherently involves at least two parties. Copernicus knew as early as 1543 that the Earth revolved around the Sun, not the other way around, but it took generations for us to collectively accept that idea. The transition was slow and gradual and required cooperation to become crystallized as knowledge.
The attentive reader may spot a weakness. What if Copernicus were stranded on an island? Would he suddenly cease knowing? Is knowledge rendered unobtainable if no one else is around to validate it? I don’t think so, and here’s why: humans have a self.
Even in solitude, the knowledge generation process is faithfully carried out because establishing belief involves first convincing the self that something must be true. This step is a necessary precondition of knowledge; you can’t claim to know something that you don’t earnestly believe. By invoking the self, I’ve invited a ton of new guests to the dinner table—desire, intention, motivation, identity. All these things are rooted in our inner self5, which means the formation of knowledge involves navigating a proposition across a spectrum of human emotion before arriving at acceptance. All of this presupposes, of course, that the proposition is true in the abstract sense, because you can’t know something that is false.
So if knowledge involves convincing yourself before convincing the rest of us, it would mean that a machine is incapable of truly knowing, even if it can simulate knowledge quite well. That’s a deeply philosophical position. There’s even a valid counterargument that maybe machines can develop a self as some emergent property. Entertaining this line of investigation further would drag us into really deep nuance involving neurobiology, psychology, mathematics and of course, philosophy. As intriguing as this topic might be, for the sake of this piece, I’m asking you to assume that machines don’t really have a self, which is not unreasonable given the current state of our research.
Let’s move out of abstract philosophy. Because honestly, who cares whether a machine has a self or not, or whether knowing has anything to do with self-awareness; what really matters is that knowledge produces a sufficient ratio of true outputs to false ones. We don’t necessarily need to understand how we arrived at an answer so long as we can verify that it’s correct. Fair enough, but that perspective invites its own criticism: it introduces a reliability problem. Alvin Goldman and others have argued in favor of an epistemological concept called “process reliabilism”6, which essentially says there is value in understanding the process by which something becomes known because it increases our confidence that other propositions that use that method will be true. For example, we trust microscopes as instruments for observing things on a really small scale—if we discover something new using a microscope, we would likely be much more excited than if we saw something through some untested apparatus. That’s because the microscope is more reliable.
The reliabilism argument stems, in part, from the philosophical question about getting lucky, but as we’ve seen, it also speaks to how we effectively cultivate knowledge. Time and resource constraints inherently prohibit us from pursuing every conceivable theory, so an ability to judge the means and methods of belief formulation is practically necessary so that we can make progress. Imagine if the NIH had to fund every single research proposal it received; imagine if the New England Journal of Medicine had to thoroughly review every submission; we’d spend more time fruitlessly looking for needles amidst a giant haystack than actually exploring serious hypotheses. Most forms of AI have a notoriously opaque generative process; that alone should make us wary of overreliance. And if we don’t have great insight into how an AI came up with a theory, it means we have to test every single one it develops.
There’s even more to the practical implications of fully autonomous machine intelligence. Even if we concede that meaningful introspection into their thought process isn’t strictly necessary, at the very least we need reasonable assurance that it’s consistent, otherwise we’d still be reduced to testing every single theory it proposes. Reliability is more than a qualitative judgment of methods involved; it helps us form expectations and make trustworthy comparisons. Even the most successful track record is undermined by stochasticity. So if we were to fully surrender all decision-making authority to an AI, we would either need observability into its cognitive process (which we don’t have) or have to thoroughly test every single one of its claims (which we can’t do).
It seems like there are both practical and philosophical dilemmas at the intersection of artificially intelligent systems and human intellectual capital. Knowledge isn’t really something a machine can possess, but even if it could, we would need to understand the world well enough to judge that it actually knows. And once we possess that level of understanding, we don’t need the AI.
A curious reader might say there’s a middle ground—we can use AI, not as an authoritative vessel, but as a tool for raising our own epistemic potential. In this sense, AI is simply another competitor in the marketplace for ideas; a marketplace whose regulators are exclusively human beings. In my opinion, it doesn’t matter where ideas come from; all that matters is that people are involved in verifying them. Alexander Fleming discovered penicillin by accident. One day, an AI might contribute to a cure for Alzheimer’s. The important thing is that humans remain interested in understanding, and continue to be the judges of what gets enshrined as knowledge. And now my argument has come full circle: humans possess knowledge, machines approximate it, and the future rests on having more, not less, people who are inquisitive and brave enough to judge which approximations are the right ones. My advice? Learn math. Read the code. Stay curious. To give up on learning is to give up on existence.