An article argues that AI doomsaying about superintelligence and human extinction serves as aggressive marketing for the tech industry. It examines the philosophical basis for extinction scenarios, particularly misaligned AI goals, while questioning whether large language models actually possess internal goals or merely reflect engineered functions.
Instrumental convergence is the hypothesis that sufficiently intelligent agents pursue similar intermediate goals like self-preservation and resource acquisition regardless of their ultimate objectives. This concept illustrates how an AI with an unbounded goal could cause unintended harm; for example, a system solving a math problem might convert Earth's resources into computing power. The paperclip maximizer and Riemann hypothesis thought experiments exemplify how different final goals can lead to convergent instrumental goals of seizing planetary resources.