Grant Sanderson's concept of 'compression is intelligence' frames intelligence as the ability to distill fundamental principles from the real world and apply them broadly. The article parallels this with machine learning, where models must be prevented from memorizing specific patterns (overfitting) and instead learn to generalize, similar to how truly intelligent people grasp abstract concepts rather than memorizing facts.
An engineer describes learning to memorize the first 100 digits of pi using memory techniques from Joshua Foer's book. The key method involves elaborative encoding and a Person-Action-Object (PAO) system, which transforms digit sequences into memorable visual glyphs placed in a memory palace. Building the PAO system took more effort than memorizing pi itself, but once established, memorizing 100 digits required only about an hour.
Grant Sanderson frames intelligence as compression—the ability to distill information and connect concepts. The author explores parallels between how AI systems generalize rather than memorize and how humans who avoid rote memorization develop deeper understanding, suggesting that constraints on memorization can paradoxically enhance learning.
Grant Sanderson's "compression is intelligence" framework suggests that true intelligence is the ability to distill and abstract information rather than memorize it. The author draws parallels between how AI systems must be prevented from overfitting to avoid memorization, and how humans who avoid rote memorization often develop stronger generalization and learning abilities. This perspective reframes the advantage of not studying as a feature that forces deeper conceptual understanding.
Researchers introduce long-horizon memorization, a challenge where language models must learn 100 tasks through continual fine-tuning without catastrophic forgetting. They show that composing multiple continual learning mechanisms—combining data, function, and weight anchors with merged LoRA—improves retention from 1.2% to 34.9%, a 28-fold improvement over naive sequential fine-tuning.