Grant Sanderson, the maths educator of 3blue1brown fame, recently started a series around the idea of “compression is intelligence”. He talks about information theory and how it relates to AI, and although the series only has a few entries, I’m already very fond of the general framing: that the ability to take the real world, distill something more fundamental out of it, and be able to relate that to other things in a useful way, is what intelligence is.
When I meet people who I think are really smart, I’ll often hear something like “yeah I didn’t study much during school”. When I say smart, I don’t mean that they know everything, but instead that they learn quickly, are good at teaching, can describe something abstractly, then link it to an example, then dive into the details, all with minimal effort. So obviously, these people didn’t need to study much, which forms a neat explanation. Except, I’ve just recently wondered if perhaps some causality may go the other way. By that I mean that part of the reason they are smart (by my definition) is because they didn’t study. Before you accuse me of being crazy, I want to talk about how we train AI, and more precisely, talk about memorisation versus generalisation in the field of machine learning.
When we train almost all AI systems, we often put in a huge amount of effort into just stopping the damn things from memorising. We sometimes call this overfitting, and there is a similar form called shortcut learning. If you have a picture of a bee, and you want a neural network to say “that’s a bee”, many naive methods will just find some small pattern of pixels in the image and learn “if I see those pixels there, it’s a bee”. This isn’t useful at all, usually it only works for exactly one picture of a bee and no other bees, and so you get amazing results on your training dataset and abysmal ones on your test. What we want the models to do is to generalise, we want them to learn “a bee has these colours and this shape and it has wings and this many legs” and to learn that it needs to learn “a leg is a thing where…” and “a wing is a thing that…” and so on and so on, all the way down to pixels, somehow. The reason doing this is hard is because with the wrong incentives, memorising is often so much “easier”. We usually train models iteratively, step by step, and at each step there’s only a few pictures or samples it needs to get better at judging. Why try to find some beautiful generalisable representation when just memorising some pixel rules is good enough for these 32 images or whatever? For this reason, big strides in machine learning have come from finding ways to stop models from memorising, or encourage them to take memorised concepts and generalise them.
Now back to us humans. I’ll use myself as an example human (yes, no AI was used in writing this, except for a few spelling checks at the end). I’ve always hated memorising things. Maybe it’s my ADHD but the why hardly matters, I never did times tables, I hated learning word translations in language classes, I never bothered to remember formulas (I’m immensely grateful for being allowed a formula book for many subjects). Instead, I’d describe wanting to always use intuition and get my answers during exams from building things up from first principles. In many ways, I had no other choice, my refusal to do most homework left it as my only option. For some things like languages, this didn’t work and that’s why I almost failed those classes. There’s hardly a consistent principle for masculine and feminine forms in French. So I was one of those “I didn’t study much” people, but I still attended every class, asked lots of questions, and I didn’t take notes. Instead, I let my mind wander as the teacher talked. It’d wander around the topic being taught and try to see how the topic connected to things I already knew (hopefully, sometimes I’ll concede I was on my phone).
Today, I work as a systems architect and I remain awful at remembering little things or forcing myself to learn small, arbitrary details. I’d call them “accidental complexity” and avoid them like the plague. But I learn quick, and seeing the big picture has been extremely valuable in my work. I wouldn’t really say one way of thinking is necessarily better than the other, but here’s the point that I’d like to make with this blog post: When we teach kids and construct curriculums, I think we should be much more careful in what kind of learning we incentivise. I’d make the bold statement that right now, we focus far too much on memorisation, and in so doing may be disincentivising people from developing fundamental understanding as they learn. I’m speculating massively here, but if each lost opportunity to generalise decreases the value of generalising in the next step then at some point you will hardly feel any need to, even if you would be better off in aggregate having some more fundamental intuition.
I don’t have any concrete ideas of what that would mean for teaching, but it does align a lot with what kinds of teachers I really appreciate. Grant Sanderson himself is a great example, he often tries to build that fundamental intuition and has also realised that sparking curiosity (for example with compelling storylines) is one of the best ways to do that.
Hopefully this post has been interesting to you. This idea itself is me generalising between topics like how teachers teach, how the field of ML has progressed, and how human brains learn, each of which i’ve learned about in academic settings but refused to just memorise. It might be a load of tenuously connected abstract junk (of which I will concede I do produce from time to time), or it may have some fundamental truth behind it. I leave that distinction up to you, feel free to write comments here.