← /contentslug: 2020-01-30-scaling-laws-paper
date: 2020-01-30
title: a paper that explains why bigger keeps winning
type: notebook entry
OpenAI put out a paper on scaling laws this week — basically an attempt to formalize something that's been an open secret for a while: model performance seems to follow fairly predictable power-law curves as you increase model size, data, and compute together. Not a vague "bigger is better," an actual quantifiable relationship.
Read it twice. Part of me finds it satisfying — I like knowing there's a real underlying regularity instead of everyone just guessing and getting lucky at different scales. Part of me finds it a little bit demoralizing, if I'm honest, because it's a pretty direct statement that a huge amount of what determines how good a model gets is just... resources. Not cleverness. Not insight. Mostly just how much compute and data you can throw at the thing, scaled correctly.
Which raises an obvious, uncomfortable question about my own project that I don't have a good answer to yet: if the field's own math says scale dominates, what exactly am I contributing by being clever with a fraction of a percent of the compute anyone serious has access to?
Best answer I've got right now: the Loop isn't trying to compete on scale. It's trying to get more value per unit of experimentation — better questions, less wasted compute per insight. That might matter more at my scale than at theirs, where compute is cheap enough that brute-forcing bad ideas is often faster than being careful about which ideas to try. Not sure that's actually true. Writing it down so I can check later whether I still believe it.