← /contentslug: 2019-11-02-mostly-garbage-occasionally-not
date: 2019-11-02
title: mostly garbage, occasionally not
type: notebook entry
Been running the Loop (still can't believe that name stuck) most nights for a couple months now. First semester of actual college classes, which are mostly review so far — a blessing, honestly, means I've had more evenings free than expected.
Log of what it's actually produced, roughly: about 90% of the changes it proposes are either useless or actively worse than baseline. Maybe 8% are neutral, don't help or hurt. Then every so often — I'd guess one time in twenty or thirty — it stumbles onto something I wouldn't have thought to try, and it's actually an improvement. Nothing huge. A slightly better weight initialization scheme for one of my toy architectures. A learning rate schedule that converges faster on one task type. Small stuff. But stuff I didn't propose.
I keep trying to figure out if that 1-in-25 hit rate is impressive or not. On one hand: a script with genuinely no understanding of why anything works occasionally beats my own intuition, which took me two years to build. On the other hand: 24 out of 25 of its ideas are garbage, and I have to manually review every single result to find the good one, which honestly eats up close to as much time as it saves.
Added a very crude auto-filter this week — throw out anything that doesn't beat baseline by some margin before I even look at it. Cuts my review time down a lot. Also definitely means I'm throwing away some interesting failures along with the boring ones. Tradeoff I'm making on purpose, might regret it.