← /contentslug: 2018-09-08-designing-not-just-reproducing
date: 2018-09-08
title: designing experiments instead of just reproducing them
type: notebook entry
Senior year started last week. Different feeling than junior year did — less "catching up," more "waiting to be done so I can do the thing I actually care about." Not great for GPA motivation. Grades are still fine. Just fine, not great.
Noticed something about how I'm working now versus a year ago. All of last year was basically: find a paper, try to reproduce a small piece of it, see if I understand why it works. This year I've started doing the opposite more often — having a question first, then designing a small experiment to answer it, and only checking the literature afterward to see if someone already answered it (usually they have, but not always the way I expected).
Case in point: spent two weeks this summer testing whether a transformer trained on my toy tasks actually needs the full attention pattern, or whether restricting it to only attend to nearby positions does almost as well on short sequences. Turns out for short sequences it barely matters. For longer ones it falls apart fast. Not a novel finding — there's already research on local/sparse attention — but I found the question myself, tested it myself, and then went looking for confirmation instead of starting from someone else's conclusion. Different feeling.
Dad asked at dinner what I'm actually trying to figure out with all this, and I didn't have a great answer. Told him "how to build something that gets better at learning on its own instead of me having to redesign it every time." He nodded like that made sense to him. It didn't, I don't think, but he's stopped needing it to. That's it's own kind of nice.