Not really a technical post, more just closing the year out. Six months ago I didn't know what an attention mechanism was. Now I've got a small transformer I built mostly by hand, trained on toy tasks, running on a GPU I bought used off someone upgrading their gaming rig. That's a weirder sentence to write than I expected. Still haven't gotten the GPU upgrade — closer, not there. Still haven't done anything on a real dataset, real scale. Everything I've built this year is a toy by any actual research standard. I know that. I'm not under any illusion that I've done something impressive by ML-research standards. I've done something impressive by "started from basically nothing in July" standards, which is a much lower bar and the only one that's actually mine to clear. Junior year is fine. Grades are fine. Dave says I can pick up more hours over winter break if I want them, which I do, mostly for GPU-fund reasons. One random thought before I close the notebook for the year, not sure it's worth anything: everything I've trained this year, I built the dataset myself, by hand, or downloaded some fixed toy set someone else made. What happens if a model could generate its own training data — not just augment existing data, but actually build the thing it learns from next? Probably that's either a solved problem I don't know about yet or a genuinely bad idea for reasons I haven't thought of. Writing it down mostly so I remember I had the thought, not because I think it's good. Anyway. 2018. See what happens.