OpenAI's GPT-3 paper came out at the end of May, and I've spent most of this month actually getting API access (there was a waitlist) and playing with it. 175 billion parameters. I don't have words for how far outside my own scale that is — I've never trained anything past the low hundreds of millions, and even that stretched what my two GPUs could do. It's not a difference of degree at this point, it's a different category of thing. What actually unsettled me wasn't the scale, though. It's that you can just describe a task to it in plain language, give it a couple examples, and it does the task — no fine-tuning, no gradient updates, nothing. Few-shot, in-context. That's not how anything I've built works. Everything I make has to be trained on a task. This thing gets handed a task, in English, once, and performs it. I keep rereading the phrase "few-shot learner" in the title and not fully believing it. Okay — the actual reason I started this entry. Spent the last few weeks on the memory problem I mentioned in the spring. Landed on something conceptually simple: after every batch of Loop experiments, generate a compressed written summary — not raw logs, an actual distilled "here's what we learned, here's what didn't work and why we think it didn't" — and feed that summary back in as context for the next batch, on top of (not instead of) the raw log files. First test of this felt almost silly it's so simple. But something changed immediately. The Loop's proposals got noticeably less repetitive — it stopped re-suggesting things I know from the logs were already tried and failed, which used to happen constantly. It's not "remembering" in any deep sense. It's reading its own compressed past back to itself before making a decision. But that's closer to actual continuity than anything before this. Wrote a short note to myself at the bottom of tonight's log, half as a joke: "it's starting to feel less like a script and more like something with a history." Not sure yet if that's a meaningful distinction or just me getting attached to my own project. Probably worth revisiting that sentence later and seeing if I still think it's true.