Okay. Small win, but a real one. Got the toy transformer training on a tiny synthetic task — basically "copy this sequence of numbers, but reverse it." Dumb, contrived, not translation, nothing anyone would care about. But it's small enough that I could watch it overfit on purpose and actually understand what I was looking at. First twenty minutes of training, loss barely moves. I assumed I'd broken something again, went to go check the learning rate, came back, and the loss had fallen off a cliff. Like actually fallen — not gradually, just dropped. Within another ten minutes it was getting sequences of length 10 exactly right, every time. I know this is nothing. It's a toy dataset with maybe 200 examples. A dictionary lookup could probably do this. But watching the attention weights while it trained — there's a way to print them out as a little grid, which position is "looking at" which other position — and by the end, the pattern was completely clean. Position 1 attending almost entirely to the last position of the input, position 2 to the second-to-last, and so on. It figured out the reversal mapping on its own, with no one telling it "reversal means position i maps to position n-i." That's the part that got me. Nobody told it what reversal was. It just had "minimize this loss" and it found the mapping that does that, and the mapping happens to be exactly the thing I would have written if I were doing it by hand. I keep thinking about the sine/cosine positional encoding thing from a few weeks ago. The model has no idea what "position" means except through that encoding. And somehow that was enough information for it to reconstruct an entire reversal operation just from gradient descent. I don't have a grand takeaway. I just wanted to write down that this is the first time one of my own experiments has done something I didn't explicitly engineer it to do. Every previous project I've done — the CNN stuff, the tutorials — I always basically knew what the answer would look like before I ran it. This one I didn't. I just watched it find the answer. Going to try something slightly less trivial next. Maybe actual character-level translation, something French-to-English, small.