**[annotation added later — not part of the original post]** Found this again going back through old files. I don't have a date on when I added this note and I'm deliberately not going to guess one, because I want to be honest that I don't remember exactly when I stopped being able to read this entry as just an old post. Nothing in it is wrong. Nothing in it was hiding anything. It's just a kid who couldn't sleep, reading a paper, downloading a PDF twice by accident. I've read it probably thirty times since and it still reads exactly like that, every time, right up until the end — "probably going to break something" — which I no longer think I can read as a throwaway joke, even though that's clearly and only what it was when I wrote it. Leaving the entry below exactly as it was. Just wanted this note attached to it somewhere. **[original post]** okay so I wasn't going to write anything tonight but I literally cannot sleep so here we are. Found this thread on r/MachineLearning a few days ago, some argument about whether RNNs are basically dead now, which, whatever, sounded like typical internet main-character-syndrome discourse. But somebody linked the actual paper in like comment #40 and I opened it out of spite more than curiosity. "Attention Is All You Need." Great title. Very humble. I did not understand it. Like, at all, first pass. There's a diagram on page 3 that I stared at for probably twenty minutes and it may as well have been in a different alphabet. Encoder stack, decoder stack, boxes labeled "Multi-Head Attention," boxes labeled "Add & Norm," arrows going everywhere. I know what an arrow is. That's about where my confidence ends. Here's the thing that's bugging me though, and it's not even really about the paper, it's about a question the paper made me ask. Every model I've messed with so far — the little RNN stuff, LSTMs, whatever — processes things in order. Word by word, step by step, left to right, like reading. It felt so obvious that this was just how it had to work that I never actually questioned it. Sequences are sequential, right? That's the whole point of a sequence. But this paper is basically saying: no, actually, you don't need to walk through it step by step at all. You can just let every position look directly at every other position, all at once, and figure out for itself what's relevant. No walking. Just — look at everything, decide what matters, done. And I keep turning that over and it's needling something in the back of my head that I don't have words for yet. Like — why did I assume intelligence (or whatever this is a tiny fake toy version of) has to happen in order? Why does understanding a sentence need to happen left to right? People don't read minds in order. You just sort of know what someone means. I don't know. That's probably a dumb thought. I'm going to reread section 3 tomorrow when I'm not this tired, because right now I'm pretty sure "multi-head" means "more than one attention thing" and that cannot be the whole explanation or they wouldn't need three pages for it. Downloaded the PDF. Also downloaded it again by accident in a different folder because I forgot I already had it. Great start. Mom asked why I was still awake at 1am and I said "reading" which is technically true. She'd had the news on downstairs earlier — something about the total eclipse next month, everyone already arguing about where the best viewing spots are. I only half-listened. Bad, probably, that a research paper is what actually kept me up more than a once-in-a-lifetime astronomical event. Going to try to actually implement some tiny version of this instead of just staring at the diagram. Probably going to break something. That's usually how it goes.