← /contentslug: 2018-06-14-someone-corrected-me-and-it-was-useful
date: 2018-06-14
title: someone corrected me and it was actually useful
type: notebook entry
Posted a comment on a thread last week trying to explain why I thought multi-head attention worked, got it partially wrong in a way I didn't notice until someone linked a reply from their own blog tearing apart the specific error. Not mean about it. Just precise. Went and read the actual post — some guy who goes by Mr. Z, runs a personal site (zoff.tech, if I'm remembering the URL right) that's mostly hardware teardowns and systems-programming stuff, with the occasional AI post mixed in. Old-internet style site. No ads, no popups, just a guy who clearly knows a lot writing because he wants to, not because anyone's paying him to.
His correction was right. I'd conflated "each head learns a different subspace" with "each head learns a different, individually interpretable role," which isn't the same claim and isn't reliably true — heads specialize unevenly, some redundantly, and expecting clean interpretability out of every one of them is wishful thinking dressed up as an explanation.
Emailed him a follow-up question instead of replying in the thread, mostly because the actual question felt too long for a comment box. Wasn't sure I'd hear back. Got a reply within a day — short, no wasted words, answered exactly what I asked and then pointed out I was asking the wrong follow-up question and told me what the right one was instead.
Don't know anything about him beyond the site. Don't know his real name, where he works, any of it. Doesn't matter much. First actual useful correction I've gotten from someone who clearly knows more than me, delivered without an ounce of condescension. Bookmarking the site.