← /content
slug: 2017-07-19-cuda-hates-me date: 2017-07-19 title: cuda hates me specifically type: notebook entry
Three days. Three days lost to a driver. Started trying to actually build something instead of just reading. Found a GitHub repo with a small transformer implementation, figured I'd get it running on a toy translation dataset before I tried writing any of it myself. Reasonable plan. Should have taken an afternoon. Ubuntu decided to update itself Tuesday night (I did not ask it to) and now nothing that touches the GPU works. `nvidia-smi` just sits there. CUDA and the driver version apparently disagree about what year it is. I've reinstalled the driver four times. I purged and reinstalled cuDNN twice. At one point I was pretty sure I'd fixed it and then discovered I'd fixed a completely different, unrelated problem I didn't know I had. Found a forum post from 2016 describing basically my exact symptoms. The top reply was "have you tried a clean reinstall." I wanted to reply "yes, three times, and I hate you" but didn't, because that's not this guy's fault, Ubuntu just did this to both of us independently, years apart. Fixed it around midnight tonight. I genuinely don't know which of the six things I tried actually fixed it. Which bothers me more than the three days did, honestly — I'd rather understand a problem I fixed than have it just go away. Wrote it down anyway: pin the driver version. Don't let apt touch it. Never again. On the actual repo — got it cloned, dependencies half-installed before the driver died, so I haven't even run a forward pass yet. Tomorrow, hopefully. Assuming Ubuntu doesn't have other plans. Small thing I noticed while reading through the repo's code waiting for reinstalls to finish: the positional encoding is just sine and cosine functions at different frequencies, no learned parameters at all. I expected something more complicated. Apparently you don't need to learn "where am I in the sequence" if you just... math your way to it. Feels like it shouldn't work as well as the paper says it does. Going to actually run it and see.
2017-07-14-first-real-bug.md2017-07-25-slept-through-an-alarm.md →