← /contentslug: 2017-08-30-deleted-two-weeks-of-results
date: 2017-08-30
title: deleted two weeks of results like an idiot
type: notebook entry
Wrote a cleanup script to clear out old checkpoint files because the SSD was almost full. Wrote it a little too aggressively. It cleared out the checkpoints I actually wanted to keep too, including the ones from the reversal-task run I was proudest of last month. No backup. I know. I know.
Spent an hour being genuinely upset about this before remembering that the actual point was understanding why it worked, not the specific weights file, and I still have all my notes and the training logs, just not the model itself. Could retrain it in twenty minutes if I really needed to. Mostly just annoyed at myself for not having a backup habit yet, five weeks in.
New rule, effective immediately: nothing gets deleted by a script I wrote in the last ten minutes without a dry run first. Learned that the expensive way, which seems to be the only way I learn some things.