> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache
Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment:
if you're using a cache, you should invalidate it on failure!
It's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.
What I meant is that if the compromised host was unable to use broken boltdb cache, the cache should be zeroed and repopulated. Was it really hours of such cache rebuild vs hours of trying to fix the boltdb?
Btw I am happy I got only small amounts of data in any of bolt databases...
This isn't a boltdb we designed. It's just containerd. I am probably not doing the outage justice, because "blitz and repopulate" is a time-honored strategy here.
Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment:
if you're using a cache, you should invalidate it on failure!