Fix prune failure due to stale lock ΒΆ
If a backup crashes, the restic repository can be left in a locked state. This will not prevent further snapshots, but will prevent prune operations. Such a state will manifest as the `IndeVets/ Prune Snapshots is missing” Dead Man’s Snitch alerting with investigation of the pod logs for the hourly snapshot showing:
Pruning aged snapshots
unable to create lock in backend: repository is already locked by PID 10 on indevets-core-prod-hourly-27765255-dmv69 by root (UID 0, GID 0)
lock was created at 2022-10-16 12:15:05 (25h8m11.241121433s ago)
storage ID 4074c3bd
the `unlock` command can be used to remove stale locks
To repair this state, it is necessary to manually unlock the repository after confirming that whatever issue caused the original crash is not persisting (i.e. the prune step is the only thing still failing).
-
Open a bash shell
If you use a zsh shell, run
bash
first to switch to bash in a subshell -
Enable exporting all set variables:
set -a
-
Locate the
IV-Restic Bucket
secret in VaultWarden and paste the note content into your bash terminal -
Run:
restic unlock