Skip to content

Fix prune failure due to stale lock ΒΆ

If a backup crashes, the restic repository can be left in a locked state. This will not prevent further snapshots, but will prevent prune operations. Such a state will manifest as the `IndeVets/ Prune Snapshots is missing” Dead Man’s Snitch alerting with investigation of the pod logs for the hourly snapshot showing:

Pruning aged snapshots
unable to create lock in backend: repository is already locked by PID 10 on indevets-core-prod-hourly-27765255-dmv69 by root (UID 0, GID 0)
lock was created at 2022-10-16 12:15:05 (25h8m11.241121433s ago)
storage ID 4074c3bd
the `unlock` command can be used to remove stale locks

To repair this state, it is necessary to manually unlock the repository after confirming that whatever issue caused the original crash is not persisting (i.e. the prune step is the only thing still failing).

  1. Open a bash shell

    If you use a zsh shell, run bash first to switch to bash in a subshell

  2. Enable exporting all set variables:

    set -a
    
  3. Locate the IV-Restic Bucket secret in VaultWarden and paste the note content into your bash terminal

  4. Run:

    restic unlock