Skip to content

Deployment Process

Production deployment

  1. Start with an automatically-generated PR to update production to a new version reviewed and ready to merge

    See GitHub Actions Workflows

  2. Verify that the Docker image for the new version have been published to GitHub at https://github.com/IndeVets/core/pkgs/container/core%2Findevets-core. Ideally, it has already been deployed to staging.

  3. Check that all DaemonSets under the kube-system namespace report as running 5 pods

    If they aren’t, check that the changes submitted in linode/linode-blockstorage-csi-driver#73 are manually applied to Linode’s csi-linode-node DaemonSet. Once that change becomes encoporated in Linode’s own releases this step can be removed.

  4. Verify latest restic backup is recent:

    set -a
# paste contents of `IV-Restic Bucket` BitWarden secret
set +a

restic snapshots

  5. If the latest snapshot is more than 10 minutes old, manually trigger a run of the hourly Kubernetes CronJob inside the production namespace

  6. Download the latest snapshot to your computer and verify that it is complete:

    # replace hash value with latest snapshot ID from above
restic dump de27e150 /indevets-core.sql.gz | gunzip > /tmp/indevets-core.sql

    The .sql file should be 1.4gb+ and contain schema, data, sequences, indexes, grants, and finally materialized views. Scan through the contents until you see data, and then jump to the bottom and make sure it ends with materialized views to ensure no dump error has cut it short.

  7. Merge the PR to bump the production version

  8. Wait for the k8s deployment PR to be ready and then review bot the kubectl diff changes shown in the PR description and the files changed shown by the GitHub PR to ensure no unexpected additional changes are going out
  9. Announce to @here in #indevets-team on Slack that a deployment is beginning
  10. Merge the deployment PR
  11. Monitor production pods until they are fully back online
  12. Verify that you can log into production and see the new version number under the user menu
  13. Announce to @channel in #indevets-team that the production is complete and ready for verification