K8s-to-VM database migration

In September 2021, CATS’ primary database was migrated from being hosted within the application’s Kubernetes cluster to being hosted in on a dedicated VM.

Sync dedicated DB from k8s

1. Connect to kubernetes cluster:

   export KUBECONFIG=~/.kube/indevets-live.yaml
   
   kubectl get nodes
   

2. Capture name of current application pod:

   APP_POD=$(kubectl -n indevets-production2 get pod -l app.kubernetes.io/name=indevets-core -o jsonpath='{.items[0].metadata.name}')
   

3. Scale destination system down to 0 replicas to ensure in-use resources don’t prevent a clean drop/recreate

4. Put source system into maintenence mode

5. Pipe Postgres dump from k8s instance to dedicated instance:

   kubectl -n indevets-production2 exec $APP_POD -c indevets-core-prod-db \
       -- bash -c 'pg_dumpall --clean -U${POSTGRES_USER}' \
       | ssh admin@primary.db.indevets.com sudo -u postgres psql
   

   Tip

   The password for the admin user on admin@primary.db.indevets.com is stored in BitWarden under the item admin@admin@primary.db.indevets.com