Deploying Environment Variables

ConfigMaps are kubernetes API objects used to store non-confidential data and Secrets are API objects used to store sensitive data. All of our secrets are also “Sealed” via the Sealed Secrets library, which encrypt our secrets so they can be securely committed into the Github repository.

Creating & Testing Temporary Changes

Note: Making live changes to the kubernetes cluster works as a quick solution, but our cluster is driven by GitOps, which means any changes that are not also represented in the repo will be lost during the next Github deployment.

In some instances, you’ll want to quickly change/override an environment variable while developing/debugging. This workflow will work for any namespace, but is primarily recommended for the sandbox environment. Changes to staging/production should be done via the methods below. In some cases, you may want to use this method first, in the interim of the next deployment. Note: these changes will be lost if the pod gets deleted for any reason.

Manually via Lens

1. Open Lens, select the deployment we want to target, and then click Edit

2. Edit the deployment templates’ environment variables via Lens editor

   • Look in the template for the containers configuration. spec.template.spec.containers
   • Find the -app container that we want to target
   • Update the .env configuration under the targeted container following the existing format:

       - name: ENV_VARIABLE_NAME
         value: ENV_VARIABLE_VALUE
   

3. Save your changes – this will trigger the deployment to spin up a new pod with the new environment variables set.

Committing Permanent Changes

Non-Secret Configuration

1. Create a new K8S ConfigMap file with the related env variables

   • Copy k8s/environments/sandbox/core-app-sandbox.yaml as a starting template. For example, copy the file to k8s/environments/sandbox/google-anaylitcs.yaml. Note: Make sure the file gets added to the appropriate namespace folder. Create a file for each environment if necessary.
   • Update/Replace: Note: Indentation/Spacing is important
   • data - Remove existing values, and place config values with format: CONFIG_NAME: CONFIG_VALUE.
   • metadata.name - Update name to match context of variables being added.
   • metadata.namespace - Update namespace to match the namespace being deployed to.

2. Add config to helm deployment value file(s) under app.envFrom.configs.

   • Apply value for ALL namespaces, or a default value, to helm-chart/values.yaml values file. If there is no default value, set to ~ to signify null
   • Apply staging values (if necessary) to: k8s/cats-staging/release-values.yaml
   • Apply production values (if necessary) to: k8s/cats-production/release-values.yaml
   • Note: Release values for PR (sandbox) environments are configured in .github/workflows/pr-k8s-deploy.yml workflow script.

Secret Configuration

1. Create a new K8S Secret file with the related env variables

   • Copy k8s/environments/sandbox/postgres-sandbox.yaml as a starting template. For example, copy the file to k8s/environments/sandbox/google-anaylitcs.yaml. Note: Make sure the file gets added to the appropriate namespace folder. Create a file for each environment if necessary.
   • Update/Replace: Note: Indentation/Spacing is important
     • data - Remove existing values, and place config values with format: CONFIG_NAME: CONFIG_VALUE.
     • metadata.name - Update name to match context of variables being added.
     • metadata.namespace - Update namespace to match the namespace being deployed to.

2. Seal it in the appropriate k8s/cats-{ENVIRONMENT}.secrets directory. Note: Once sealed, you can delete the original Secret file.

3. Add config to helm deployment value file(s) under app.envFrom.secrets.

   • Apply value for ALL namespaces, or a default value, to helm-chart/values.yaml values file. If there is no default value, set to ~ to signify null
   • Apply staging values (if necessary) to: k8s/cats-staging/release-values.yaml
   • Apply production values (if necessary) to: k8s/cats-production/release-values.yaml
   • Note: Release values for PR (sandbox) environments are configured in .github/workflows/pr-k8s-deploy.yml workflow script.

Deploying Committed Changes

Via Github

1. Push the aforementioned SealedSecret / ConfigMap file(s)/helm template changes to Github

   • Create a new PR with the SealedSecret/ConfigMap file(s) / helm template changes, and merge it into develop. Merging into the develop branch will trigger Github Actions to create a new PR (Deploy releases/k8s-manifests) for deploying the changes to the K8S cluster
   • Note: If there are already pending K8S manifest changes waiting to be deployed, merging will update the existing PR. This can cause some issues if there are K8S updates that are not ready to be deployed. In this case, you’ll want to continue with the Deploy K8S Resources Manually instructions.

2. Merge Deploy releases/k8s-manifests PR, which will trigger another Github Action to deploy the changes directly to the cluster. Note: This step can be destructive. Be sure to verify changes in the PR before merging

Once the changes are deployed, you’ll need to finally restart the pod(s) that need the latest ENV variable changes. You can do this manually by deleting the pod in Lens or by re-triggering the Github Deployment.

Manually

1. Project the k8s-manifests holobranch (e.g. git holo project k8s-manifests --commit-to=BRANCH_NAME) and checkout that branch.

   • Note: Projected Resources are grouped by namespace.

2. Confirm cluster changes

   • Run kubectl diff -Rf DIRECTORY to verify the changes
     • -R flag is recursive for directories
     • -f flag is for the k8s manifest file we want to be diff’d

3. Deploy cluster changes

   • kubectl apply -Rf DIRECTORY to apply an entire directories changes
   • kubectl apply -f indevets-sandbox/ConfigMap/core-app-sandbox.yaml to apply changes from a singular K8S resource file.