Skip to content

Openshift

Jump to bottom
afwilcox edited this page Sep 25, 2025 · 6 revisions

Introduction

Openshift is used to host Natsuite in dev, test, and production in the emerald environment. The license plates for these namespaces are f208ae-dev, f208ae-test, and f208ae-prod

URLS:

The dev namespace contains sandboxed instances of NatSuite for each PR in GitHub. The test environment contains at most two instances (one for test releases, and one for a hotfix). The prod environment contains at most one instance.

Argo is used for OpenShift deployments.

Gitops Repo

The gitops repo contains the helm charts to deploy the application to OpenShift. This is required because Argo (used for our deployment pipeline) only has access to this repo (not the case management repo). A github action has been setup in the case management repo to update the values file, which will trigger a deployment of the case management application to Emerald.

URL: https://github.com/[bcgov-c/tenant-gitops-f208ae](https://github.com/bcgov-c/tenant-gitops-f208ae)

Argo

Argo is the gitops component of our pipeline. Once a pull request is approved in the gitops repo, Argo will see the change and deploy the new images to Emerald.

I had to login using my github account to setup argo. All that is required on this end is to create an application. Here's my yaml for that:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dc0a4a-compenf-dev
spec:
  destination:
    name: ''
    namespace: dc0a4a-dev
    server: 'https://kubernetes.default.svc'
  source:
    path: charts/nr-compliance-enforcement-cm
    repoURL: 'git@github.com:bcgov-c/tenant-gitops-dc0a4a'
    targetRevision: update-helm-chart-18
  sources: []
  project: dc0a4a

Adding New Argo Users

BCGov's ArgoCD UI references a CRD for its permission set, in order to add new users the CRD must be editted to contain their github / gov email.

Example for viewing the resource contents: oc get gitopsteam/gitopsteam-compliance-enforcement-cm -n dc0a4a-tools -oyaml

You can use oc edit to directly modify the CRD and save it. We currently do not version control this CRD but plan to in the future.

Deploying New Infrastructure

This assumes you've already added the prerequisite GitHub action job changes to build and push tagged images to GHCR in your app repo

Currently we do not have a guardrails-protected way to stage infrastructure changes, although we will add this in the future

No guardrails implications:

  • If faulty templates are pushed to the tenant repo, ALL environments immediately detect them, and will enter a degraded status and stop updating until the templates are fixed
  • New infrastructure will immediately try to roll to ALL environments unless proper templating control flows are used. See https://helm.sh/docs/chart_template_guide/control_structures/
  • Argo reads from main as its ref, so all infra changes need to merge to main to have any effect, including PR-based environments

Until we have guardrails, here's the proposed flow:

newService:
  enabled: false
  deploymentStrategy: Recreate
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 1
    targetCPUUtilizationPercentage: 80
  #-- the service for the component. for inter namespace communication, use the service name as the hostname.
  service:
    type: ClusterIP

It is critical you update the first line of each template yaml file to refer to your new service's enabled flag, otherwise your workload will rollout across all environments, including prod

For example: change {{- if and .Values.global.enabled .Values.backend.enabled }} to {{- if and .Values.global.enabled .Values.newService.enabled }}

Ensure that these are updated FOR EACH template file in your new template folder

  • Before you commit your changes you should verify the templates are valid when populated by a values file, eg:
~/projects/tenant-gitops-dc0a4a/charts/apps$ helm template . --values values/.outputs/values-static.yaml
  • Helm will produce errors you need to fix, otherwise the resulting yaml will be produced to your terminal indicating your templates are valid

Note that helm template contains many flags you can use to experiment with the output of your templates, like the release name

Experimenting with deploying the new service can be done in 2 ways:

GitOps Tenant Repo Method

Argo Override Method

Or force your new service's enabled flag to true using Argo's value override feature. image

  • Click the left-most item, this represents the overall Argo Application configuration managing the dev environment
  • Click the Manifest tab in the drawer that opens
  • Under parameters add the following stanza:
      - name: newService.enabled
        value: true

Argo should now see the templates for your new service and roll it out.

Adding New Secrets

Currently without mounted vault secrets, our flow for secrets is obtuse. We will migrate to vault-managed secrets soon

New secret changes effect all environments at once, including prod UNLESS you add conditional if's around the new changes. Refer to the example in the adding infra section

Modify https://github.com/bcgov-c/tenant-gitops-dc0a4a/blob/main/charts/apps/templates/secret.yaml with your new secret value. If the value is supposed to be something like a unique password credential, set a placeholder string. This will be overridden in Argo.

Don't forget to verify the secret changes with helm template !

Before merging in your secret changes to main, prepare the override by opening the argo application, eg: https://gitops-shared.apps.emerald.devops.gov.bc.ca/applications/openshift-bcgov-gitops-shared/dc0a4a-compenf-dev-0?resource=, navigate to the Manifest tab in the drawer. You will see existing overrides for database passwords. Add your's in a similar fashion.

Currently, every environment needs to manually have its manifest editted to contain the new value overrides

Finally, merge in your secret changes. If everything goes smoothly you can gradually roll out the secret by adjusting the if conditional guards

Artifactory:

Artifactory is where are application images are stored. We need to store the images here because Argo can't access GHCR.

        docker pull --platform linux/amd64 artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement/frontend:latest
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement/backend:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement/database:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)
        docker pull --platform linux/amd64 [artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement/migrations:latest](http://artifacts.developer.gov.bc.ca/github-docker-remote/bcgov/nr-compliance-enforcement-cm/frontend:latest)

Install Openshift cli & Helm

For macOS and Windows you can use Homebrew or Chocolatey to install both helm and openshift commands

macOS

Install homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install openshift-cli

brew install helm

windows

Install Chocolatey:

https://chocolatey.org/install

choco install openshift-cli

choco install kubernetes-helm

Connecting to database in openshift

  1. Ensure you have oc on your path (can be downloaded here https://console.apps.silver.devops.gov.bc.ca/command-line-tools)
  2. Using the link above, click the "Copy login command" link to get the login token
  3. In a terminal window, paste the login command
  4. Select the project that namespace (e.g. oc project f208ae-test)
  5. Find the database pod by running the command oc get pods
  6. Port forward to the database pod oc port-forward {podname} 5433:5432 (note 5433 is arbitrary, this is the port that you're going to connect to locally)
  7. Use your database explorer of choice (I like DBeaver) to connect to the database. You can connect to the database using the host localhost, and the port 5433 (or whatever port you indicated on step 6 above).
  8. The password for the database can be found in Vault

Openshift Quotas

Since each PR can create an instance of Natsuite in dev, we may run into quota issues where we don't have enough resources to run each sandboxed instance of Natsuite.

Troubleshooting

If a deployment is failing, it's probably because the long-running quota has been exceeded. This is mostly a problem in the development environment which scales based on the number of open PR.

How to determine if there is a quota problem?

  1. Log into the OpenShift console as an administrator.
  2. Under the Administration section select ResourceQuotas.
  3. From the list select compute-long-running-quota.

image

  1. If anything is in the Yellow or the Red some attention is needed.

image

Strategies for freeing up resources

Look for unwanted running pods

  1. Log into the OpenShift console as an administrator.
  2. Under the Workloads section select Pods.
  3. Filter the list by Status to put the running pods to the top of the list.
  4. Make sure that all the pods that are running are desirable. For example, if there are any Pods relating to a closed pull request, this indicates that something didn't get cleaned up properly.

image

Temporarily remove pods for a Pull Request (Development only)

So you looked at the Pods, and you can't find any that don't belong. Unfortunately you are going to have to make a decision about who gets deployed and who doesn't. Examine the pull requests and determine which ones are the priority to be deployed. If there is one that is still a work in progress, it might be a good candidate to get removed from the dev environment.

  1. Log into the OpenShift console as an administrator.
  2. Under the Workloads section select DeploymentConfigs.
  3. Type the number of the Pull Request you want to remove in the search box
  4. For each item in the list under the ... menu select DeleteDeploymentConfig

image

This will delete all of the application Pods until either a new commit is issued for the Pull Request, or the deployment actions are re-run.

image

Removing the NATS Pods

The NATS container is special as it is deployed through Helm.

  1. Log into the OpenShift console as a Developer
  2. Select Helm
  3. Type the number of the Pull Request you want to remove in the search box
  4. From the ... menu select Delete Helm Release

image

To redeploy the NATS container, go into the pull Re-run the Deploy NATS to OpenShift action in the same fashion as described above

Requesting an increase of quota/limits

Requests for quota updates can be made here: https://registry.developer.gov.bc.ca/registry/user/dashboard/requests

More information is here: https://docs.developer.gov.bc.ca/request-quota-increase-for-openshift-project-set/

Setup notes

Steps Taken:

Create Artifactory Repo (refer to documentation above in Artifactory section)
Create ArgoCD to create github-c (gitops) repo for manifests (refer to documentation above in Argo section)
Update pipeline to push helm-charts to gitops repo (tenant-gitops-dc0a4a)
Create application in ArgoCD
Create network policy to allow traffic from all pods in TOOLS to bcgov.github.io

Clone this wiki locally