New way provision of provision has been introduced.

[alsmk]

Addressed issues:

• Add "Message of the day" warning SSH users that they are operating in a production environment #6361
• Kubernetes: Build provision workflow #9137

Implementation checklist:

• A new environment has been created with name test-e2e
• A new inventory file has been created with the same name of corresponding env
• Filtered ansible taks. Added some new. Deleted which are not neccessary
• Introduced new playbook for k8s-cluster setup
• Github secret has been added with name ssh-key
• Introduced a new workflow which will provision the entire cluster with some others setup
• Ran the workflow , Test the cluster working correctly
• Deploy OpenCRVS on the new cluster

Test results:

• ✅ Single node cluster provision: https://github.com/opencrvs/infrastructure/actions/runs/16720899863/job/47324901398
• ✅ Multinode cluster setup: https://github.com/opencrvs/infrastructure/actions/runs/16721378694/job/47326389033
• ✅ Adding node to existing cluster: https://github.com/opencrvs/infrastructure/actions/runs/16722389580/job/47329548509
• ✅ Self hosted runner is available: https://github.com/opencrvs/infrastructure/actions/runs/16722461413/job/47329778097
  

Test on clean environment:

• ✅ Provision: https://github.com/opencrvs/infrastructure/actions/runs/16758949937
• ✅ Deploy dependencies: https://github.com/opencrvs/infrastructure/actions/runs/16758141352
• ✅ Deploy OpenCRVS with data reset: https://github.com/opencrvs/infrastructure/actions/runs/16758657828

Final notes:

• List of required secrets:
  • ENCRYPTION_KEY, key to encrypt /data filesystem
  • K8S_RUNNER_TOKEN, token for github self-hosted runner
  • SSH_KEY, ssh key for provision workflow. Note, we have a plans to avoid direct connection from github to OpenCRVS servers (VMs)
  • DISK_SPACE (variable), /data/ partition size
  • DOMAIN (variable), public domain
• Optimize tags, not all ansible tags are needed

[adskyiproger]

Lets store variables to group_vars

[adskyiproger]

Store to group vars

[adskyiproger]

What if /etc/apt/sources.list.d/kubernetes.list exists?

[adskyiproger]

I like it

[adskyiproger]

Optional: Add metrics server: https://github.com/kubernetes-sigs/metrics-server

[euanmillar]

@adskyiproger @rikukissa if I am understanding this correctly, then the previous provision and deploy process is entirely replaced by Kubernetes in this repo. I thought that the strategy for this repo would be to maintain existing Docker Swarm deployment and provisioning whilst providing Kubernetes in parallel. We would maintian both until such time as we were confident that one could be replaced. Therefore I thought that there would be some sub folders and duplication e.g. workflows/k8s_provision.yml infrastructure/swarm, infrastructure/k8s

[rikukissa]

@euanmillar That was the original idea, but we quickly realised templating Docker Swarm isn’t worth it. It’s much less work to keep the current infrastructure setup in the country config and continue developing it there, while developing the Kubernetes setup here and making it configurable from the start.

[adskyiproger]

@euanmillar , Initial goal was to keep both solutions in the same repository. One repository to maintain will also work and there is an example in e2e repository. We have 2 deployment scripts coexisting (swarm, k8s).
Kubernetes solution is build using minimal set of tools (ansible, helm), you may find Tiltfile, but it will be hidden temporary in documentation.

Countryconfig template has few pitfalls that make moving infrastructure for docker swarm very tricky:

• All deployment logic on remote ssh from Github to target VM (push model). In general the way we do deployment today is ok, but we have cases like Philippines that doesn't work well with push model and on my opinion Philippines is a good example of infrastructure security. Kubernetes implementation was build with pull model in mind, all infrastructure could be hosted behind firewall and connection to github will be made by
• yarn is widely used to call incapsulated shell (bash) scripts, which creates confusion for newcomers, helm and kubectl allows to avoid writing shell, here are examples how can we make same stuff without scripting:
  • Rotate passwords: https://github.com/opencrvs/infrastructure/blob/develop/charts/opencrvs-services/templates/postgres-secrets.yaml#L12-L13
  • Issue SSL certificate and keys for auth, gateway: https://github.com/opencrvs/infrastructure/blob/develop/charts/opencrvs-services/templates/opencrvs-token-ssl.yaml#L1
  • Run additional jobs on deployment (migration): https://github.com/opencrvs/infrastructure/blob/develop/charts/opencrvs-services/templates/data-migration-job.yaml#L7
  • Run data reset as single helm command: https://github.com/opencrvs/infrastructure/blob/develop/charts/opencrvs-services/templates/data-cleanup-job.yaml
• Shell scripts have a lot of docker run commands, as a result we need to maintain both versions (swarm and kubernetes). In both cases scripts need to have connection to workloads, but the way to connect workload makes them incompatible.

Since we need to refactor all logic for kubernetes, it would be easier to maintain and keep code clean in separate repository.