• Nutanix GitOps Examples
  • Getting Started
    • Option 1: Using Devbox NIX Shell
    • Using Devcontainer
    • Bootstrapping New NKE Cluster
      • Silent Bootstrap
  • Appendix
    • Directory Structure

Nutanix GitOps Examples

Getting Started

Option 1: Using Devbox NIX Shell

This project uses devbox to manage its development environment.

Install devbox and accept all defaults:

curl -fsSL https://get.jetpack.io/devbox | bash

Start the devbox shell and if nix isn't available, you will be prompted to install:

devbox shell

Using Devcontainer

See Devcontainer Tutorial on using Devcontainer.json - https://code.visualstudio.com/docs/devcontainers/tutorial

Bootstrapping New NKE Cluster

Silent Bootstrap

1. Set K8S_CLUSTER_NAME environment variable and copy ./.env.sample.yaml to ./.env.${K8S_CLUSTER_NAME}.yaml

   export K8S_CLUSTER_NAME=flux-kind-local
   cp ./.env.sample.yaml ./.env.${K8S_CLUSTER_NAME}.yaml

2. Update .env.${K8S_CLUSTER_NAME}.yaml with required values. If parameter is enabled, update required parameters below each section

   For example, if kube_vip.enabled is true, then uncomment and configure kube_vip.ipam_range:

     kube_vip:
       enabled: false
       ## min. 2 ips for range (i.e., 10.38.110.22-10.38.110.23)
       ipam_range: required

3. [Optional] Generate and Validate Configurations

   task bootstrap:generate_cluster_configs
   cat .local/${K8S_CLUSTER_NAME}/.env
   cat clusters/${K8S_CLUSTER_NAME}/platform/cluster-configs.yaml

4. [Optional] Validate Encrypted Secrets

   task sops:decrypt

5. Select New (or Switching to Existing) Cluster and Download NKE Creds

   eval $(task nke:switch-shell-env) && \
   task nke:download-creds && \
   kubectl get nodes

6. Run Flux Bootstrapping - task bootstrap:silent

   task bootstrap:silent

   NOTE: if there are any issues, troubleshot using task ts:flux-collect. You can re-run task bootstrap:silent as many times needed

7. Monitor on New Terminal

   eval $(task nke:switch-shell-env) && \
   task flux:watch

   NOTE: if there are any issues, update local git repo, push up changes and run task flux:reconcile

8. [Optional] Post Install - Taint GPU Nodepool with dedicated=gpu:NoSchedule

   if undesired workloads already running on gpu nodepools, drain nodes using task kubectl:drain_gpu_nodes

   ## taint gpu nodes with label nvidia.com/gpu.present=true
   task kubectl:taint_gpu_nodes
   
   ## view taint configurations on all nodes
   kubectl get nodes -o='custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect'

Appendix

Directory Structure

.
├── configs # Directory for env configs used by local scripts
│   ├── _common
│   │   └── .env # Global env configs 
│   └── ...
├── scripts # Common path for local scripts
├── clusters
│   ├── _profiles # Directory for the varying different profiles
│   │   ├── _base # Base for all cluster profiles (things installed in all variants)
│   │       └── kustomization.yaml # Kustomize ref to one or many platform _base/<service> and _components/feature
│   │   ├── llm-management # Management cluster profile (defines management specific applications and platform variants)
│   │   └── llm-workloads # Workloads cluster profile (defines workload specific applications and platform variants)
│   ├── nke-management-cluster # An example cluster instance generated from `_templates/management-cluster`
│   │   ├── flux-system # Generated by flux bootstrap
│   │   └── platform # generated from _templates
│   │       ├── kustomization.yaml # Maps to `management` profile above and injects secrets/config in the cluster
│   │       ├── cluster-secrets.sops.yaml
│   │       └── cluster-configs.yaml
│   └── nke-workload-cluster-00 # An example cluster instance generated from `_templates/[prod|non-prod]-workload-cluster`
└── platform # Contains catalogue of all the platform services
    ├── cert-manager
    ├── ingress-nginx # Example service
    │   ├── _base # Base implementation of this service     
    │       ├── kustomization.yaml # Kustomize ref to ingress-nginx.yaml
    │       └── ingress-nginx.yaml # helm chart release details for ingress-nginx
    │   └── nodeport # Feature to expose nginx in a NodePort instead of in a LoadBalancer
    │       ├── kustomization.yaml # Component ref to _base and ingress-nginx-patch
    │       └── ingress-nginx-patch.yaml # patch that overrides ingress-nginx helm chart
    └── ...