Merge pull request #77 from opencadc/kueue-dep

Kueue dep

Author: at88mph

helm/applications/canfar/README.md

@@ -1,4 +1,10 @@
-# CHANGELOG for Skaha User Session API (Chart 1.0.3)
+# CHANGELOG for Skaha User Session API (Chart 1.0.4)
+
+## 2025.09.11 (1.0.4)
+- Provide Kueue examples with documentation
+- Fix typo in headless priority class name setting
+- Fix environment variable name to properly be uppercase for headless priority class
+- Bump Skaha API image to `1.0.3`
 
 ## 2025.09.10 (1.0.3)
 - Fix to display GPU Cores

helm/applications/canfar/SkahaDependFlow.png

@@ -15,13 +15,13 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.0.3
+version: 1.0.4
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "1.0.2"
+appVersion: "1.0.3"
 
 dependencies:
   - name: "redis"

helm/applications/canfar/skaha.png

@@ -63,6 +63,7 @@ The following table lists the configurable parameters for the Skaha Helm chart:
 | `deployment.skaha.adminsGroup` | GMS style Group URI for Skaha admins to belong to | `""` |
 | `deployment.skaha.headlessGroup` | GMS style Group URI whose members can submit headless jobs | `""` |
 | `deployment.skaha.headlessPriorityGroup` | GMS style Group URI whose member's headless jobs can pre-empt other's.  Useful fortight deadlines in processing | `""` |
+| `deployment.skaha.headlessPriorityClass` | Name of the `priorityClass` for headless jobs to allow some pre-emption | `""` |
 | `deployment.skaha.loggingGroups` | List of GMS style Group URIs whose members can alter the log level.  See [cadc-log](https://github.com/opencadc/core/tree/main/cadc-log) regarding the `/logControl` endpoint. | `[]` |
 | `deployment.skaha.posixMapperResourceID` | Resource ID (URI) for the POSIX Mapper service containing the UIDs and GIDs | `""` |
 | `deployment.skaha.oidcURI` | URI (or URL) for the OIDC service | `""` |
@@ -84,10 +85,10 @@ The following table lists the configurable parameters for the Skaha Helm chart:
 | `deployment.skaha.sessions.minEphemeralStorage` | Minimum ephemeral storage, in [Kubernetes quantity](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/), for interactive sessions.  Defaults to 20Gi. | `"20Gi"` |
 | `deployment.skaha.sessions.maxEphemeralStorage` | Maximum ephemeral storage, in [Kubernetes quantity](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/), for interactive sessions.  Defaults to 200Gi. | `"200Gi"` |
 | `deployment.skaha.sessions.initContainerImage` | Init container image for Skaha User Sessions. | `redis-7.4.2-alpine3.21` |
-| `deployment.skaha.sessions.queue.default.queueName` | Name of the default `LocalQueue` instance from Kueue for all types | `""` |
-| `deployment.skaha.sessions.queue.default.priorityClass` | Name of the `priorityClass` for the all types to allow some pre-emption | `""` |
-| `deployment.skaha.sessions.queue.<typename>.queueName` | Name of the `LocalQueue` instance from Kueue for the given type | `""` |
-| `deployment.skaha.sessions.queue.<typename>.priorityClass` | Name of the `priorityClass` for the given type to allow some pre-emption | `""` |
+| `deployment.skaha.sessions.kueue.default.queueName` | Name of the default `LocalQueue` instance from Kueue for all types | `""` |
+| `deployment.skaha.sessions.kueue.default.priorityClass` | Name of the `priorityClass` for the all types to allow some pre-emption | `""` |
+| `deployment.skaha.sessions.kueue.<typename>.queueName` | Name of the `LocalQueue` instance from Kueue for the given type | `""` |
+| `deployment.skaha.sessions.kueue.<typename>.priorityClass` | Name of the `priorityClass` for the given type to allow some pre-emption | `""` |
 | `deployment.skaha.sessions.hostname` | Hostname to access user sessions on.  Defaults to `deployment.hostname` | `deployment.hostname` |
 | `deployment.skaha.sessions.tls` | TLS configuration for the User Sessions IngressRoute. | `{}` |
 | `deployment.skaha.sessions.extraVolumes` | List of extra `volume` and `volumeMount` to be mounted in User Sessions.  See the `values.yaml` file for examples. | `[]` |
@@ -106,46 +107,20 @@ Ensure that `tolerations` and `nodeAffinity` are at the expected indentation!  T
 ## Kueue
 Skaha leverages Kueue for efficient job queueing and management when properly installed and configured in your cluster. For detailed information on Kueue's features and setup, refer to the [Kueue documentation](https://kueue.sigs.k8s.io/docs/).
 
-Choosing to install Kueue:
-`values.yaml`
-```yaml
-kueue:
-  # Set to false by default
-  install: true
-```
+### Installation
+https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version
 
-Will install the Kueue Chart, with a default `ClusterQueue`, and whatever defined `LocalQueues` were declared in the `deployment.skaha.sessions.queue` section:
+Will install the Kueue Chart, with a default `ClusterQueue`, and whatever defined `LocalQueues` were declared in the `deployment.skaha.sessions.kueue` section:
 ```yaml
 deployment:
   skaha:
     sessions:
-      queue:
+      kueue:
         notebook:
           queueName: some-local-queue
           priorityClass: med
 ```
 
-In which case Helm would ensure the `some-local-queue` `LocalQueue` is installed.
-
-Kueue will also need to know about the Kubernetes Cluster configuration.  Setting the values to 60% to 80% of the cluster resources is recommended for optimal performance.
-```yaml
-kueue:
-  install: true
-  # 60% of cluster resources
-  clusterQueueResources:
-    - name: "cpu"
-      nominalQuota:   "28"
-      borrowingLimit: "0"
-      lendingLimit:   "0"
-    - name: "memory"
-      nominalQuota:   "100Gi"
-      borrowingLimit: "0Gi"
-      lendingLimit:   "0Gi"
-    - name: "ephemeral-storage"
-      nominalQuota:   "500Gi"
-      borrowingLimit: "0Gi"
-      lendingLimit:   "0Gi"
-```
 
 To determine your cluster's allocatable resources, checkout a small Python utility (requires [`uv`](https://github.com/astral-sh/uv?tab=readme-ov-file#installation)):
 https://github.com/opencadc/deployments/tree/main/configs/kueue/kueuer

helm/applications/skaha/CHANGELOG.md

@@ -0,0 +1,85 @@
+# Example Cluster Queue configuration for Kueue.  Usually a single ClusterQueue is sufficient, but read the documentation for more details.
+# @see https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
+#
+# Priority Classes are used in Skaha's configuration to allow Workloads to be prioritized.
+#
+# jenkinsd 2025.09.11
+#
+---
+# Resource Flavors
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ResourceFlavor
+metadata:
+  name: canfar-science-platform-default
+---
+# ClusterQueue
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ClusterQueue
+metadata:
+  name: canfar-science-platform-cluster-queue
+spec:
+  namespaceSelector:
+    matchExpressions:
+      - key: kubernetes.io/metadata.name
+        operator: In
+        values: [ skaha-workload ]
+  queueingStrategy: BestEffortFIFO
+  cohort: canfar-science-platform-cohort
+  resourceGroups:
+    - coveredResources:
+      - "cpu"
+      - "memory"
+      - "ephemeral-storage"
+      - "nvidia.com/gpu"
+      flavors:
+        - name: "canfar-science-platform-default"
+        # These values represent the total resources available to this ClusterQueue, and requires some knowledge of the cluster's capacity.
+        # Omit the nvidia.com/gpu resource if GPUs are not available in the cluster (also from the coveredResources above).
+          resources:
+            - name: "cpu"
+              nominalQuota:   "1680"
+              borrowingLimit: "420"
+              lendingLimit:   "420"
+            - name: "memory"
+              nominalQuota:   "6562Gi"
+              borrowingLimit: "1640Gi"
+              lendingLimit:   "1640Gi"
+            - name: "ephemeral-storage"
+              nominalQuota:   "52636Gi"
+              borrowingLimit: "13159Gi"
+              lendingLimit:   "13159Gi"
+            - name: "nvidia.com/gpu"
+              nominalQuota: "18"
+              borrowingLimit: "0"
+              lendingLimit: "0"
+  preemption:
+    reclaimWithinCohort: LowerPriority
+    borrowWithinCohort:
+      policy: LowerPriority
+      maxPriorityThreshold: 10000
+    withinClusterQueue: LowerPriority
+  stopPolicy: None
+---
+# WorkloadPriorityClass
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: WorkloadPriorityClass
+metadata:
+  name: low
+value: 10000
+description: "Low Priority"
+---
+# WorkloadPriorityClass
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: WorkloadPriorityClass
+metadata:
+  name: medium
+value: 100000
+description: "Medium Priority"
+---
+# WorkloadPriorityClass
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: WorkloadPriorityClass
+metadata:
+  name: high
+value: 1000000
+description: "High Priority"

helm/applications/skaha/Chart.yaml

@@ -0,0 +1,14 @@
+# Example Local Queue configuration for Kueue.  This assumes the skaha-workload namespace exists, and the referenced ClusterQueue exists.
+# This a single LocalQueue, but feel free to add as many as needed.
+# @see https://kueue.sigs.k8s.io/docs/concepts/local_queue/
+#
+# jenkinsd 2025.09.11
+#
+---
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: LocalQueue
+metadata:
+  namespace: skaha-workload
+  name: canfar-science-platform-local-queue
+spec:
+  clusterQueue: canfar-science-platform-cluster-queue

helm/applications/skaha/README.md

@@ -1,4 +1,3 @@
-{{- if eq .Values.kueue.install true }}
 # Grant the service account from the Released and Workload namespaces access to the Kueue resources
 #
 # Workload Namespace RBAC
@@ -7,8 +6,8 @@
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: {{ .Release.Name }}-kueue-role
-  namespace: {{ .Values.skahaWorkload.namespace }}
+  name: canfar-science-platform-kueue-role
+  namespace: skaha-workload
 rules:
 - apiGroups:
   - "kueue.x-k8s.io"
@@ -22,16 +21,16 @@ rules:
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
-  name: {{ .Release.Name }}-read-kueue
-  namespace: {{ .Values.skahaWorkload.namespace }}
+  name: canfar-science-platform-read-kueue
+  namespace: skaha-workload
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
-  name: {{ .Release.Name }}-kueue-role
+  name: canfar-science-platform-kueue-role
 subjects:
 - kind: ServiceAccount
-  name: {{ .Values.deployment.skaha.serviceAccountName }}
-  namespace: {{ .Release.Namespace }}
+  name: skaha
+  namespace: skaha-system
 
 #
 # Released Namespace RBAC
@@ -40,8 +39,8 @@ subjects:
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: {{ .Release.Name }}-kueue-role
-  namespace: {{ .Release.Namespace }}
+  name: canfar-science-platform-kueue-role
+  namespace: skaha-system
 rules:
 - apiGroups:
   - "kueue.x-k8s.io"
@@ -55,14 +54,13 @@ rules:
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
-  name: {{ .Release.Name }}-read-kueue
-  namespace: {{ .Release.Namespace }}
+  name: canfar-science-platform-read-kueue
+  namespace: skaha-system
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
-  name: {{ .Release.Name }}-kueue-role
+  name: canfar-science-platform-kueue-role
 subjects:
 - kind: ServiceAccount
-  name: {{ .Values.deployment.skaha.serviceAccountName }}
-  namespace: {{ .Release.Namespace }}
-{{- end }}
+  name: skaha
+  namespace: skaha-system

helm/applications/skaha/kueue/examples/clusterQueue.config.yaml

@@ -8,29 +8,26 @@ Namespace: {{ .Release.Namespace }}
 Skaha workload namespace: {{ .Values.skahaWorkload.namespace | default "skaha-workload" }}
 
 Kueue:
-{{- if .Values.kueue.install }}
-- Installed by this release: yes (chart {{ "kueue" }})
-- ClusterQueue: {{ .Release.Name }}-cluster-queue
-- LocalQueues (in {{ .Values.skahaWorkload.namespace }}):
-{{- $queues := (default dict .Values.deployment.skaha.sessions.queue) }}
+{{- if .Values.deployment.skaha.sessions.kueue }}
+  - It's expected that the Kueue CRDs and controllers to be present if you set session queues.
+{{- $queues := (default dict .Values.deployment.skaha.sessions.kueue) }}
 {{- if $queues }}
 {{- range $type, $cfg := $queues }}
   - {{ $cfg.queueName | default (printf "%s-%s" $.Release.Name $type) }} (type: {{ $type }})
 {{- end }}
 {{- else }}
-  - none configured; Jobs will run without Kueue unless annotated by Skaha.
+  - none configured; Jobs will NOT run with Kueue unless annotated by Skaha.
 {{- end }}
 {{- else }}
-- Installed by this release: no
-- Expect Kueue CRDs and controllers to be present if you set session queues.
+  - Not configured
 {{- end }}
 
 Quick checks:
-- kubectl get deployment {{ .Release.Name }}-skaha-tomcat -n {{ .Release.Namespace }}
-- kubectl get clusterqueue {{ .Release.Name }}-cluster-queue 2>/dev/null || true
-- kubectl get localqueue -n {{ .Values.skahaWorkload.namespace }} 2>/dev/null || true
+  - kubectl get deployment {{ .Release.Name }}-skaha-tomcat -n {{ .Release.Namespace }}
+  - kubectl get clusterqueues 2>/dev/null || true
+  - kubectl get localqueues -n {{ .Values.skahaWorkload.namespace }} 2>/dev/null || true
 
 If user Jobs remain Pending:
-- Ensure LocalQueue exists in {{ .Values.skahaWorkload.namespace }} and references ClusterQueue {{ .Release.Name }}-cluster-queue.
-- Verify ClusterQueue quotas (Values.kueue.clusterQueueResources) are sufficient.
+- Ensure LocalQueue exists in {{ .Values.skahaWorkload.namespace }} and references your configured ClusterQueue.
+- Verify ClusterQueue quotas are sufficient.
 - Confirm priority classes referenced by sessions exist or are created.