Testing Advanced Node-Pool Scheduling

[MattiaSarti]

Context

According to the new spec, Charmed Kubeflow should support advanced node-pool scheduling capabilities to run on different node pools while respective several segregation and customization requirements.

What needs to get done

Test that the platform of Charmed Kubeflow and user Profiles' workloads can be successfully deployed on node pools of mixed hardware following the process outlined in the new spec for advanced node-pool scheduling.

Definition of Done

It is tested whether the platform of Charmed Kubeflow and user Profiles' workloads can be successfully deployed on node pools of mixed hardware following the process outlined in the new spec for advanced node-pool scheduling.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-8494.

This message was autogenerated

[MattiaSarti]

Testing Advanced Node-Pool Scheduling on AKS

Alternative 2 from the spec was tested, being it more relevant to scaling to zero than Alternative 1, and with Kubeflow-platform components on the same node pool (as all Kubeflow charms support only one CPU architecture, at the time of writing).

Setup

procedure

multipass launch --cpus 2 --memory 4G --disk 30G --name advanced-scheduling-tests
multipass shell advanced-scheduling-tests

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

JUJU_VERSION=3.6
JUJU_AGENT_VERSION=3.6.9
K8S_VERSION=1.32

CHARMED_KUBEFLOW_TRACK=1.10
CHARMED_KUBEFLOW_USERNAME=admin@example.com

CLUSTER_NAME=advanced-scheduling
RESOURCE_GROUP_NAME=advanced-scheduling-tests
RESOURCE_LOCATION=westeurope

# By default, the output of Azure CLI commands is printed to the stdout stream. Without redirecting the stdout stream, contents in it will be stored in the build log of the action. Configure Azure CLI to not show output in the console screen or print in the log by setting the environment variable AZURE_CORE_OUTPUT to none. If you need the output of a specific command, override the default setting using the argument --output with your format of choice. For more information on output options with the Azure CLI, see Format output: https://learn.microsoft.com/cli/azure/format-output-azure-cli
# AZURE_CORE_OUTPUT=none

sudo apt install -y pipx
pipx install tox
pipx ensurepath

sudo snap install charmcraft --classic
sudo snap install juju --channel=${JUJU_VERSION}/stable
sudo snap install kubectl --classic
sudo snap install terraform  --channel latest/stable --classic

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

az login  # NOTE: select a subscription registered for the provider 'Microsoft.ContainerService'

az group create --name ${RESOURCE_GROUP_NAME} --location ${RESOURCE_LOCATION}

Procedure Testing

Step 1: Label and Taint Your Node Pools

procedure

# node pool for the control plane of K8s:
az aks create \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --name ${CLUSTER_NAME} \
    --kubernetes-version ${K8S_VERSION} \
    --node-count 2 \
    --node-vm-size Standard_D8s_v3 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --no-ssh-key \
    --nodepool-name "k8s" \
    --nodepool-labels "k8s-control-plane=true" \
    --nodepool-taints "CriticalAddonsOnly=true:NoSchedule"

# node pool for Juju's system:
az aks nodepool add \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --cluster-name ${CLUSTER_NAME} \
    --node-count 1 \
    --node-vm-size Standard_D4s_v3 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --nodepool-name "juju"

# node pool (one single CPU architecture) for the Kubeflow platform:
az aks nodepool add \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --cluster-name ${CLUSTER_NAME} \
    --node-count 1 \
    --node-vm-size Standard_D8s_v3 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --nodepool-name "kubeflow" \
    --labels "kubeflow-platform=true" \
    --node-taints "kubeflow-platform=true:NoSchedule"

# default node pools for Kubeflow users' workloads:
az aks nodepool add \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --cluster-name ${CLUSTER_NAME} \
    --node-count 1 \
    --node-vm-size Standard_D2s_v3 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --nodepool-name "a" \
    --labels "kubeflow-default-node-pool=a" \
    --node-taints "kubeflow-default-node-pool=a:NoSchedule"
az aks nodepool add \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --cluster-name ${CLUSTER_NAME} \
    --node-count 1 \
    --node-vm-size Standard_D2s_v3 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --nodepool-name "b" \
    --labels "kubeflow-default-node-pool=b" \
    --node-taints "kubeflow-default-node-pool=b:NoSchedule"

# specialized node pool (with special hardware) for Kubeflow users' workloads:
az aks nodepool add \
    --resource-group ${RESOURCE_GROUP_NAME} \
    --cluster-name ${CLUSTER_NAME} \
    --node-count 1 \
    --node-vm-size Standard_D2pls_v6 \
    --node-osdisk-size 100 \
    --node-osdisk-type Managed \
    --os-sku Ubuntu \
    --nodepool-name "x" \
    --labels "special-hardware=x" \
    --node-taints "special-hardware=x:NoSchedule"
# for ARM64 nodes, see https://learn.microsoft.com/en-us/azure/aks/use-arm64-vms#add-a-node-pool-with-an-arm64-vm

results

ubuntu@advanced-scheduling-tests:~$ kubectl get nodes 
NAME                               STATUS   ROLES    AGE   VERSION
aks-a-33719789-vmss000000          Ready    <none>   35h   v1.32.10
aks-b-75544582-vmss000000          Ready    <none>   35h   v1.32.10
aks-juju-38601234-vmss000000       Ready    <none>   35h   v1.32.10
aks-k8s-27176324-vmss000000        Ready    <none>   35h   v1.32.10
aks-k8s-27176324-vmss000001        Ready    <none>   35h   v1.32.10
aks-kubeflow-38183491-vmss000000   Ready    <none>   35h   v1.32.10
aks-x-76501254-vmss000000          Ready    <none>   35h   v1.32.10
ubuntu@advanced-scheduling-tests:~$ kubectl get nodes -o json | jq '.items[] | "name:", .metadata.name, "labels:", .metadata.labels, "taints:", .spec.taints, "- - - - - - - - - - - - - - - - - - - -"'
"name:"
"aks-a-33719789-vmss000000"
"labels:"
{
  "agentpool": "a",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D2s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "kubeflow-default-node-pool": "a",
  "kubernetes.azure.com/agentpool": "a",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "2ac3bac2-22a0-11f1-97be-aacfd6716d69",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "2",
  "kubernetes.azure.com/sku-memory": "8192",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-a-33719789-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D2s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "kubeflow-default-node-pool",
    "value": "a"
  }
]
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-b-75544582-vmss000000"
"labels:"
{
  "agentpool": "b",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D2s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "kubeflow-default-node-pool": "b",
  "kubernetes.azure.com/agentpool": "b",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "f9337083-22a0-11f1-8f6a-6a4cdf39e179",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "2",
  "kubernetes.azure.com/sku-memory": "8192",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-b-75544582-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D2s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "kubeflow-default-node-pool",
    "value": "b"
  }
]
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-juju-38601234-vmss000000"
"labels:"
{
  "agentpool": "juju",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D4s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "kubernetes.azure.com/agentpool": "juju",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "7e74cacc-229e-11f1-afd4-1255ff2c4b2f",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "4",
  "kubernetes.azure.com/sku-memory": "16384",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-juju-38601234-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D4s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
null
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-k8s-27176324-vmss000000"
"labels:"
{
  "agentpool": "k8s",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D8s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "k8s-control-plane": "true",
  "kubernetes.azure.com/agentpool": "k8s",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "b98036df-229d-11f1-a1e6-b6861d0c250e",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "system",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "8",
  "kubernetes.azure.com/sku-memory": "32768",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-k8s-27176324-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D8s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "CriticalAddonsOnly",
    "value": "true"
  }
]
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-k8s-27176324-vmss000001"
"labels:"
{
  "agentpool": "k8s",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D8s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "k8s-control-plane": "true",
  "kubernetes.azure.com/agentpool": "k8s",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "b98036df-229d-11f1-a1e6-b6861d0c250e",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "system",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "8",
  "kubernetes.azure.com/sku-memory": "32768",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-k8s-27176324-vmss000001",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D8s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "CriticalAddonsOnly",
    "value": "true"
  }
]
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-kubeflow-38183491-vmss000000"
"labels:"
{
  "agentpool": "kubeflow",
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "Standard_D8s_v3",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "kubeflow-platform": "true",
  "kubernetes.azure.com/agentpool": "kubeflow",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "ef494811-229e-11f1-8f6a-6a4cdf39e179",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "8",
  "kubernetes.azure.com/sku-memory": "32768",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "aks-kubeflow-38183491-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D8s_v3",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "kubeflow-platform",
    "value": "true"
  }
]
"- - - - - - - - - - - - - - - - - - - -"
"name:"
"aks-x-76501254-vmss000000"
"labels:"
{
  "agentpool": "x",
  "beta.kubernetes.io/arch": "arm64",
  "beta.kubernetes.io/instance-type": "Standard_D2pls_v6",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "westeurope",
  "failure-domain.beta.kubernetes.io/zone": "0",
  "kubernetes.azure.com/agentpool": "x",
  "kubernetes.azure.com/azure-cni-overlay": "true",
  "kubernetes.azure.com/cluster": "MC_advanced-scheduling-tests_advanced-scheduling_westeurope",
  "kubernetes.azure.com/consolidated-additional-properties": "573704fe-22a1-11f1-bf36-0ec3ea146428",
  "kubernetes.azure.com/kubelet-identity-client-id": "d20c94a0-005e-4af4-ac80-f4e9eaa04246",
  "kubernetes.azure.com/kubelet-serving-ca": "cluster",
  "kubernetes.azure.com/localdns-state": "disabled",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/network-name": "aks-vnet-37824556",
  "kubernetes.azure.com/network-resourcegroup": "advanced-scheduling-tests",
  "kubernetes.azure.com/network-stateless-cni": "false",
  "kubernetes.azure.com/network-subnet": "aks-subnet",
  "kubernetes.azure.com/network-subscription": "1807b644-9585-4279-91f2-c953a29762e9",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2arm64containerd-202602.13.5",
  "kubernetes.azure.com/nodenetwork-vnetguid": "e0197699-22f8-4eeb-a548-1445eaee8a17",
  "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/os-sku-effective": "Ubuntu2204",
  "kubernetes.azure.com/os-sku-requested": "Ubuntu",
  "kubernetes.azure.com/podnetwork-type": "overlay",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/sku-cpu": "2",
  "kubernetes.azure.com/sku-memory": "4096",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "arm64",
  "kubernetes.io/hostname": "aks-x-76501254-vmss000000",
  "kubernetes.io/os": "linux",
  "node.kubernetes.io/instance-type": "Standard_D2pls_v6",
  "special-hardware": "x",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "topology.disk.csi.azure.com/zone": "",
  "topology.kubernetes.io/region": "westeurope",
  "topology.kubernetes.io/zone": "0"
}
"taints:"
[
  {
    "effect": "NoSchedule",
    "key": "special-hardware",
    "value": "x"
  }
]
"- - - - - - - - - - - - - - - - - - - -"

Step 2: Set Up Your Juju Controller

procedure

az aks get-credentials --resource-group ${RESOURCE_GROUP_NAME} --name ${CLUSTER_NAME} --admin

juju add-k8s testing-on-aks --client
juju bootstrap testing-on-aks testing-on-aks --agent-version ${JUJU_AGENT_VERSION}

Step 3: Set Up a Former namespace-node-affinity-operator

procedure

juju add-model temp-namespace-node-affinity
juju switch temp-namespace-node-affinity

juju deploy --trust --channel 2.2/stable --revision 248 namespace-node-affinity temp-namespace-node-affinity
juju wait-for application temp-namespace-node-affinity

namespace_node_affinity_settings=$(cat << EOF
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
EOF
)
juju config temp-namespace-node-affinity settings_yaml="$namespace_node_affinity_settings"

results

ubuntu@advanced-scheduling-tests:~$ juju status
Model                         Controller      Cloud/Region               Version  SLA          Timestamp
temp-namespace-node-affinity  testing-on-aks  testing-on-aks/westeurope  3.6.9    unsupported  21:09:50+01:00

App                           Version  Status  Scale  Charm                    Channel     Rev  Address       Exposed  Message
temp-namespace-node-affinity           active      1  namespace-node-affinity  2.2/stable  248  10.0.105.199  no       

Unit                             Workload  Agent  Address       Ports  Message
temp-namespace-node-affinity/0*  active    idle   10.244.2.136         
ubuntu@advanced-scheduling-tests:~$ juju config temp-namespace-node-affinity settings_yaml
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists

Step 4: Create the Kubeflow-Platform Juju Model

procedure

juju add-model kubeflow

Step 5: Label the Kubeflow-Platform Model’s Namespace(s)

procedure

kubectl label namespaces kubeflow namespace-node-affinity=enabled

kubectl create namespace knative-eventing
kubectl label namespaces knative-eventing namespace-node-affinity=enabled

kubectl create namespace knative-serving
kubectl label namespaces knative-serving namespace-node-affinity=enabled

Step 6: Set Up a Latter namespace-node-affinity-operator

procedure

juju switch kubeflow

juju deploy --trust --channel 2.2/stable --revision 248 namespace-node-affinity namespace-node-affinity
juju wait-for application namespace-node-affinity

namespace_node_affinity_settings=$(cat << EOF
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-eventing: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-serving: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
EOF
)
juju config namespace-node-affinity settings_yaml="$namespace_node_affinity_settings"

results

ubuntu@advanced-scheduling-tests:~$ juju status
Model     Controller      Cloud/Region               Version  SLA          Timestamp
kubeflow  testing-on-aks  testing-on-aks/westeurope  3.6.9    unsupported  21:12:30+01:00

App                      Version  Status  Scale  Charm                    Channel     Rev  Address     Exposed  Message
namespace-node-affinity           active      1  namespace-node-affinity  2.2/stable  248  10.0.35.48  no       

Unit                        Workload  Agent  Address       Ports  Message
namespace-node-affinity/0*  active    idle   10.244.3.123         
ubuntu@advanced-scheduling-tests:~$ juju config namespace-node-affinity settings_yaml
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-eventing: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-serving: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists

Step 7: Tear Down the Former namespace-node-affinity-operator

procedure

juju switch temp-namespace-node-affinity

juju remove-application --destroy-storage --no-prompt temp-namespace-node-affinity
juju destroy-model --destroy-storage --no-prompt temp-namespace-node-affinity

juju switch kubeflow

results

NAMESPACE                   NAME                                                   READY   STATUS    RESTARTS       AGE     IP             NODE                               NOMINATED NODE   READINESS GATES
controller-testing-on-aks   controller-0                                           3/3     Running   12 (33h ago)   35h     10.244.2.106   aks-juju-38601234-vmss000000       <none>           <none>
controller-testing-on-aks   modeloperator-57d54668f6-5qqhb                         1/1     Running   0              35h     10.244.2.37    aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 azure-cns-52tz6                                        1/1     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 azure-cns-7pktx                                        1/1     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 azure-cns-fwkhs                                        1/1     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 azure-cns-gpc74                                        1/1     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 azure-cns-lc9p6                                        1/1     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 azure-cns-qjskv                                        1/1     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 azure-cns-vdrcf                                        1/1     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 azure-ip-masq-agent-24rv7                              1/1     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 azure-ip-masq-agent-2pqmb                              1/1     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 azure-ip-masq-agent-9ktrb                              1/1     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 azure-ip-masq-agent-mxfmh                              1/1     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 azure-ip-masq-agent-smrbc                              1/1     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 azure-ip-masq-agent-vpd92                              1/1     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 azure-ip-masq-agent-wkd7g                              1/1     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-4nm5n                               1/1     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-7lbpv                               1/1     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-9hqkm                               1/1     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 cloud-node-manager-ljmh9                               1/1     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 cloud-node-manager-mtrqt                               1/1     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 cloud-node-manager-z6mfs                               1/1     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-zdd4r                               1/1     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 coredns-5f6c7445ff-f957b                               1/1     Running   0              36h     10.244.0.56    aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 coredns-5f6c7445ff-fs4d8                               1/1     Running   0              36h     10.244.1.13    aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 coredns-autoscaler-849f5d766b-87v8j                    1/1     Running   0              36h     10.244.0.167   aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azuredisk-node-5n56x                               3/3     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 csi-azuredisk-node-k648p                               3/3     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-kbkvf                               3/3     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-pcwsz                               3/3     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azuredisk-node-sfxtb                               3/3     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-tfrrv                               3/3     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 csi-azuredisk-node-xg5d4                               3/3     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 csi-azurefile-node-5m75v                               3/3     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 csi-azurefile-node-ckdff                               3/3     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 csi-azurefile-node-jf9t7                               3/3     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 csi-azurefile-node-n6whg                               3/3     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 csi-azurefile-node-q2cfl                               3/3     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azurefile-node-tcr7l                               3/3     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 csi-azurefile-node-x4985                               3/3     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 konnectivity-agent-5b4f9f88cc-8xnzx                    1/1     Running   0              36h     10.244.0.132   aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 konnectivity-agent-5b4f9f88cc-c2995                    1/1     Running   0              36h     10.244.1.38    aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 konnectivity-agent-autoscaler-776c759bd8-sf4nb         1/1     Running   0              36h     10.244.0.84    aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 kube-proxy-29j75                                       1/1     Running   0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 kube-proxy-2chbp                                       1/1     Running   0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 kube-proxy-6r7n7                                       1/1     Running   0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 kube-proxy-bswbr                                       1/1     Running   0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 kube-proxy-f7s8j                                       1/1     Running   0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 kube-proxy-g5dph                                       1/1     Running   0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 kube-proxy-h486w                                       1/1     Running   0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 metrics-server-748c846f5-4fndh                         2/2     Running   0              36h     10.244.1.244   aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 metrics-server-748c846f5-snhj8                         2/2     Running   0              36h     10.244.1.107   aks-k8s-27176324-vmss000000        <none>           <none>
kubeflow                    modeloperator-749987c848-nkcn2                         1/1     Running   0              3m36s   10.244.2.173   aks-juju-38601234-vmss000000       <none>           <none>
kubeflow                    namespace-node-affinity-0                              1/1     Running   0              2m36s   10.244.3.123   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    namespace-node-affinity-pod-webhook-5dd86d986b-5j5td   1/1     Running   0              118s    10.244.3.211   aks-kubeflow-38183491-vmss000000   <none>           <none>

Step 8: Deploy Charmed Kubeflow

procedure

git clone https://github.com/canonical/charmed-kubeflow-solutions.git /home/ubuntu/charmed-kubeflow-solutions
git -C /home/ubuntu/charmed-kubeflow-solutions checkout track/${CHARMED_KUBEFLOW_TRACK}
cd /home/ubuntu/charmed-kubeflow-solutions/modules/kubeflow-spark
terraform init
# NOTE: edge risk and revision for Profile controller to include required changes:
terraform apply -auto-approve \
  -var create_model=false \
  -var risk=edge \
  -var knative_eventing_revision=964 \
  -var knative_serving_revision=963 \
  -var kubeflow_profiles_revision=839 \
  -var dex_static_username=${CHARMED_KUBEFLOW_USERNAME} \
  -var dex_static_password=admin

results

ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ juju status
Model     Controller      Cloud/Region               Version  SLA          Timestamp
kubeflow  testing-on-aks  testing-on-aks/westeurope  3.6.9    unsupported  21:26:42+01:00

App                      Version                  Status  Scale  Charm                      Channel         Rev  Address       Exposed  Message
admission-webhook                                 active      1  admission-webhook          1.10/edge       593  10.0.252.173  no       
argo-controller                                   active      1  argo-controller            3.5/edge        894  10.0.218.144  no       
dex-auth                                          active      1  dex-auth                   2.41/edge       806  10.0.34.49    no       
envoy                                             active      1  envoy                      2.4/edge        533  10.0.153.14   no       
integration-hub                                   active      1  spark-integration-hub-k8s  3/edge          115  10.0.198.82   no       
istio-ingressgateway                              active      1  istio-gateway              1.24/edge      1546  10.0.67.28    no       
istio-pilot                                       active      1  istio-pilot                1.24/edge      1486  10.0.161.207  no       
jupyter-controller                                active      1  jupyter-controller         1.10/edge      1468  10.0.83.124   no       
jupyter-ui                                        active      1  jupyter-ui                 1.10/edge      1391  10.0.126.210  no       
katib-controller                                  active      1  katib-controller           0.18/edge      1262  10.0.178.113  no       
katib-db                 8.0.44-0ubuntu0.22.04.1  active      3  mysql-k8s                  8.0/stable      346  10.0.142.234  no       
katib-db-manager                                  active      1  katib-db-manager           0.18/edge      1222  10.0.21.123   no       
katib-ui                                          active      1  katib-ui                   0.18/edge      1225  10.0.121.32   no       
kfp-api                                           active      1  kfp-api                    2.5/edge       2435  10.0.43.9     no       
kfp-db                   8.0.44-0ubuntu0.22.04.1  active      3  mysql-k8s                  8.0/stable      346  10.0.234.204  no       
kfp-metadata-writer                               active      1  kfp-metadata-writer        2.5/edge       1502  10.0.115.93   no       
kfp-persistence                                   active      1  kfp-persistence            2.5/edge       2444  10.0.55.4     no       
kfp-profile-controller                            active      1  kfp-profile-controller     2.5/edge       2406  10.0.253.85   no       
kfp-schedwf                                       active      1  kfp-schedwf                2.5/edge       2455  10.0.164.10   no       
kfp-ui                                            active      1  kfp-ui                     2.5/edge       2449  10.0.7.121    no       
kfp-viewer                                        active      1  kfp-viewer                 2.5/edge       2476  10.0.190.110  no       
kfp-viz                                           active      1  kfp-viz                    2.5/edge       2391  10.0.157.198  no       
knative-eventing                                  active      1  knative-eventing           1.16/edge       964  10.0.157.32   no       
knative-operator                                  active      1  knative-operator           1.16/edge       926  10.0.11.20    no       
knative-serving                                   active      1  knative-serving            1.16/edge       963  10.0.41.82    no       
kserve-controller                                 active      1  kserve-controller          0.14/edge      1119  10.0.73.194   no       
kubeflow-dashboard                                active      1  kubeflow-dashboard         1.10/edge       948  10.0.190.40   no       
kubeflow-integrator                               active      1  data-kubeflow-integrator   1/edge           26  10.0.15.253   no       
kubeflow-profiles                                 active      1  kubeflow-profiles          1.10/edge       839  10.0.114.253  no       
kubeflow-roles                                    active      1  kubeflow-roles             1.10/edge       405  10.0.94.195   no       
kubeflow-volumes                                  active      1  kubeflow-volumes           1.10/edge       585  10.0.27.14    no       
metacontroller-operator                           active      1  metacontroller-operator    4.11/edge       565  10.0.207.252  no       
minio                                             active      1  minio                      1.10/edge       637  10.0.133.81   no       
mlmd                                              active      1  mlmd                       ckf-1.10/edge   425  10.0.249.39   no       
namespace-node-affinity                           active      1  namespace-node-affinity    2.2/stable      248  10.0.35.48    no       
oidc-gatekeeper                                   active      1  oidc-gatekeeper            ckf-1.10/edge   615  10.0.59.220   no       
pvcviewer-operator                                active      1  pvcviewer-operator         1.10/edge       445  10.0.27.20    no       
resource-dispatcher                               active      1  resource-dispatcher        latest/edge     546  10.0.138.84   no       
tensorboard-controller                            active      1  tensorboard-controller     1.10/edge       659  10.0.144.215  no       
tensorboards-web-app                              active      1  tensorboards-web-app       1.10/edge       644  10.0.153.141  no       
training-operator                                 active      1  training-operator          1.9/edge        714  10.0.254.114  no       

Unit                        Workload  Agent  Address       Ports  Message
admission-webhook/0*        active    idle   10.244.3.208         
argo-controller/0*          active    idle   10.244.3.231         
dex-auth/0*                 active    idle   10.244.3.2           
envoy/0*                    active    idle   10.244.3.35          
integration-hub/0*          active    idle   10.244.3.177         
istio-ingressgateway/0*     active    idle   10.244.3.99          
istio-pilot/0*              active    idle   10.244.3.77          
jupyter-controller/0*       active    idle   10.244.3.46          
jupyter-ui/0*               active    idle   10.244.3.3           
katib-controller/0*         active    idle   10.244.3.9           
katib-db-manager/0*         active    idle   10.244.3.67          
katib-db/0                  active    idle   10.244.3.29          
katib-db/1                  active    idle   10.244.3.162         
katib-db/2*                 active    idle   10.244.3.225         Primary
katib-ui/0*                 active    idle   10.244.3.43          
kfp-api/0*                  active    idle   10.244.3.96          
kfp-db/0*                   active    idle   10.244.3.72          Primary
kfp-db/1                    active    idle   10.244.3.125         
kfp-db/2                    active    idle   10.244.3.101         
kfp-metadata-writer/0*      active    idle   10.244.3.48          
kfp-persistence/0*          active    idle   10.244.3.133         
kfp-profile-controller/0*   active    idle   10.244.3.174         
kfp-schedwf/0*              active    idle   10.244.3.68          
kfp-ui/0*                   active    idle   10.244.3.11          
kfp-viewer/0*               active    idle   10.244.3.122         
kfp-viz/0*                  active    idle   10.244.3.146         
knative-eventing/0*         active    idle   10.244.3.250         
knative-operator/0*         active    idle   10.244.3.203         
knative-serving/0*          active    idle   10.244.3.34          
kserve-controller/0*        active    idle   10.244.3.100         
kubeflow-dashboard/0*       active    idle   10.244.3.97          
kubeflow-integrator/0*      active    idle   10.244.3.134         
kubeflow-profiles/0*        active    idle   10.244.3.155         
kubeflow-roles/0*           active    idle   10.244.3.220         
kubeflow-volumes/0*         active    idle   10.244.3.25          
metacontroller-operator/0*  active    idle   10.244.3.64          
minio/0*                    active    idle   10.244.3.200         
mlmd/0*                     active    idle   10.244.3.32          
namespace-node-affinity/0*  active    idle   10.244.3.123         
oidc-gatekeeper/0*          active    idle   10.244.3.69          
pvcviewer-operator/0*       active    idle   10.244.3.219         
resource-dispatcher/0*      active    idle   10.244.3.182         
tensorboard-controller/0*   active    idle   10.244.3.15          
tensorboards-web-app/0*     active    idle   10.244.3.44          
training-operator/0*        active    idle   10.244.3.159
ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ kubectl get pods -A -o wide
NAMESPACE                   NAME                                                       READY   STATUS      RESTARTS       AGE     IP             NODE                               NOMINATED NODE   READINESS GATES
controller-testing-on-aks   controller-0                                               3/3     Running     12 (34h ago)   35h     10.244.2.106   aks-juju-38601234-vmss000000       <none>           <none>
controller-testing-on-aks   modeloperator-57d54668f6-5qqhb                             1/1     Running     0              35h     10.244.2.37    aks-juju-38601234-vmss000000       <none>           <none>
knative-eventing            eventing-controller-55b958475b-w7vkn                       1/1     Running     0              3m9s    10.244.3.66    aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            eventing-webhook-cbd594f5d-zxqjl                           1/1     Running     0              3m7s    10.244.3.224   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            imc-controller-6995655795-tkzcv                            1/1     Running     0              3m5s    10.244.3.180   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            imc-dispatcher-cb8f6cd56-cghwl                             1/1     Running     0              3m4s    10.244.3.27    aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            job-sink-bd6dc7f66-7q2rr                                   1/1     Running     0              3m8s    10.244.3.140   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            mt-broker-controller-7995bb765c-zj45j                      1/1     Running     0              3m2s    10.244.3.195   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            mt-broker-filter-6d5687557d-br7nx                          1/1     Running     0              3m3s    10.244.3.106   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            mt-broker-ingress-7bfff9f8b9-66jhv                         1/1     Running     0              3m3s    10.244.3.240   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-eventing            storage-version-migration-eventing-eventing-1.16.0-srrtr   0/1     Completed   0              3m2s    10.244.3.98    aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             activator-69c465664d-68l62                                 1/1     Running     0              96s     10.244.3.181   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             autoscaler-7b88489f7d-bzgbm                                1/1     Running     0              96s     10.244.3.6     aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             autoscaler-hpa-58fc746885-xftdm                            1/1     Running     0              94s     10.244.3.117   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             cleanup-serving-serving-1.16.0-7kn55                       0/1     Completed   0              93s     10.244.3.230   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             controller-bb6844d5f-cqclw                                 1/1     Running     0              95s     10.244.3.121   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             net-istio-controller-9698b78f7-vvmgv                       1/1     Running     0              92s     10.244.3.149   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             net-istio-webhook-6f7c8b849f-5686b                         1/1     Running     0              92s     10.244.3.207   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             storage-version-migration-serving-serving-1.16.0-xdzrw     1/1     Running     0              93s     10.244.3.173   aks-kubeflow-38183491-vmss000000   <none>           <none>
knative-serving             webhook-695c7f4845-5sptg                                   1/1     Running     0              94s     10.244.3.255   aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 azure-cns-52tz6                                            1/1     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 azure-cns-7pktx                                            1/1     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 azure-cns-fwkhs                                            1/1     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 azure-cns-gpc74                                            1/1     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 azure-cns-lc9p6                                            1/1     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 azure-cns-qjskv                                            1/1     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 azure-cns-vdrcf                                            1/1     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 azure-ip-masq-agent-24rv7                                  1/1     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 azure-ip-masq-agent-2pqmb                                  1/1     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 azure-ip-masq-agent-9ktrb                                  1/1     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 azure-ip-masq-agent-mxfmh                                  1/1     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 azure-ip-masq-agent-smrbc                                  1/1     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 azure-ip-masq-agent-vpd92                                  1/1     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 azure-ip-masq-agent-wkd7g                                  1/1     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-4nm5n                                   1/1     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-7lbpv                                   1/1     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-9hqkm                                   1/1     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 cloud-node-manager-ljmh9                                   1/1     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 cloud-node-manager-mtrqt                                   1/1     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 cloud-node-manager-z6mfs                                   1/1     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 cloud-node-manager-zdd4r                                   1/1     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 coredns-5f6c7445ff-f957b                                   1/1     Running     0              36h     10.244.0.56    aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 coredns-5f6c7445ff-fs4d8                                   1/1     Running     0              36h     10.244.1.13    aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 coredns-autoscaler-849f5d766b-87v8j                        1/1     Running     0              36h     10.244.0.167   aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azuredisk-node-5n56x                                   3/3     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 csi-azuredisk-node-k648p                                   3/3     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-kbkvf                                   3/3     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-pcwsz                                   3/3     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azuredisk-node-sfxtb                                   3/3     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 csi-azuredisk-node-tfrrv                                   3/3     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 csi-azuredisk-node-xg5d4                                   3/3     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 csi-azurefile-node-5m75v                                   3/3     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 csi-azurefile-node-ckdff                                   3/3     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 csi-azurefile-node-jf9t7                                   3/3     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 csi-azurefile-node-n6whg                                   3/3     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 csi-azurefile-node-q2cfl                                   3/3     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 csi-azurefile-node-tcr7l                                   3/3     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 csi-azurefile-node-x4985                                   3/3     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 konnectivity-agent-5b4f9f88cc-8xnzx                        1/1     Running     0              36h     10.244.0.132   aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 konnectivity-agent-5b4f9f88cc-c2995                        1/1     Running     0              36h     10.244.1.38    aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 konnectivity-agent-autoscaler-776c759bd8-sf4nb             1/1     Running     0              36h     10.244.0.84    aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 kube-proxy-29j75                                           1/1     Running     0              36h     10.224.0.4     aks-k8s-27176324-vmss000001        <none>           <none>
kube-system                 kube-proxy-2chbp                                           1/1     Running     0              36h     10.224.0.8     aks-a-33719789-vmss000000          <none>           <none>
kube-system                 kube-proxy-6r7n7                                           1/1     Running     0              36h     10.224.0.10    aks-x-76501254-vmss000000          <none>           <none>
kube-system                 kube-proxy-bswbr                                           1/1     Running     0              36h     10.224.0.6     aks-juju-38601234-vmss000000       <none>           <none>
kube-system                 kube-proxy-f7s8j                                           1/1     Running     0              36h     10.224.0.5     aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 kube-proxy-g5dph                                           1/1     Running     0              36h     10.224.0.7     aks-kubeflow-38183491-vmss000000   <none>           <none>
kube-system                 kube-proxy-h486w                                           1/1     Running     0              36h     10.224.0.9     aks-b-75544582-vmss000000          <none>           <none>
kube-system                 metrics-server-748c846f5-4fndh                             2/2     Running     0              36h     10.244.1.244   aks-k8s-27176324-vmss000000        <none>           <none>
kube-system                 metrics-server-748c846f5-snhj8                             2/2     Running     0              36h     10.244.1.107   aks-k8s-27176324-vmss000000        <none>           <none>
kubeflow                    admission-webhook-0                                        2/2     Running     0              8m24s   10.244.3.208   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    argo-controller-0                                          2/2     Running     0              8m42s   10.244.3.231   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    dex-auth-0                                                 2/2     Running     0              8m26s   10.244.3.2     aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    envoy-0                                                    2/2     Running     0              7m43s   10.244.3.35    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    integration-hub-0                                          2/2     Running     0              8m33s   10.244.3.177   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    istio-ingressgateway-0                                     1/1     Running     0              9m51s   10.244.3.99    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    istio-ingressgateway-workload-6bf99b9bf4-4xz8h             1/1     Running     0              8m17s   10.244.3.52    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    istio-pilot-0                                              1/1     Running     0              9m57s   10.244.3.77    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    istiod-5bc66d7479-lf27n                                    1/1     Running     0              8m40s   10.244.3.49    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    jupyter-controller-0                                       2/2     Running     0              8m31s   10.244.3.46    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    jupyter-ui-0                                               2/2     Running     0              7m57s   10.244.3.3     aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-controller-0                                         2/2     Running     0              7m31s   10.244.3.9     aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-db-0                                                 2/2     Running     0              9m8s    10.244.3.29    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-db-1                                                 2/2     Running     0              9m8s    10.244.3.162   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-db-2                                                 2/2     Running     0              9m8s    10.244.3.225   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-db-manager-0                                         2/2     Running     0              9m33s   10.244.3.67    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    katib-ui-0                                                 2/2     Running     0              10m     10.244.3.43    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-api-0                                                  2/2     Running     0              9m3s    10.244.3.96    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-db-0                                                   2/2     Running     0              9m5s    10.244.3.72    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-db-1                                                   2/2     Running     0              9m5s    10.244.3.125   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-db-2                                                   2/2     Running     0              9m5s    10.244.3.101   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-metadata-writer-0                                      2/2     Running     0              9m20s   10.244.3.48    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-persistence-0                                          2/2     Running     0              9m8s    10.244.3.133   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-profile-controller-0                                   2/2     Running     0              9m18s   10.244.3.174   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-schedwf-0                                              2/2     Running     0              8m2s    10.244.3.68    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-ui-0                                                   2/2     Running     0              8m21s   10.244.3.11    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-viewer-0                                               2/2     Running     0              8m56s   10.244.3.122   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kfp-viz-0                                                  2/2     Running     0              8m13s   10.244.3.146   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    knative-eventing-0                                         1/1     Running     0              9m22s   10.244.3.250   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    knative-operator-0                                         3/3     Running     0              8m21s   10.244.3.203   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    knative-serving-0                                          1/1     Running     0              9m      10.244.3.34    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kserve-controller-0                                        2/2     Running     0              9m28s   10.244.3.100   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kubeflow-dashboard-0                                       2/2     Running     0              8m9s    10.244.3.97    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kubeflow-integrator-0                                      1/1     Running     0              9m2s    10.244.3.134   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kubeflow-profiles-0                                        3/3     Running     0              9m41s   10.244.3.155   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kubeflow-roles-0                                           1/1     Running     0              9m31s   10.244.3.220   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    kubeflow-volumes-0                                         2/2     Running     0              8m45s   10.244.3.25    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    metacontroller-operator-0                                  1/1     Running     0              9m20s   10.244.3.64    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    metacontroller-operator-charm-0                            1/1     Running     0              8m12s   10.244.3.19    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    minio-0                                                    2/2     Running     0              8m53s   10.244.3.200   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    mlmd-0                                                     2/2     Running     0              8m18s   10.244.3.32    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    modeloperator-749987c848-nkcn2                             1/1     Running     0              16m     10.244.2.173   aks-juju-38601234-vmss000000       <none>           <none>
kubeflow                    namespace-node-affinity-0                                  1/1     Running     0              15m     10.244.3.123   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    namespace-node-affinity-pod-webhook-5dd86d986b-5j5td       1/1     Running     0              15m     10.244.3.211   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    oidc-gatekeeper-0                                          2/2     Running     0              8m51s   10.244.3.69    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    pvcviewer-operator-0                                       2/2     Running     0              8m49s   10.244.3.219   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    resource-dispatcher-0                                      2/2     Running     0              7m23s   10.244.3.182   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    tensorboard-controller-0                                   2/2     Running     0              7m37s   10.244.3.15    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    tensorboards-web-app-0                                     2/2     Running     0              10m     10.244.3.44    aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    training-operator-0                                        1/1     Running     0              8m46s   10.244.3.159   aks-kubeflow-38183491-vmss000000   <none>           <none>
kubeflow                    training-operator-6f8d557494-9wb52                         1/1     Running     0              5m37s   10.244.3.137   aks-kubeflow-38183491-vmss000000   <none>           <none>

Step 9: (Re)Configure Your Kubeflow Profiles’ Default Node Pools

procedure

namespace_node_affinity_settings=$(cat << EOF
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-eventing: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-serving: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
profile-i: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - a
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: a
profile-j: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - a
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: a
profile-k: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - b
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: b
EOF
)
juju config namespace-node-affinity settings_yaml="$namespace_node_affinity_settings"

results

ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ juju status namespace-node-affinity
Model     Controller      Cloud/Region               Version  SLA          Timestamp
kubeflow  testing-on-aks  testing-on-aks/westeurope  3.6.9    unsupported  21:30:19+01:00

App                      Version  Status  Scale  Charm                    Channel     Rev  Address     Exposed  Message
namespace-node-affinity           active      1  namespace-node-affinity  2.2/stable  248  10.0.35.48  no       

Unit                        Workload  Agent  Address       Ports  Message
namespace-node-affinity/0*  active    idle   10.244.3.123         
ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ juju config namespace-node-affinity settings_yaml
kubeflow: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-eventing: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
knative-serving: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-platform
        operator: Exists
  tolerations:
    - effect: NoSchedule
      key: kubeflow-platform
      operator: Exists
profile-i: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - a
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: a
profile-j: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - a
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: a
profile-k: |
  excludedLabels:
    exclude-me-from-namespace-node-affinity-operator: "true"
  nodeSelectorTerms:
    - matchExpressions:
      - key: kubeflow-default-node-pool
        operator: In
        values:
          - b
  tolerations:
    - effect: NoSchedule
      key: kubeflow-default-node-pool
      operator: Equal
      value: b

Step 10: Create Some (New) Kubeflow Profiles

procedure

for profile_name in profile-i profile-j profile-k;
do
kubectl apply -f - << EOF
apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: $profile_name
spec:
  owner:
    kind: User
    name: ${CHARMED_KUBEFLOW_USERNAME}
EOF
done

results

ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ kubectl get profiles.kubeflow.org
NAME        AGE
profile-i   17s
profile-j   16s
profile-k   16s
ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-solutions/modules/kubeflow-spark$ kubectl get namespaces --show-labels profile-i profile-j profile-k
NAME        STATUS   AGE   LABELS
profile-i   Active   46s   app.kubernetes.io/part-of=kubeflow-profile,istio-injection=enabled,katib-metricscollector-injection=enabled,katib.kubeflow.org/metrics-collector-injection=enabled,kubernetes.io/metadata.name=profile-i,namespace-node-affinity=enabled,pipelines.kubeflow.org/enabled=true,pod-security.kubernetes.io/enforce=privileged,serving.kubeflow.org/inferenceservice=enabled,user.kubeflow.org/enabled=true
profile-j   Active   43s   app.kubernetes.io/part-of=kubeflow-profile,istio-injection=enabled,katib-metricscollector-injection=enabled,katib.kubeflow.org/metrics-collector-injection=enabled,kubernetes.io/metadata.name=profile-j,namespace-node-affinity=enabled,pipelines.kubeflow.org/enabled=true,pod-security.kubernetes.io/enforce=privileged,serving.kubeflow.org/inferenceservice=enabled,user.kubeflow.org/enabled=true
profile-k   Active   43s   app.kubernetes.io/part-of=kubeflow-profile,istio-injection=enabled,katib-metricscollector-injection=enabled,katib.kubeflow.org/metrics-collector-injection=enabled,kubernetes.io/metadata.name=profile-k,namespace-node-affinity=enabled,pipelines.kubeflow.org/enabled=true,pod-security.kubernetes.io/enforce=privileged,serving.kubeflow.org/inferenceservice=enabled,user.kubeflow.org/enabled=true
ubuntu@advanced-scheduling-tests:~$ kubectl get pods -o wide -n profile-i
NAME                                               READY   STATUS    RESTARTS   AGE   IP             NODE                        NOMINATED NODE   READINESS GATES
ml-pipeline-ui-artifact-6d575b7484-l7k64           2/2     Running   0          8h    10.244.4.100   aks-a-33719789-vmss000000   <none>           <none>
ml-pipeline-visualizationserver-68fd78454f-58jcc   2/2     Running   0          8h    10.244.4.118   aks-a-33719789-vmss000000   <none>           <none>

Step 11: Deploy Some (New) Profiles’ Workloads…

…To Their Respective Profiles’ Node Pools

procedure

From the Dashbard's UI (at the external IP of the K8s Service istio-ingressgateway-workload), after logging in with the default static Dex identity, create a Notebook in any of the Profiles already associated with such an identity, to see the Notebook scheduled in the respective default node pool for such a Profile.

results

ubuntu@advanced-scheduling-tests:~$ kubectl get pods -o wide -n profile-i
NAME                                               READY   STATUS    RESTARTS   AGE   IP             NODE                        NOMINATED NODE   READINESS GATES
ml-pipeline-ui-artifact-6d575b7484-l7k64           2/2     Running   0          8h    10.244.4.100   aks-a-33719789-vmss000000   <none>           <none>
ml-pipeline-visualizationserver-68fd78454f-58jcc   2/2     Running   0          8h    10.244.4.118   aks-a-33719789-vmss000000   <none>           <none>
notebook-in-default-profile-node-pool-0            2/2     Running   0          27s   10.244.4.56    aks-a-33719789-vmss000000   <none>           <none>
ubuntu@advanced-scheduling-tests:~$ kubectl get pods -o wide -n profile-k
NAME                                               READY   STATUS    RESTARTS   AGE     IP             NODE                        NOMINATED NODE   READINESS GATES
ml-pipeline-ui-artifact-6d575b7484-jv8lm           2/2     Running   0          11h     10.244.5.117   aks-b-75544582-vmss000000   <none>           <none>
ml-pipeline-visualizationserver-68fd78454f-48d8j   2/2     Running   0          11h     10.244.5.60    aks-b-75544582-vmss000000   <none>           <none>
notebook-in-default-profile-node-pool-0            2/2     Running   0          2m41s   10.244.5.153   aks-b-75544582-vmss000000   <none>           <none>

…To Some Other, Arbitrary Node Pools

procedure

affinity_options=$(cat << EOF
- configKey: special_hardware_x
  displayName: "Special Hardware X"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: special-hardware
                operator: In
                values: [x]
EOF
)
tolerations_options=$(cat << EOF
- groupKey: special_hardware_x
  displayName: "Special Hardware X"
  tolerations:
    - key: special-hardware
      operator: Equal
      value: x
      effect: NoSchedule
EOF
)
juju config jupyter-ui affinity-options="$affinity_options"
juju config jupyter-ui tolerations-options="$tolerations_options"

for profile_name in profile-i profile-j profile-k;
do
kubectl apply -f - << EOF
apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: disable-namespace-node-affinity-operator
  namespace: $profile_name
spec:
  desc: "Disable namespace-node-affinity-operator"
  selector:
    matchLabels:
      disable-namespace-node-affinity-operator: "true"
  labels:
    exclude-me-from-namespace-node-affinity-operator: "true"
EOF
done

From the Dashbard's UI (at the external IP of the K8s Service istio-ingressgateway-workload), after logging in with the default static Dex identity, create a Notebook in any of the Profiles already associated with such an identity while using the custom image kubeflownotebookswg/jupyter-scipy:v1.10.0-rc.1 and selecting "Special Hardware X" for "Affinity", "Tolerations" and "Configurations" (under "Advanced Options"), to see the Notebook scheduled in the node pool for such special hardware.

results

ubuntu@advanced-scheduling-tests:~$ kubectl get pods -o wide -n profile-i
NAME                                               READY   STATUS    RESTARTS   AGE   IP             NODE                        NOMINATED NODE   READINESS GATES
ml-pipeline-ui-artifact-6d575b7484-l7k64           2/2     Running   0          8h    10.244.4.100   aks-a-33719789-vmss000000   <none>           <none>
ml-pipeline-visualizationserver-68fd78454f-58jcc   2/2     Running   0          8h    10.244.4.118   aks-a-33719789-vmss000000   <none>           <none>
notebook-in-special-hardware-node-pool-0           2/2     Running   0          32s   10.244.6.83    aks-x-76501254-vmss000000   <none>           <none>

User-Acceptance Testing

procedure

git clone https://github.com/canonical/charmed-kubeflow-uats.git /home/ubuntu/charmed-kubeflow-uats
git -C /home/ubuntu/charmed-kubeflow-uats checkout track/${CHARMED_KUBEFLOW_TRACK}
cd /home/ubuntu/charmed-kubeflow-uats
tox -e kubeflow-remote -- --bundle ""

results

ubuntu@advanced-scheduling-tests:~/charmed-kubeflow-uats$ tox -e kubeflow-remote -- --bundle ""
kubeflow-remote: commands[0]> poetry install --no-root
Installing dependencies from lock file

No dependencies to install or update
kubeflow-remote: commands[1]> poetry run pytest -vv --tb native /home/ubuntu/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow and not feast' --model kubeflow --bundle ''
==================================================================== test session starts =====================================================================
platform linux -- Python 3.12.3, pytest-8.4.0, pluggy-1.6.0 -- /home/ubuntu/charmed-kubeflow-uats/.tox/kubeflow-remote/bin/python
cachedir: .tox/kubeflow-remote/.pytest_cache
rootdir: /home/ubuntu/charmed-kubeflow-uats
configfile: pyproject.toml
plugins: asyncio-0.21.2, anyio-4.9.0, operator-0.42.0, dependency-0.6.0
asyncio: mode=Mode.STRICT
collected 3 items                                                                                                                                            

driver/test_kubeflow_workloads.py::test_bundle_correctness 
----------------------------------------------------------------------- live log setup -----------------------------------------------------------------------
WARNING  juju.client.connection:connection.py:858 unexpected facade SSHServer received from the controller
INFO     pytest_operator.plugin:plugin.py:795 Connecting to existing model testing-on-aks:kubeflow on unspecified cloud
WARNING  juju.client.connection:connection.py:858 unexpected facade SSHServer received from the controller
WARNING  juju.client.connection:connection.py:858 unexpected facade SSHSession received from the controller
WARNING  juju.client.connection:connection.py:858 unexpected facade SSHTunneler received from the controller
WARNING  juju.client.connection:connection.py:858 unexpected facade SSHServer received from the controller
SKIPPED (charm_list empty. Cannot test bundle correctness)
driver/test_kubeflow_workloads.py::test_create_profile 
----------------------------------------------------------------------- live log setup -----------------------------------------------------------------------
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:164 Creating Profile test-kubeflow...
INFO     httpx:_client.py:1025 HTTP Request: POST https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles "HTTP/1.1 201 Created"
----------------------------------------------------------------------- live log call ------------------------------------------------------------------------
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/api/v1/namespaces/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:48 Waiting for namespace test-kubeflow to become 'Active': phase == Active
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1alpha1/namespaces/test-kubeflow/poddefaults/access-ml-pipeline "HTTP/1.1 404 Not Found"
INFO     utils:utils.py:70 Waiting for PodDefault access-ml-pipeline to be created in namespace test-kubeflow..
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1alpha1/namespaces/test-kubeflow/poddefaults/access-ml-pipeline "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:285 Sleeping for 10s to allow the creation of PodDefaults in test-kubeflow namespace..
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1alpha1/namespaces/test-kubeflow/poddefaults "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:295 PodDefaults in test-kubeflow namespace are ['access-ml-pipeline', 'pyspark-notebook', 'pyspark-pipeline'].
PASSED
driver/test_kubeflow_workloads.py::test_kubeflow_workloads 
----------------------------------------------------------------------- live log setup -----------------------------------------------------------------------
INFO     utils:utils.py:179 Adding security-policy PodDefault...
INFO     httpx:_client.py:1025 HTTP Request: POST https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1alpha1/namespaces/test-kubeflow/poddefaults "HTTP/1.1 201 Created"
----------------------------------------------------------------------- live log call ------------------------------------------------------------------------
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:324 Starting Kubernetes Job test-kubeflow/test-kubeflow to run notebook tests...
INFO     httpx:_client.py:1025 HTTP Request: POST https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs "HTTP/1.1 201 Created"
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:78 Retrying in 2 seconds (attempts: 1)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 4 seconds (attempts: 2)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 8 seconds (attempts: 3)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 16 seconds (attempts: 4)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 5)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 6)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 7)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 8)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 9)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 10)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 11)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 12)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 13)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 14)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 15)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 16)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 17)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 18)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 19)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 20)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:114 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:78 Retrying in 32 seconds (attempts: 21)
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:107 Job test-kubeflow/test-kubeflow completed successfully!
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:352 Fetching Job logs...
##### git-sync initContainer logs #####
INFO: detected pid 1, running init handler
{"logger":"","ts":"2026-03-19 20:33:37.287227","caller":{"file":"main.go","line":722},"level":0,"msg":"starting up","pid":12,"uid":65533,"gid":65533,"home":"/tmp","flags":["--add-user=false","--change-permissions=0","--cookie-file=false","--depth=1","--exechook-backoff=3s","--exechook-timeout=30s","--git=git","--git-gc=always","--group-write=true","--help=false","--http-metrics=false","--http-pprof=false","--link=charmed-kubeflow-uats","--man=false","--max-failures=0","--max-sync-failures=0","--one-time=true","--password=REDACTED","--period=10s","--ref=2cb2a05ddcf367881e8347f466fcc18b6f581f8d","--repo=https://github.com/canonical/charmed-kubeflow-uats","--root=/tests","--ssh=false","--ssh-key-file=/etc/git-secret/ssh","--ssh-known-hosts=true","--ssh-known-hosts-file=/etc/git-secret/known_hosts","--stale-worktree-timeout=0s","--submodules=recursive","--sync-timeout=2m0s","--timeout=0","--v=-1","--verbose=0","--version=false","--wait=0","--webhook-backoff=3s","--webhook-method=POST","--webhook-success-status=200","--webhook-timeout=1s"]}
{"logger":"","ts":"2026-03-19 20:33:37.301389","caller":{"file":"main.go","line":1248},"level":0,"msg":"repo directory failed checks or was empty","path":"/tests"}
{"logger":"","ts":"2026-03-19 20:33:37.301468","caller":{"file":"main.go","line":1258},"level":0,"msg":"initializing repo directory","path":"/tests"}
{"logger":"","ts":"2026-03-19 20:33:37.697770","caller":{"file":"main.go","line":1797},"level":0,"msg":"update required","ref":"2cb2a05ddcf367881e8347f466fcc18b6f581f8d","local":"","remote":"2cb2a05ddcf367881e8347f466fcc18b6f581f8d","syncCount":0}
{"logger":"","ts":"2026-03-19 20:33:38.272176","caller":{"file":"main.go","line":1848},"level":0,"msg":"updated successfully","ref":"2cb2a05ddcf367881e8347f466fcc18b6f581f8d","remote":"2cb2a05ddcf367881e8347f466fcc18b6f581f8d","syncCount":1}
{"logger":"","ts":"2026-03-19 20:33:38.272591","caller":{"file":"main.go","line":1022},"level":0,"msg":"exiting after one sync","status":0}
##### test-kubeflow container logs #####
============================= test session starts ==============================
platform linux -- Python 3.11.11, pytest-9.0.2, pluggy-1.5.0 -- /opt/conda/bin/python3.11
cachedir: .pytest_cache
rootdir: /tests/.worktrees/2cb2a05ddcf367881e8347f466fcc18b6f581f8d/tests
configfile: pytest.ini
plugins: anyio-4.8.0
collecting ... collected 10 items / 5 deselected / 5 selected

test_notebooks.py::test_notebook[katib-integration] 
-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:55 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU, use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:62 Note that Kubeflow Trainer V2 tests will not be run. In order to run tests for Kubeflow Trainer V2, use the `--include-kubeflow-trainer-tests` flag e.g. `tox -e kubeflow-local -- --include-kubeflow-trainer-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:68 Running katib-integration.ipynb...
PASSED                                                                   [ 20%]
test_notebooks.py::test_notebook[kfp-v2-integration] 
-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:55 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU, use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:62 Note that Kubeflow Trainer V2 tests will not be run. In order to run tests for Kubeflow Trainer V2, use the `--include-kubeflow-trainer-tests` flag e.g. `tox -e kubeflow-local -- --include-kubeflow-trainer-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:68 Running kfp-v2-integration.ipynb...
PASSED                                                                   [ 40%]
test_notebooks.py::test_notebook[kserve-integration] 
-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:55 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU, use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:62 Note that Kubeflow Trainer V2 tests will not be run. In order to run tests for Kubeflow Trainer V2, use the `--include-kubeflow-trainer-tests` flag e.g. `tox -e kubeflow-local -- --include-kubeflow-trainer-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:68 Running kserve-integration.ipynb...
PASSED                                                                   [ 60%]
test_notebooks.py::test_notebook[test-pod-security-standards] 
-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:55 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU, use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:62 Note that Kubeflow Trainer V2 tests will not be run. In order to run tests for Kubeflow Trainer V2, use the `--include-kubeflow-trainer-tests` flag e.g. `tox -e kubeflow-local -- --include-kubeflow-trainer-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:68 Running test-pod-security-standards.ipynb...
PASSED                                                                   [ 80%]
test_notebooks.py::test_notebook[training-integration] 
-------------------------------- live log call ---------------------------------
INFO     test_notebooks:test_notebooks.py:55 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU, use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:62 Note that Kubeflow Trainer V2 tests will not be run. In order to run tests for Kubeflow Trainer V2, use the `--include-kubeflow-trainer-tests` flag e.g. `tox -e kubeflow-local -- --include-kubeflow-trainer-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO     test_notebooks:test_notebooks.py:68 Running training-integration.ipynb...
PASSED                                                                   [100%]

================= 5 passed, 5 deselected in 535.49s (0:08:55) ==================
PASSED
--------------------------------------------------------------------- live log teardown ----------------------------------------------------------------------
INFO     utils:utils.py:191 Deleting security-policy PodDefault...
INFO     httpx:_client.py:1025 HTTP Request: DELETE https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1alpha1/namespaces/test-kubeflow/poddefaults/security-policy "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:177 Deleting Profile test-kubeflow...
INFO     httpx:_client.py:1025 HTTP Request: DELETE https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow?propagationPolicy=Foreground "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:utils.py:152 Waiting for Profile test-kubeflow to be deleted..
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:utils.py:152 Waiting for Profile test-kubeflow to be deleted..
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:utils.py:152 Waiting for Profile test-kubeflow to be deleted..
INFO     httpx:_client.py:1025 HTTP Request: GET https://advanced-s-advanced-schedul-1807b6-uq4cveou.hcp.westeurope.azmk8s.io/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 404 Not Found"
INFO     test_kubeflow_workloads:utils.py:152 Waiting for Profile test-kubeflow to be deleted..
INFO     pytest_operator.plugin:plugin.py:937 Model status:

Model     Controller      Cloud/Region               Version  SLA          Timestamp
kubeflow  testing-on-aks  testing-on-aks/westeurope  3.6.9    unsupported  21:43:34+01:00

App                      Version                  Status  Scale  Charm                      Channel         Rev  Address       Exposed  Message
admission-webhook                                 active      1  admission-webhook          1.10/edge       593  10.0.252.173  no       
argo-controller                                   active      1  argo-controller            3.5/edge        894  10.0.218.144  no       
dex-auth                                          active      1  dex-auth                   2.41/edge       806  10.0.34.49    no       
envoy                                             active      1  envoy                      2.4/edge        533  10.0.153.14   no       
integration-hub                                   active      1  spark-integration-hub-k8s  3/edge          115  10.0.198.82   no       
istio-ingressgateway                              active      1  istio-gateway              1.24/edge      1546  10.0.67.28    no       
istio-pilot                                       active      1  istio-pilot                1.24/edge      1486  10.0.161.207  no       
jupyter-controller                                active      1  jupyter-controller         1.10/edge      1468  10.0.83.124   no       
jupyter-ui                                        active      1  jupyter-ui                 1.10/edge      1391  10.0.126.210  no       
katib-controller                                  active      1  katib-controller           0.18/edge      1262  10.0.178.113  no       
katib-db                 8.0.44-0ubuntu0.22.04.1  active      3  mysql-k8s                  8.0/stable      346  10.0.142.234  no       
katib-db-manager                                  active      1  katib-db-manager           0.18/edge      1222  10.0.21.123   no       
katib-ui                                          active      1  katib-ui                   0.18/edge      1225  10.0.121.32   no       
kfp-api                                           active      1  kfp-api                    2.5/edge       2435  10.0.43.9     no       
kfp-db                   8.0.44-0ubuntu0.22.04.1  active      3  mysql-k8s                  8.0/stable      346  10.0.234.204  no       
kfp-metadata-writer                               active      1  kfp-metadata-writer        2.5/edge       1502  10.0.115.93   no       
kfp-persistence                                   active      1  kfp-persistence            2.5/edge       2444  10.0.55.4     no       
kfp-profile-controller                            active      1  kfp-profile-controller     2.5/edge       2406  10.0.253.85   no       
kfp-schedwf                                       active      1  kfp-schedwf                2.5/edge       2455  10.0.164.10   no       
kfp-ui                                            active      1  kfp-ui                     2.5/edge       2449  10.0.7.121    no       
kfp-viewer                                        active      1  kfp-viewer                 2.5/edge       2476  10.0.190.110  no       
kfp-viz                                           active      1  kfp-viz                    2.5/edge       2391  10.0.157.198  no       
knative-eventing                                  active      1  knative-eventing           1.16/edge       964  10.0.157.32   no       
knative-operator                                  active      1  knative-operator           1.16/edge       926  10.0.11.20    no       
knative-serving                                   active      1  knative-serving            1.16/edge       963  10.0.41.82    no       
kserve-controller                                 active      1  kserve-controller          0.14/edge      1119  10.0.73.194   no       
kubeflow-dashboard                                active      1  kubeflow-dashboard         1.10/edge       948  10.0.190.40   no       
kubeflow-integrator                               active      1  data-kubeflow-integrator   1/edge           26  10.0.15.253   no       
kubeflow-profiles                                 active      1  kubeflow-profiles          1.10/edge       839  10.0.114.253  no       
kubeflow-roles                                    active      1  kubeflow-roles             1.10/edge       405  10.0.94.195   no       
kubeflow-volumes                                  active      1  kubeflow-volumes           1.10/edge       585  10.0.27.14    no       
metacontroller-operator                           active      1  metacontroller-operator    4.11/edge       565  10.0.207.252  no       
minio                                             active      1  minio                      1.10/edge       637  10.0.133.81   no       
mlmd                                              active      1  mlmd                       ckf-1.10/edge   425  10.0.249.39   no       
namespace-node-affinity                           active      1  namespace-node-affinity    2.2/stable      248  10.0.35.48    no       
oidc-gatekeeper                                   active      1  oidc-gatekeeper            ckf-1.10/edge   615  10.0.59.220   no       
pvcviewer-operator                                active      1  pvcviewer-operator         1.10/edge       445  10.0.27.20    no       
resource-dispatcher                               active      1  resource-dispatcher        latest/edge     546  10.0.138.84   no       
tensorboard-controller                            active      1  tensorboard-controller     1.10/edge       659  10.0.144.215  no       
tensorboards-web-app                              active      1  tensorboards-web-app       1.10/edge       644  10.0.153.141  no       
training-operator                                 active      1  training-operator          1.9/edge        714  10.0.254.114  no       

Unit                        Workload  Agent  Address       Ports  Message
admission-webhook/0*        active    idle   10.244.3.208         
argo-controller/0*          active    idle   10.244.3.231         
dex-auth/0*                 active    idle   10.244.3.2           
envoy/0*                    active    idle   10.244.3.35          
integration-hub/0*          active    idle   10.244.3.177         
istio-ingressgateway/0*     active    idle   10.244.3.99          
istio-pilot/0*              active    idle   10.244.3.77          
jupyter-controller/0*       active    idle   10.244.3.46          
jupyter-ui/0*               active    idle   10.244.3.3           
katib-controller/0*         active    idle   10.244.3.9           
katib-db-manager/0*         active    idle   10.244.3.67          
katib-db/0                  active    idle   10.244.3.29          
katib-db/1                  active    idle   10.244.3.162         
katib-db/2*                 active    idle   10.244.3.225         Primary
katib-ui/0*                 active    idle   10.244.3.43          
kfp-api/0*                  active    idle   10.244.3.96          
kfp-db/0*                   active    idle   10.244.3.72          Primary
kfp-db/1                    active    idle   10.244.3.125         
kfp-db/2                    active    idle   10.244.3.101         
kfp-metadata-writer/0*      active    idle   10.244.3.48          
kfp-persistence/0*          active    idle   10.244.3.133         
kfp-profile-controller/0*   active    idle   10.244.3.174         
kfp-schedwf/0*              active    idle   10.244.3.68          
kfp-ui/0*                   active    idle   10.244.3.11          
kfp-viewer/0*               active    idle   10.244.3.122         
kfp-viz/0*                  active    idle   10.244.3.146         
knative-eventing/0*         active    idle   10.244.3.250         
knative-operator/0*         active    idle   10.244.3.203         
knative-serving/0*          active    idle   10.244.3.34          
kserve-controller/0*        active    idle   10.244.3.100         
kubeflow-dashboard/0*       active    idle   10.244.3.97          
kubeflow-integrator/0*      active    idle   10.244.3.134         
kubeflow-profiles/0*        active    idle   10.244.3.155         
kubeflow-roles/0*           active    idle   10.244.3.220         
kubeflow-volumes/0*         active    idle   10.244.3.25          
metacontroller-operator/0*  active    idle   10.244.3.64          
minio/0*                    active    idle   10.244.3.200         
mlmd/0*                     active    idle   10.244.3.32          
namespace-node-affinity/0*  active    idle   10.244.3.123         
oidc-gatekeeper/0*          active    idle   10.244.3.69          
pvcviewer-operator/0*       active    idle   10.244.3.219         
resource-dispatcher/0*      active    idle   10.244.3.182         
tensorboard-controller/0*   active    idle   10.244.3.15          
tensorboards-web-app/0*     active    idle   10.244.3.44          
training-operator/0*        active    idle   10.244.3.159         

INFO     pytest_operator.plugin:plugin.py:943 Juju error logs:

unit-tensorboards-web-app-0: 21:17:17 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kubeflow-profiles-0: 21:17:40 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kubeflow-roles-0: 21:17:50 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-knative-eventing-0: 21:17:59 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-metacontroller-operator-0: 21:18:02 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-knative-serving-0: 21:18:22 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-pvcviewer-operator-0: 21:18:38 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-oidc-gatekeeper-0: 21:18:38 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-training-operator-0: 21:18:43 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kubeflow-volumes-0: 21:18:45 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-jupyter-controller-0: 21:18:56 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-istio-pilot-0: 21:19:02 ERROR unit.istio-pilot/0.juju-log ingress-auth:21: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1198, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 496, in main
    manager.run()
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 480, in run
    self._emit()
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 415, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 459, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 336, in reconcile
    self._send_gateway_info()
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 633, in _send_gateway_info
    gateway_up=self._is_gateway_up,
               ^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 823, in _is_gateway_up
    return self._is_gateway_service_up and self._is_gateway_object_up
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 814, in _is_gateway_service_up
    if _get_gateway_address_from_svc(svc) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1052, in _get_gateway_address_from_svc
    gateway_address = _get_address_from_loadbalancer(svc)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1067, in _get_address_from_loadbalancer
    if len(ingresses) != 1:
       ^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
unit-istio-pilot-0: 21:19:02 ERROR juju.worker.uniter.operation hook "ingress-auth-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-dex-auth-0: 21:19:05 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-admission-webhook-0: 21:19:07 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-istio-pilot-0: 21:19:09 ERROR unit.istio-pilot/0.juju-log ingress-auth:21: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1198, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 494, in main
    manager = _Manager(charm_class, use_juju_for_storage=use_juju_for_storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 313, in __init__
    self.charm = self._charm_class(self.framework)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 115, in __init__
    cert_subject=self._cert_subject,
                 ^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 473, in _cert_subject
    svc_address = _get_gateway_address_from_svc(svc)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1052, in _get_gateway_address_from_svc
    gateway_address = _get_address_from_loadbalancer(svc)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/src/charm.py", line 1067, in _get_address_from_loadbalancer
    if len(ingresses) != 1:
       ^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
unit-istio-pilot-0: 21:19:09 ERROR juju.worker.uniter.operation hook "ingress-auth-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-knative-operator-0: 21:19:14 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kubeflow-dashboard-0: 21:19:22 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-katib-db-manager-0: 21:19:30 ERROR unit.katib-db-manager/0.juju-log Failed to handle <PebbleReadyEvent on <ops.model.Container object at 0x7fe2cb635160> via KatibDBManagerOperator/on/katib_db_manager_pebble_ready[16]> with error: Please add required database relation: relational-db
unit-mlmd-0: 21:19:31 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-katib-db-manager-0: 21:19:33 ERROR unit.katib-db-manager/0.juju-log Failed to handle <ConfigChangedEvent via KatibDBManagerOperator/on/config_changed[21]> with error: Please add required database relation: relational-db
unit-jupyter-ui-0: 21:19:33 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-envoy-0: 21:19:41 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kserve-controller-0: 21:19:49 ERROR unit.kserve-controller/0.juju-log Failed to handle <InstallEvent via KServeControllerCharm/on/install[1]> with error: Please relate to istio-pilot:gateway-info
unit-tensorboard-controller-0: 21:19:50 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kserve-controller-0: 21:19:57 ERROR unit.kserve-controller/0.juju-log Missing or incomplete local gateway data.
unit-kserve-controller-0: 21:19:57 ERROR unit.kserve-controller/0.juju-log Failed to handle <LeaderElectedEvent via KServeControllerCharm/on/leader_elected[16]> with error: Waiting for local gateway data.
unit-kserve-controller-0: 21:19:58 ERROR unit.kserve-controller/0.juju-log Missing or incomplete local gateway data.
unit-kserve-controller-0: 21:19:58 ERROR unit.kserve-controller/0.juju-log Failed to handle <PebbleReadyEvent on <ops.model.Container object at 0x7fbd26a89970> via KServeControllerCharm/on/kserve_controller_pebble_ready[21]> with error: Waiting for local gateway data.
unit-kserve-controller-0: 21:20:00 ERROR unit.kserve-controller/0.juju-log Missing or incomplete local gateway data.
unit-kserve-controller-0: 21:20:00 ERROR unit.kserve-controller/0.juju-log Failed to handle <ConfigChangedEvent via KServeControllerCharm/on/config_changed[26]> with error: Waiting for local gateway data.
unit-resource-dispatcher-0: 21:20:03 ERROR juju.worker.proxyupdater error writing apt proxy config file: open /etc/apt/apt.conf.d/95-juju-proxy-settings: permission denied
unit-kserve-controller-0: 21:20:04 ERROR unit.kserve-controller/0.juju-log ingress-gateway:17: Missing or incomplete local gateway data.
unit-kserve-controller-0: 21:20:04 ERROR unit.kserve-controller/0.juju-log ingress-gateway:17: Failed to handle <RelationChangedEvent app='istio-pilot' unit=None on <ops.model.Relation ingress-gateway:17> via KServeControllerCharm/on/ingress_gateway_relation_changed[36]> with error: Waiting for local gateway data.
unit-kserve-controller-0: 21:20:08 ERROR unit.kserve-controller/0.juju-log ingress-gateway:17: Missing or incomplete local gateway data.
unit-kserve-controller-0: 21:20:08 ERROR unit.kserve-controller/0.juju-log ingress-gateway:17: Failed to handle <RelationChangedEvent app='istio-pilot' unit='istio-pilot/0' on <ops.model.Relation ingress-gateway:17> via KServeControllerCharm/on/ingress_gateway_relation_changed[46]> with error: Waiting for local gateway data.
unit-kfp-api-0: 21:20:32 ERROR unit.kfp-api/0.juju-log Failed to handle <InstallEvent via KfpApiOperator/on/install[1]> with error: Please add required relation object-storage
unit-kfp-api-0: 21:20:44 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[26]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-kfp-api-0: 21:20:46 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent on <ops.model.Container object at 0x7fa95514f8f0> via KfpApiOperator/on/apiserver_pebble_ready[31]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-kfp-api-0: 21:20:48 ERROR unit.kfp-api/0.juju-log Failed to handle <ConfigChangedEvent via KfpApiOperator/on/config_changed[36]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-kfp-api-0: 21:20:52 ERROR unit.kfp-api/0.juju-log kfp-api:19: Failed to handle <RelationChangedEvent app='kfp-persistence' unit=None on <ops.model.Relation kfp-api:19> via KfpApiOperator/on/kfp_api_relation_changed[46]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-kfp-api-0: 21:20:56 ERROR unit.kfp-api/0.juju-log kfp-api:19: Failed to handle <RelationChangedEvent app='kfp-persistence' unit='kfp-persistence/0' on <ops.model.Relation kfp-api:19> via KfpApiOperator/on/kfp_api_relation_changed[56]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-oidc-gatekeeper-0: 21:21:27 ERROR unit.oidc-gatekeeper/0.juju-log Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:27 ERROR unit.oidc-gatekeeper/0.juju-log Failed to handle <LeaderElectedEvent via OIDCGatekeeperOperator/on/leader_elected[31]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:30 ERROR unit.oidc-gatekeeper/0.juju-log Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:30 ERROR unit.oidc-gatekeeper/0.juju-log Failed to handle <PebbleReadyEvent on <ops.model.Container object at 0x7ffacf4580e0> via OIDCGatekeeperOperator/on/oidc_authservice_pebble_ready[36]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:32 ERROR unit.oidc-gatekeeper/0.juju-log Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:32 ERROR unit.oidc-gatekeeper/0.juju-log Failed to handle <ConfigChangedEvent via OIDCGatekeeperOperator/on/config_changed[41]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:34 ERROR unit.oidc-gatekeeper/0.juju-log Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:34 ERROR unit.oidc-gatekeeper/0.juju-log Failed to handle <StartEvent via OIDCGatekeeperOperator/on/start[46]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:36 ERROR unit.oidc-gatekeeper/0.juju-log ingress:20: Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:36 ERROR unit.oidc-gatekeeper/0.juju-log ingress:20: Failed to handle <RelationChangedEvent app='istio-pilot' unit=None on <ops.model.Relation ingress:20> via OIDCGatekeeperOperator/on/ingress_relation_changed[51]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:38 ERROR unit.oidc-gatekeeper/0.juju-log ingress-auth:21: Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:38 ERROR unit.oidc-gatekeeper/0.juju-log ingress-auth:21: Failed to handle <RelationChangedEvent app='istio-pilot' unit=None on <ops.model.Relation ingress-auth:21> via OIDCGatekeeperOperator/on/ingress_auth_relation_changed[56]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:43 ERROR unit.oidc-gatekeeper/0.juju-log ingress:20: Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:43 ERROR unit.oidc-gatekeeper/0.juju-log ingress:20: Failed to handle <RelationChangedEvent app='istio-pilot' unit='istio-pilot/0' on <ops.model.Relation ingress:20> via OIDCGatekeeperOperator/on/ingress_relation_changed[66]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:21:48 ERROR unit.oidc-gatekeeper/0.juju-log ingress-auth:21: Empty or missing data. Got: No data found in relation dex-oidc-config data bag.
unit-oidc-gatekeeper-0: 21:21:48 ERROR unit.oidc-gatekeeper/0.juju-log ingress-auth:21: Failed to handle <RelationChangedEvent app='istio-pilot' unit='istio-pilot/0' on <ops.model.Relation ingress-auth:21> via OIDCGatekeeperOperator/on/ingress_auth_relation_changed[76]> with error: Empty or missing data in dex-oidc-config relation. This may be transient, but if it persists it is likely an error.
unit-oidc-gatekeeper-0: 21:22:00 ERROR unit.oidc-gatekeeper/0.juju-log oidc-client:26: Failed to handle <RelationChangedEvent app='dex-auth' unit='dex-auth/0' on <ops.model.Relation oidc-client:26> via OIDCGatekeeperOperator/on/oidc_client_relation_changed[91]> with error: List of oidc-client versions not found for apps: dex-auth
unit-oidc-gatekeeper-0: 21:22:03 ERROR unit.oidc-gatekeeper/0.juju-log dex-oidc-config:25: Failed to handle <DexOidcConfigUpdatedEvent app=None unit=None on <ops.model.Relation dex-oidc-config:25> via OIDCGatekeeperOperator/DexOidcConfigRequirer[dex-oidc-config]/on/updated[97]> with error: List of oidc-client versions not found for apps: dex-auth
unit-oidc-gatekeeper-0: 21:22:03 ERROR unit.oidc-gatekeeper/0.juju-log dex-oidc-config:25: Failed to handle <RelationChangedEvent app='dex-auth' unit='dex-auth/0' on <ops.model.Relation dex-oidc-config:25> via OIDCGatekeeperOperator/on/dex_oidc_config_relation_changed[96]> with error: List of oidc-client versions not found for apps: dex-auth
unit-oidc-gatekeeper-0: 21:22:08 ERROR unit.oidc-gatekeeper/0.juju-log dex-oidc-config:25: Failed to handle <DexOidcConfigUpdatedEvent app=None unit=None on <ops.model.Relation dex-oidc-config:25> via OIDCGatekeeperOperator/DexOidcConfigRequirer[dex-oidc-config]/on/updated[103]> with error: List of oidc-client versions not found for apps: dex-auth
unit-oidc-gatekeeper-0: 21:22:08 ERROR unit.oidc-gatekeeper/0.juju-log dex-oidc-config:25: Failed to handle <RelationChangedEvent app='dex-auth' unit=None on <ops.model.Relation dex-oidc-config:25> via OIDCGatekeeperOperator/on/dex_oidc_config_relation_changed[102]> with error: List of oidc-client versions not found for apps: dex-auth
unit-kfp-api-0: 21:22:10 ERROR unit.kfp-api/0.juju-log object-storage:49: Failed to handle <RelationChangedEvent app='minio' unit='minio/0' on <ops.model.Relation object-storage:49> via KfpApiOperator/on/object_storage_relation_changed[66]> with error: List of <ops.model.Relation object-storage:49> versions not found for apps: minio
unit-oidc-gatekeeper-0: 21:22:11 ERROR unit.oidc-gatekeeper/0.juju-log ingress:20: Failed to handle <RelationChangedEvent app='istio-pilot' unit=None on <ops.model.Relation ingress:20> via OIDCGatekeeperOperator/on/ingress_relation_changed[108]> with error: List of oidc-client versions not found for apps: dex-auth
unit-minio-0: 21:22:16 ERROR unit.minio/0.juju-log object-storage:47: execute_components caught unhandled exception when executing configure_charm for relation:object_storage
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 295, in get_sdi_interface
    interface = get_interface(charm, relation_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 388, in get_interface
    instance.get_data()
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 172, in get_data
    rel_data = self.unwrap(relation)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 269, in unwrap
    version = self.get_version(relation)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 111, in get_version
    raise errors.UnversionedRelation(relation)
serialized_data_interface.errors.UnversionedRelation: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 92, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 51, in configure_charm
    self._configure_app(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 88, in _configure_app
    self._configure_app_leader(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 223, in _configure_app_leader
    interface = self.get_interface()
                ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 231, in get_interface
    return get_sdi_interface(self._charm, self._relation_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 298, in get_sdi_interface
    raise ErrorWithStatus(str(err), WaitingStatus) from err
charmed_kubeflow_chisme.exceptions._with_status.ErrorWithStatus: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui
unit-kfp-api-0: 21:22:23 ERROR unit.kfp-api/0.juju-log object-storage:49: Failed to handle <RelationChangedEvent app='minio' unit=None on <ops.model.Relation object-storage:49> via KfpApiOperator/on/object_storage_relation_changed[71]> with error: List of <ops.model.Relation kfp-viz:35> versions not found for apps: kfp-viz
unit-minio-0: 21:22:27 ERROR unit.minio/0.juju-log object-storage:48: execute_components caught unhandled exception when executing configure_charm for relation:object_storage
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 295, in get_sdi_interface
    interface = get_interface(charm, relation_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 388, in get_interface
    instance.get_data()
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 172, in get_data
    rel_data = self.unwrap(relation)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 269, in unwrap
    version = self.get_version(relation)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 111, in get_version
    raise errors.UnversionedRelation(relation)
serialized_data_interface.errors.UnversionedRelation: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 92, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 51, in configure_charm
    self._configure_app(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 88, in _configure_app
    self._configure_app_leader(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 223, in _configure_app_leader
    interface = self.get_interface()
                ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 231, in get_interface
    return get_sdi_interface(self._charm, self._relation_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 298, in get_sdi_interface
    raise ErrorWithStatus(str(err), WaitingStatus) from err
charmed_kubeflow_chisme.exceptions._with_status.ErrorWithStatus: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui
unit-kfp-api-0: 21:22:36 ERROR unit.kfp-api/0.juju-log kfp-api:27: Failed to handle <RelationChangedEvent app='kfp-ui' unit='kfp-ui/0' on <ops.model.Relation kfp-api:27> via KfpApiOperator/on/kfp_api_relation_changed[81]> with error: List of <ops.model.Relation kfp-viz:35> versions not found for apps: kfp-viz
unit-minio-0: 21:22:38 ERROR unit.minio/0.juju-log object-storage:49: execute_components caught unhandled exception when executing configure_charm for relation:object_storage
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 295, in get_sdi_interface
    interface = get_interface(charm, relation_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 388, in get_interface
    instance.get_data()
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 172, in get_data
    rel_data = self.unwrap(relation)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 269, in unwrap
    version = self.get_version(relation)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 111, in get_version
    raise errors.UnversionedRelation(relation)
serialized_data_interface.errors.UnversionedRelation: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 92, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 51, in configure_charm
    self._configure_app(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 88, in _configure_app
    self._configure_app_leader(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 223, in _configure_app_leader
    interface = self.get_interface()
                ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 231, in get_interface
    return get_sdi_interface(self._charm, self._relation_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 298, in get_sdi_interface
    raise ErrorWithStatus(str(err), WaitingStatus) from err
charmed_kubeflow_chisme.exceptions._with_status.ErrorWithStatus: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui
unit-kfp-api-0: 21:22:45 ERROR unit.kfp-api/0.juju-log kfp-viz:35: Failed to handle <RelationChangedEvent app='kfp-viz' unit='kfp-viz/0' on <ops.model.Relation kfp-viz:35> via KfpApiOperator/on/kfp_viz_relation_changed[91]> with error: List of <ops.model.Relation kfp-viz:35> versions not found for apps: kfp-viz
unit-minio-0: 21:22:49 ERROR unit.minio/0.juju-log object-storage:50: execute_components caught unhandled exception when executing configure_charm for relation:object_storage
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 295, in get_sdi_interface
    interface = get_interface(charm, relation_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 388, in get_interface
    instance.get_data()
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 172, in get_data
    rel_data = self.unwrap(relation)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 269, in unwrap
    version = self.get_version(relation)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/serialized_data_interface/sdi.py", line 111, in get_version
    raise errors.UnversionedRelation(relation)
serialized_data_interface.errors.UnversionedRelation: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 92, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 51, in configure_charm
    self._configure_app(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 88, in _configure_app
    self._configure_app_leader(event)
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 223, in _configure_app_leader
    interface = self.get_interface()
                ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 231, in get_interface
    return get_sdi_interface(self._charm, self._relation_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-minio-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/serialised_data_interface_components.py", line 298, in get_sdi_interface
    raise ErrorWithStatus(str(err), WaitingStatus) from err
charmed_kubeflow_chisme.exceptions._with_status.ErrorWithStatus: List of <ops.model.Relation object-storage:47> versions not found for apps: kfp-ui
unit-kfp-api-0: 21:22:52 ERROR unit.kfp-api/0.juju-log kfp-viz:35: Failed to handle <RelationChangedEvent app='kfp-viz' unit=None on <ops.model.Relation kfp-viz:35> via KfpApiOperator/on/kfp_viz_relation_changed[96]> with error: List of <ops.model.Relation kfp-api:27> versions not found for apps: kfp-ui
unit-tensorboard-controller-0: 21:22:53 ERROR unit.tensorboard-controller/0.juju-log Kubernetes service patch failed:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"controller.juju.is/id\":\"6c9a2b3e-69bc-4755-8613-a4f81f0c67fc\",\"juju.is/version\":\"3.6.9\",\"model.juju.is/id\":\"c59e3827-7970-4346-8dd9-e34c2bf63cf6\"},\"creationTimestamp\":\"2026-03-19T20:18:00Z\",\"labels\":{\"app.kubernetes.io/managed-by\":\"juju\",\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"managedFields\":[{\"manager\":\"juju\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2026-03-19T20:18:00Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:controller.juju.is/id\":{},\"f:juju.is/version\":{},\"f:model.juju.is/id\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\"f:internalTrafficPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":65535,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}}],\"name\":\"tensorboard-controller\",\"namespace\":\"kubeflow\",\"resourceVersion\":\"1351788\",\"uid\":\"c2fd95e1-22d3-4a2c-a425-a9240ef34a48\"},\"spec\":{\"clusterIP\":\"10.0.144.215\",\"clusterIPs\":[\"10.0.144.215\"],\"internalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"ports\":[{\"name\":\"tensorboard-controller-metrics\",\"port\":\"8080\",\"targetPort\":\"8080\"}],\"selector\":{\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"sessionAffinity\":\"None\",\"type\":\"ClusterIP\"},\"status\":{\"loadBalancer\":{}}}": json: cannot unmarshal string into Go struct field ServicePort.spec.ports.port of type int32
unit-kfp-api-0: 21:22:59 ERROR unit.kfp-api/0.juju-log kfp-api:27: Failed to handle <RelationChangedEvent app='kfp-ui' unit=None on <ops.model.Relation kfp-api:27> via KfpApiOperator/on/kfp_api_relation_changed[101]> with error: Waiting for object-storage relation data
unit-kfp-api-0: 21:23:19 ERROR unit.kfp-api/0.juju-log object-storage:49: Failed to generate container configuration.
unit-kfp-api-0: 21:23:20 ERROR unit.kfp-api/0.juju-log object-storage:49: Failed to handle <RelationChangedEvent app='minio' unit=None on <ops.model.Relation object-storage:49> via KfpApiOperator/on/object_storage_relation_changed[106]> with error: Please add required database relation: eg. relational-db
unit-katib-db-manager-0: 21:24:06 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Failed to handle <RelationJoinedEvent app='katib-db' unit='katib-db/2' on <ops.model.Relation relational-db:51> via KatibDBManagerOperator/on/relational_db_relation_joined[46]> with error: Waiting for relational-db data
unit-kfp-api-0: 21:24:12 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to generate container configuration.
unit-kfp-api-0: 21:24:12 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to handle <RelationJoinedEvent app='kfp-db' unit='kfp-db/1' on <ops.model.Relation relational-db:52> via KfpApiOperator/on/relational_db_relation_joined[116]> with error: Waiting for relational-db data
unit-katib-db-manager-0: 21:24:12 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Failed to handle <RelationChangedEvent app='katib-db' unit='katib-db/2' on <ops.model.Relation relational-db:51> via KatibDBManagerOperator/on/relational_db_relation_changed[51]> with error: Waiting for relational-db data
unit-katib-db-manager-0: 21:24:20 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Failed to handle <RelationJoinedEvent app='katib-db' unit='katib-db/0' on <ops.model.Relation relational-db:51> via KatibDBManagerOperator/on/relational_db_relation_joined[56]> with error: Waiting for relational-db data
unit-kfp-api-0: 21:24:23 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to generate container configuration.
unit-kfp-api-0: 21:24:23 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to handle <RelationChangedEvent app='kfp-db' unit='kfp-db/1' on <ops.model.Relation relational-db:52> via KfpApiOperator/on/relational_db_relation_changed[121]> with error: Waiting for relational-db data
unit-katib-db-manager-0: 21:24:26 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Failed to handle <RelationChangedEvent app='katib-db' unit='katib-db/0' on <ops.model.Relation relational-db:51> via KatibDBManagerOperator/on/relational_db_relation_changed[61]> with error: Waiting for relational-db data
unit-kfp-api-0: 21:24:32 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to generate container configuration.
unit-kfp-api-0: 21:24:33 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to handle <RelationJoinedEvent app='kfp-db' unit='kfp-db/0' on <ops.model.Relation relational-db:52> via KfpApiOperator/on/relational_db_relation_joined[126]> with error: Waiting for relational-db data
unit-kfp-api-0: 21:24:42 ERROR unit.kfp-api/0.juju-log relational-db:52: Missing attribute 'password' in relational-db relation data
unit-kfp-api-0: 21:24:42 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to generate container configuration.
unit-kfp-api-0: 21:24:42 ERROR unit.kfp-api/0.juju-log relational-db:52: Failed to handle <RelationChangedEvent app='kfp-db' unit='kfp-db/0' on <ops.model.Relation relational-db:52> via KfpApiOperator/on/relational_db_relation_changed[131]> with error: Incorrect/incomplete data found in relation relational-db. See logs
unit-katib-db-manager-0: 21:24:42 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Missing attribute 'username' in relational-db relation data
unit-katib-db-manager-0: 21:24:42 ERROR unit.katib-db-manager/0.juju-log relational-db:51: Failed to handle <RelationChangedEvent app='katib-db' unit=None on <ops.model.Relation relational-db:51> via KatibDBManagerOperator/on/relational_db_relation_changed[66]> with error: Incorrect/incomplete data found in relation relational-db. See logs
unit-tensorboard-controller-0: 21:26:59 ERROR unit.tensorboard-controller/0.juju-log Kubernetes service patch failed:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"controller.juju.is/id\":\"6c9a2b3e-69bc-4755-8613-a4f81f0c67fc\",\"juju.is/version\":\"3.6.9\",\"model.juju.is/id\":\"c59e3827-7970-4346-8dd9-e34c2bf63cf6\"},\"creationTimestamp\":\"2026-03-19T20:18:00Z\",\"labels\":{\"app.kubernetes.io/managed-by\":\"juju\",\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"managedFields\":[{\"manager\":\"juju\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2026-03-19T20:18:00Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:controller.juju.is/id\":{},\"f:juju.is/version\":{},\"f:model.juju.is/id\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\"f:internalTrafficPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":65535,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}}],\"name\":\"tensorboard-controller\",\"namespace\":\"kubeflow\",\"resourceVersion\":\"1351788\",\"uid\":\"c2fd95e1-22d3-4a2c-a425-a9240ef34a48\"},\"spec\":{\"clusterIP\":\"10.0.144.215\",\"clusterIPs\":[\"10.0.144.215\"],\"internalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"ports\":[{\"name\":\"tensorboard-controller-metrics\",\"port\":\"8080\",\"targetPort\":\"8080\"}],\"selector\":{\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"sessionAffinity\":\"None\",\"type\":\"ClusterIP\"},\"status\":{\"loadBalancer\":{}}}": json: cannot unmarshal string into Go struct field ServicePort.spec.ports.port of type int32
unit-tensorboard-controller-0: 21:32:33 ERROR unit.tensorboard-controller/0.juju-log Kubernetes service patch failed:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"controller.juju.is/id\":\"6c9a2b3e-69bc-4755-8613-a4f81f0c67fc\",\"juju.is/version\":\"3.6.9\",\"model.juju.is/id\":\"c59e3827-7970-4346-8dd9-e34c2bf63cf6\"},\"creationTimestamp\":\"2026-03-19T20:18:00Z\",\"labels\":{\"app.kubernetes.io/managed-by\":\"juju\",\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"managedFields\":[{\"manager\":\"juju\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2026-03-19T20:18:00Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:controller.juju.is/id\":{},\"f:juju.is/version\":{},\"f:model.juju.is/id\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\"f:internalTrafficPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":65535,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}}],\"name\":\"tensorboard-controller\",\"namespace\":\"kubeflow\",\"resourceVersion\":\"1351788\",\"uid\":\"c2fd95e1-22d3-4a2c-a425-a9240ef34a48\"},\"spec\":{\"clusterIP\":\"10.0.144.215\",\"clusterIPs\":[\"10.0.144.215\"],\"internalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"ports\":[{\"name\":\"tensorboard-controller-metrics\",\"port\":\"8080\",\"targetPort\":\"8080\"}],\"selector\":{\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"sessionAffinity\":\"None\",\"type\":\"ClusterIP\"},\"status\":{\"loadBalancer\":{}}}": json: cannot unmarshal string into Go struct field ServicePort.spec.ports.port of type int32
unit-tensorboard-controller-0: 21:37:03 ERROR unit.tensorboard-controller/0.juju-log Kubernetes service patch failed:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"controller.juju.is/id\":\"6c9a2b3e-69bc-4755-8613-a4f81f0c67fc\",\"juju.is/version\":\"3.6.9\",\"model.juju.is/id\":\"c59e3827-7970-4346-8dd9-e34c2bf63cf6\"},\"creationTimestamp\":\"2026-03-19T20:18:00Z\",\"labels\":{\"app.kubernetes.io/managed-by\":\"juju\",\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"managedFields\":[{\"manager\":\"juju\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2026-03-19T20:18:00Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:controller.juju.is/id\":{},\"f:juju.is/version\":{},\"f:model.juju.is/id\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\"f:internalTrafficPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":65535,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}}],\"name\":\"tensorboard-controller\",\"namespace\":\"kubeflow\",\"resourceVersion\":\"1351788\",\"uid\":\"c2fd95e1-22d3-4a2c-a425-a9240ef34a48\"},\"spec\":{\"clusterIP\":\"10.0.144.215\",\"clusterIPs\":[\"10.0.144.215\"],\"internalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"ports\":[{\"name\":\"tensorboard-controller-metrics\",\"port\":\"8080\",\"targetPort\":\"8080\"}],\"selector\":{\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"sessionAffinity\":\"None\",\"type\":\"ClusterIP\"},\"status\":{\"loadBalancer\":{}}}": json: cannot unmarshal string into Go struct field ServicePort.spec.ports.port of type int32
unit-tensorboard-controller-0: 21:42:47 ERROR unit.tensorboard-controller/0.juju-log Kubernetes service patch failed:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"controller.juju.is/id\":\"6c9a2b3e-69bc-4755-8613-a4f81f0c67fc\",\"juju.is/version\":\"3.6.9\",\"model.juju.is/id\":\"c59e3827-7970-4346-8dd9-e34c2bf63cf6\"},\"creationTimestamp\":\"2026-03-19T20:18:00Z\",\"labels\":{\"app.kubernetes.io/managed-by\":\"juju\",\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"managedFields\":[{\"manager\":\"juju\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2026-03-19T20:18:00Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:controller.juju.is/id\":{},\"f:juju.is/version\":{},\"f:model.juju.is/id\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}}},\"f:spec\":{\"f:internalTrafficPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":65535,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}}],\"name\":\"tensorboard-controller\",\"namespace\":\"kubeflow\",\"resourceVersion\":\"1351788\",\"uid\":\"c2fd95e1-22d3-4a2c-a425-a9240ef34a48\"},\"spec\":{\"clusterIP\":\"10.0.144.215\",\"clusterIPs\":[\"10.0.144.215\"],\"internalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"ports\":[{\"name\":\"tensorboard-controller-metrics\",\"port\":\"8080\",\"targetPort\":\"8080\"}],\"selector\":{\"app.kubernetes.io/name\":\"tensorboard-controller\"},\"sessionAffinity\":\"None\",\"type\":\"ClusterIP\"},\"status\":{\"loadBalancer\":{}}}": json: cannot unmarshal string into Go struct field ServicePort.spec.ports.port of type int32

INFO     pytest_operator.plugin:plugin.py:1025 Forgetting model main...


========================================================== 2 passed, 1 skipped in 613.26s (0:10:13) ==========================================================
  kubeflow-remote: OK (616.05=setup[0.05]+cmd[1.12,614.89] seconds)
  congratulations :) (616.07 seconds)

Teardown

procedure

az group delete --name ${RESOURCE_GROUP_NAME} --yes

if [ "$(az group exists --name MC_${RESOURCE_GROUP_NAME}_${CLUSTER_NAME}_${RESOURCE_LOCATION})" = "true" ]; then
az group delete --name MC_${RESOURCE_GROUP_NAME}_${CLUSTER_NAME}_${RESOURCE_LOCATION} --yes
fi