Talos deployment stuck on control panel initialisation

[madbbb]

Cluster creation stucks on Controlplane creation with strange messages in clusterctl describe cluster

Cluster/unovi                                           False  Warning   ScalingUp                 21m    Scaling up control plane to 274883263688 replicas (actual 2)   
├─ClusterInfrastructure - HetznerCluster/test                                                                                                                           
└─ControlPlane - TalosControlPlane/test-control-plane  False  Warning   ScalingUp                 21m    Scaling up control plane to 274883263688 replicas (actual 2)   
  └─3 Machines...                                       False  Info      WaitingForInfrastructure  21m    See test-control-plane-475lf, test-control-plane-8vxcp, ...  

Why it tries to scale to 274883263688 when I only have set 3 in env vars and TalosControlPlane object has 3 replicas set?

I build image with packer build templates/node-image/talos-image/image.json and initialised cluster with:
clusterctl generate cluster test --kubernetes-version v1.25.2 --control-plane-machine-count=3 --worker-machine-count=3 --flavor hcloud-talos-packer > test-cluster.yaml

Here are my env vars:

export CLUSTER_NAME=test \
export HCLOUD_SSH_KEY=bbb \
export HCLOUD_REGION=fsn1 \
export CONTROL_PLANE_MACHINE_COUNT=3 \
export WORKER_MACHINE_COUNT=3 \
export KUBERNETES_VERSION=1.25.2 \
export HCLOUD_CONTROL_PLANE_MACHINE_TYPE=cx21 \
export HCLOUD_WORKER_MACHINE_TYPE=cpx31 \
export HCLOUD_IMAGE_NAME=caph-debian-11-2023-03-24-020211 \
export HCLOUD_TOKEN=token

Provisioned cluster with:

clusterctl init --bootstrap talos --control-plane talos --infrastructure hetzner

LoadBalancer and Placement groups are created and this is it. Is there any log to check? Could you advise me where to look?

[janiskemper]

You can, of course, give us a few logs to check whether something might be relevant. In general, I have not seen this weird behavior. Seems like something that happened because of some strange edge case... Probably not because of CAPH

[madbbb]

Hello @janiskemper! Here are generated cluster.yaml and TalosControlPlane dump, please tell me if there any other logs I can share (I have no idea):

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"TalosControlPlane","metadata":{"annotations":{},"name":"unovi-control-plane","namespace":"default"},"spec":{"controlPlaneConfig":{"controlplane":{"configPatches":[{"op":"add","path":"/cluster/externalCloudProvider","value":{"enabled":true}}],"generateType":"controlplane","talosVersion":"v1.3.6"}},"infrastructureTemplate":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta1","kind":"HCloudMachineTemplate","name":"unovi-control-plane"},"replicas":3,"version":"v1.26.2"}}
  creationTimestamp: "2023-03-24T14:24:22Z"
  finalizers:
  - talos.controlplane.cluster.x-k8s.io
  generation: 1
  labels:
    cluster.x-k8s.io/cluster-name: unovi
  name: unovi-control-plane
  namespace: default
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: unovi
    uid: 9007c343-7303-4354-8c4d-6e5be8e4ccab
  resourceVersion: "4533"
  uid: e231bf23-cd65-476e-b36b-cf8fff75eff6
spec:
  controlPlaneConfig:
    controlplane:
      configPatches:
      - op: add
        path: /cluster/externalCloudProvider
        value:
          enabled: true
      generateType: controlplane
      hostname: {}
      talosVersion: v1.3.6
    init:
      generateType: ""
      hostname: {}
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: HCloudMachineTemplate
    name: unovi-control-plane
  replicas: 3
  rolloutStrategy:
    rollingUpdate:
      maxSurge: 1
    type: RollingUpdate
  version: v1.26.2
status:
  conditions:
  - lastTransitionTime: "2023-03-24T14:24:24Z"
    message: Scaling up control plane to 274883263688 replicas (actual 2)
    reason: ScalingUp
    severity: Warning
    status: "False"
    type: Ready
  - lastTransitionTime: "2023-03-24T14:24:24Z"
    reason: WaitingForTalosBoot
    severity: Info
    status: "False"
    type: Available
  - lastTransitionTime: "2023-03-24T14:26:56Z"
    message: no addresses were found for node "unovi-control-plane-475lf"
    reason: ControlPlaneComponentsInspectionFailed
    severity: Warning
    status: "False"
    type: ControlPlaneComponentsHealthy
  - lastTransitionTime: "2023-03-24T14:26:56Z"
    message: no addresses were found for node "unovi-control-plane-475lf"
    reason: EtcdClusterUnhealthy
    severity: Warning
    status: "False"
    type: EtcdClusterHealthyCondition
  - lastTransitionTime: "2023-03-24T14:26:56Z"
    message: no addresses were found for node "unovi-control-plane-475lf"
    reason: WaitingForTalosBoot
    severity: Info
    status: "False"
    type: MachinesBootstrapped
  - lastTransitionTime: "2023-03-24T14:26:56Z"
    message: 1 of 2 completed
    reason: WaitingForInfrastructure @ Machine/unovi-control-plane-8vxcp
    severity: Info
    status: "False"
    type: MachinesReady
  - lastTransitionTime: "2023-03-24T14:24:24Z"
    message: Scaling up control plane to 274883263688 replicas (actual 2)
    reason: ScalingUp
    severity: Warning
    status: "False"
    type: Resized
  observedGeneration: 1
  replicas: 3
  selector: cluster.x-k8s.i

full generated cluster.yaml

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
name: unovi-md-0
namespace: default
spec:
template:
  spec:
    configPatches:
    - op: add
      path: /cluster/externalCloudProvider
      value:
        enabled: true
    generateType: worker
    talosVersion: v1.3.6
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: unovi
namespace: default
spec:
controlPlaneRef:
  apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
  kind: TalosControlPlane
  name: unovi-control-plane
infrastructureRef:
  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: HetznerCluster
  name: unovi
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
labels:
  nodepool: unovi-md-0
name: unovi-md-0
namespace: default
spec:
clusterName: unovi
replicas: 3
selector:
  matchLabels: null
template:
  metadata:
    labels:
      nodepool: unovi-md-0
  spec:
    bootstrap:
      configRef:
        apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
        kind: KubeadmConfigTemplate
        name: unovi-md-0
    clusterName: unovi
    failureDomain: fsn1
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: HCloudMachineTemplate
      name: unovi-md-0
    version: v1.26.2
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
name: unovi-control-plane-unhealthy-5m
namespace: default
spec:
clusterName: unovi
maxUnhealthy: 100%
nodeStartupTimeout: 20m
selector:
  matchLabels:
    cluster.x-k8s.io/control-plane: ""
unhealthyConditions:
- status: Unknown
  timeout: 300s
  type: Ready
- status: "False"
  timeout: 300s
  type: Ready
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
name: unovi-md-0-unhealthy-5m
namespace: default
spec:
clusterName: unovi
maxUnhealthy: 100%
nodeStartupTimeout: 20m
selector:
  matchLabels:
    nodepool: unovi-md-0
unhealthyConditions:
- status: Unknown
  timeout: 300s
  type: Ready
- status: "False"
  timeout: 300s
  type: Ready
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
name: unovi-control-plane
namespace: default
spec:
controlPlaneConfig:
  controlplane:
    configPatches:
    - op: add
      path: /cluster/externalCloudProvider
      value:
        enabled: true
    generateType: controlplane
    talosVersion: v1.3.6
infrastructureTemplate:
  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: HCloudMachineTemplate
  name: unovi-control-plane
replicas: 3
version: v1.26.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCloudMachineTemplate
metadata:
name: unovi-control-plane
namespace: default
spec:
template:
  spec:
    imageName: caph-debian-11-2023-03-24-020211
    placementGroupName: control-plane
    type: cx21
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCloudMachineTemplate
metadata:
name: unovi-md-0
namespace: default
spec:
template:
  spec:
    imageName: caph-debian-11-2023-03-24-020211
    placementGroupName: md-0
    type: cpx31
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HetznerCluster
metadata:
name: unovi
namespace: default
spec:
controlPlaneEndpoint:
  host: ""
  port: 443
controlPlaneLoadBalancer:
  region: fsn1
controlPlaneRegions:
- fsn1
hcloudNetwork:
  enabled: false
hcloudPlacementGroups:
- name: control-plane
  type: spread
- name: md-0
  type: spread
hetznerSecretRef:
  key:
    hcloudToken: hcloud
  name: hetzner
sshKeys:
  hcloud:
  - name: bbb

[janiskemper]

The logs from caph operator/controller would be interesting for me. Then I can tell you if something is going wrong on CAPH side!

[madbbb]

Thank you for pointing out where to check logs. Controller logs showed that I set wrong image name. I found another issue after https://github.com/syself/cluster-api-provider-hetzner/pull/700/files

[janiskemper]

Just for the record: Did you also see this in the events? I would have said it should be like that. So if it is not, we should update it accordingly! Can you maybe reproduce this and check the events?