v25-1.1-beta2 Redpanda Operator via FluxCD Failing

[PeregrineFalcon]

Continuing a conversation from over in theSlack with @chrisseto.

Deploying the CRD and HelmRelease with the following kustomization(s):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/redpanda-data/redpanda-operator//operator/config/crd?ref=v25.1.1-beta2
  - redpanda-operator-helm-release.yaml

with redpanda-operator-helm-release.yaml as follows:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redpanda-operator-helm-release
  namespace: redpanda
spec:
  releaseName: redpanda-operator
  chart:
    spec:
      chart: operator
      sourceRef:
        kind: HelmRepository
        name: redpanda
        namespace: flux-system
  interval: 1h
  timeout: 10m
  install:
    remediation:
      retries: 3
  values:
    image:
      tag: v25.1.1-beta2
    #    createAdditionalControllerCRs is required for a workaround for v25.1.1-beta2, should be fixed in beta3+
    createAdditionalControllerCRs: true

For redpanda using:

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
  namespace: redpanda
spec:
  clusterSpec:
    tls:
      enabled: false
    external:
      enabled: true
    statefulset:
      replicas: 3
      extraVolumes: |-
        - name: redpanda-io-config
          configMap:
            name: redpanda-io-config
      extraVolumeMounts: |-
        - name: redpanda-io-config
          mountPath: /etc/redpanda-io-config
      additionalRedpandaCmdFlags:
        - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml"

As far as the cluster, I have confirmed that this is the chart as it is represented in the running cluster:

additionalCmdFlags: []
affinity: {}
clusterDomain: cluster.local
commonLabels: {}
config:
  apiVersion: controller-runtime.sigs.k8s.io/v1alpha1
  health:
    healthProbeBindAddress: :8081
  kind: ControllerManagerConfig
  leaderElection:
    leaderElect: true
    resourceName: aa9fc693.vectorized.io
  metrics:
    bindAddress: 127.0.0.1:8080
  webhook:
    port: 9443
fullnameOverride: ""
image:
  pullPolicy: IfNotPresent
  repository: docker.redpanda.com/redpandadata/redpanda-operator
  tag: v25.1.1-beta2
imagePullSecrets: []
kubeRbacProxy:
  image:
    pullPolicy: IfNotPresent
    repository: gcr.io/kubebuilder/kube-rbac-proxy
    tag: v0.14.0
  logLevel: 10
logLevel: info
monitoring:
  enabled: false
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podLabels: {}
podTemplate:
  metadata: {}
  spec:
    containers:
    - name: manager
      resources: {}
    securityContext:
      runAsUser: 65532
rbac:
  create: true
  createAdditionalControllerCRs: true
  createRPKBundleCRs: false
replicaCount: 1
resources: {}
scope: Namespace
serviceAccount:
  automountServiceAccountToken: false
  create: true
strategy:
  type: RollingUpdate
tolerations: []
webhook:
  enabled: false
webhookSecretName: webhook-server-cert

Errors I'm seeing on the redpanda-0 pod are as follows:

MountVolume.SetUp failed for volume "redpanda-configurator" : secret "redpanda-configurator" not found

and of the four containers all have a similar error:

Failed to load logs: container "redpanda-configurator" in pod "redpanda-0" is waiting to start: PodInitializing
Reason: BadRequest (400)

with what appears to the be cause of:

MountVolume.SetUp failed for volume "redpanda-configurator" : secret "redpanda-configurator" not found

[chrisseto]

Could you please share the logs of the operator?

Though I think I see what the issue is. It appears you're not using the v25.1.1-beta2 helm chart of the operator.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redpanda-operator-helm-release
  namespace: redpanda
spec:
  releaseName: redpanda-operator
  chart:
    spec:
      chart: operator
      # CHANGE #1 HERE: v25.1.1 is still in beta. Helm will not install it unless you explicitly specify it.
      version: v25.1.1-beta2
      sourceRef:
        kind: HelmRepository
        name: redpanda
        namespace: flux-system
  interval: 1h
  timeout: 10m
  install:
    remediation:
      retries: 3
  values:
    # CHANGE #2: Don't specify image here. The chart will set the image for you.
    #    createAdditionalControllerCRs is required for a workaround for v25.1.1-beta2, should be fixed in beta3+
    createAdditionalControllerCRs: true

If you use the helm CLI you can see the same issue:

❯ helm search repo redpanda/operator
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
redpanda/operator       v2.4.2          v2.4.2          Redpanda operator helm chart
❯ helm search repo redpanda/operator --devel
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
redpanda/operator       v25.1.1-beta2   v25.1.1-beta2   Redpanda operator helm chart

[PeregrineFalcon]

yeah that seems to have resolved it.

A separate issue has risen up, for v25.1.1-beta2 did the io-config.yaml format change? I haven't made any changes but it's telling me its malformed.

[chrisseto]

From slack:

Detecting the current cloud provider and VM
System check - STARTED
System check "NTP Synced" failed with non-fatal error "couldn't check NTP with timedatectl or ntpstat"
System check "Dir '/var/lib/redpanda/data' scheduler tuned" failed with non-fatal error "open : no such file or directory"
System check "Dir '/var/lib/redpanda/data' nomerges tuned" failed with non-fatal error "open : no such file or directory"
System check "Dir '/var/lib/redpanda/data' IRQs affinity static" failed with non-fatal error "no such file or directory"
System check "Dir '/var/lib/redpanda/data' IRQs affinity set" failed with non-fatal error "no such file or directory"
System check "Fstrim systemd service and timer active" failed with non-fatal error "dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory"
System check "RFS Table entries" failed with non-fatal error "open /proc/sys/net/core/rps_sock_flow_entries: no such file or directory"
System check - PASSED
We'd love to hear about your experience with Redpanda:
https://redpanda.com/feedback
Starting redpanda...
Running:
/opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --reserve-memory=205M --smp=1 --lock-memory=false --default-log-level=info --io-properties-file=/etc/redpanda-io-config/io-config.yaml --memory=2048M
Welcome to the Redpanda community!
Documentation: https://docs.redpanda.com - Product documentation site
GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions
GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase
Support: https://support.redpanda.com - Contact the support team privately
Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience
Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation!
Twitter: https://twitter.com/redpandadata - All the latest Redpanda news!
Could not initialize seastar: YAML::BadFile (bad file: /etc/redpanda-io-config/io-config.yaml)

That's not the best error message from seastar. Your io-config.yaml did not get mounted. This is a big part of why we're still in beta, we've made some break changes to the API that haven't yet been realized in the CRD yet. We removed a lot of fields that we duplicated by the more clear podTemplate field, namely everything that starts with extra.

If you update your CR to this:

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
  namespace: redpanda
spec:
  clusterSpec:
    tls:
      enabled: false
    external:
      enabled: true
    statefulset:
      replicas: 3
      podTemplate:
        spec:
          containers:
          - name: redpanda
            volumeMounts:
            - name: redpanda-io-config
              mountPath: /etc/redpanda-io-config
          volumes:
          - name: redpanda-io-config
            configMap:
              name: redpanda-io-config
      additionalRedpandaCmdFlags:
        - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml"

[PeregrineFalcon]

This is all working with v25.1.1-beta2.

Steps that resolved issues:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redpanda-operator-helm-release
  namespace: redpanda
spec:
  releaseName: redpanda-operator
  chart:
    spec:
      chart: operator
 	 # Step 1: Upgrade to v25.1.1-beta2
      version: v25.1.1-beta2
      sourceRef:
        kind: HelmRepository
        name: redpanda
        namespace: flux-system
  interval: 1h
  timeout: 10m
  install:
    remediation:
      retries: 3
  values:
 	# Step 2: Add rbac entry for createAdditionalControllerCRs
	rbac:
    	#    createAdditionalControllerCRs is required for a workaround for v25.1.1-beta2, should be fixed in beta3+
	    createAdditionalControllerCRs: true

and

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
  namespace: redpanda
spec:
  clusterSpec:
    tls:
      enabled: false
    external:
      enabled: true
    statefulset:
      replicas: 3
      podTemplate:
        spec:
          containers:
          - name: redpanda
            volumeMounts:
            - name: redpanda-io-config
              mountPath: /etc/redpanda-io-config
          volumes:
          - name: redpanda-io-config
            configMap:
              name: redpanda-io-config
      additionalRedpandaCmdFlags:
        - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml"