[🐛 Bug]: Cannot install Helm Chart >= 0.33.0 - "Secret 'helm-operation-xxxx' is invalid"

[LQuartier]

What happened?

Hello,
i'm currently running the Helm Chart Version 0.32.0, albeit with image Tags for 4.24.0
We're using:

Rancher Version 2.9.3
Helm v2.16.8-rancher2
Kubernetes v1.30.5+rke2r1

I've tried upgrading the Helm Chart to newer Version multiple times in the past, but i'm always running into this Error:

Secret "helm-operation-xxxx" is invalid: data: Too long: must have at most 1048576 bytes

Versions 0.32.0 and lower work fine.
I am not sure what i can try on my end to make the upgrade Possible.
If you have any Pointers, i would be grateful.

Command used to start Selenium Grid with Docker (or Kubernetes)

autoscaling:
  annotations:
    helm.sh/hook: post-install,post-upgrade,post-rollback
    helm.sh/hook-weight: '1'
  deregisterLifecycle: null
  enableWithExistingKEDA: false
  enabled: false
  patchObjectFinalizers:
    activeDeadlineSeconds: 120
    annotations:
      helm.sh/hook: post-install,post-upgrade,post-rollback,pre-delete
      helm.sh/hook-delete-policy: hook-succeeded,before-hook-creation
      helm.sh/hook-weight: '-1'
    enabled: true
  scaledJobOptions:
    failedJobsHistoryLimit: 0
    jobTargetRef:
      backoffLimit: 0
      completions: 1
      parallelism: 1
    scalingStrategy:
      strategy: default
    successfulJobsHistoryLimit: 0
  scaledObjectOptions:
    scaleTargetRef:
      kind: Deployment
  scaledOptions:
    maxReplicaCount: 16
    minReplicaCount: 0
    pollingInterval: 10
  scalingType: job
  terminationGracePeriodSeconds: 3600
basicAuth:
  enabled: false
  password: admin
  username: admin
busConfigMap:
  annotations: {}
chromeNode:
  annotations: {}
  deploymentEnabled: true
  deregisterLifecycle: null
  dshmVolumeSizeLimit: 1Gi
  enabled: true
  extraEnvFrom: null
  extraEnvironmentVariables:
    - name: SE_OPTS
      value: '--enable-managed-downloads true'
    - name: TZ
      value: Europe/Berlin
    - name: SCREEN_WIDTH
      value: '1920'
    - name: SCREEN_HEIGHT
      value: '1080'
    - name: SCREEN_DEPTH
      value: '24'
    - name: SE_NODE_SESSION_TIMEOUT
      value: '300'
    - name: SE_VNC_NO_PASSWORD
      value: '1'
    - name: SE_VNC_VIEW_ONLY
      value: '1'
  extraVolumeMounts: []
  extraVolumes: []
  hostAliases: null
  hpa:
    browserName: chrome
    platformName: linux
    sessionBrowserName: chrome
    unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}'
    url: '{{ template "seleniumGrid.graphqlURL" . }}'
  imageName: node-chrome
  imagePullPolicy: IfNotPresent
  imagePullSecret: ''
  initContainers: []
  labels: {}
  lifecycle: {}
  livenessProbe:
    enabled: false
    failureThreshold: 6
    initialDelaySeconds: 30
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 60
  nodePort: null
  nodeSelector: {}
  port: 5555
  ports: []
  priorityClassName: ''
  readinessProbe:
    enabled: false
    failureThreshold: 10
    initialDelaySeconds: 10
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  replicas: 60
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  securityContext: {}
  service:
    annotations: {}
    enabled: true
    loadBalancerIP: ''
    ports: null
    type: ClusterIP
  sidecars: []
  startupProbe:
    enabled: true
    failureThreshold: 12
    initialDelaySeconds: 0
    path: /status
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 60
  terminationGracePeriodSeconds: 30
  tolerations: []
  scaledOptions:
    maxReplicaCount: 20
    minReplicaCount: 5
    pollingInterval: 10
components:
  distributor:
    annotations: {}
    imageName: distributor
    imagePullPolicy: IfNotPresent
    imagePullSecret: ''
    livenessProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 10
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    newSessionThreadPoolSize: null
    nodePort: 30553
    nodeSelector: {}
    port: 5553
    priorityClassName: ''
    readinessProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 12
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    resources: {}
    securityContext: {}
    serviceAnnotations: {}
    serviceType: ClusterIP
    startupProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 5
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    tolerations: []
  eventBus:
    annotations: {}
    imageName: event-bus
    imagePullPolicy: IfNotPresent
    imagePullSecret: ''
    nodePort: 30557
    nodeSelector: {}
    port: 5557
    priorityClassName: ''
    publishNodePort: 30442
    publishPort: 4442
    resources: {}
    securityContext: {}
    serviceAnnotations: {}
    serviceType: ClusterIP
    subscribeNodePort: 30443
    subscribePort: 4443
    tolerations: []
  extraEnvFrom: null
  extraEnvironmentVariables:
    - name: SE_SESSION_REQUEST_TIMEOUT
      value: '3600'
  router:
    annotations: {}
    disableUI: false
    imageName: router
    imagePullPolicy: IfNotPresent
    imagePullSecret: ''
    livenessProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 10
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    loadBalancerIP: ''
    nodePort: 30444
    nodeSelector: {}
    port: 4444
    priorityClassName: ''
    readinessProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 12
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    resources: {}
    securityContext: {}
    serviceAnnotations: {}
    serviceType: ClusterIP
    startupProbe:
      enabled: true
      failureThreshold: 10
      initialDelaySeconds: 5
      path: /readyz
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    tolerations: []
  sessionMap:
    annotations: {}
    imageName: sessions
    imagePullPolicy: IfNotPresent
    imagePullSecret: ''
    nodeSelector: {}
    port: 5556
    priorityClassName: ''
    resources: {}
    securityContext: {}
    serviceAnnotations: {}
    serviceType: ClusterIP
    tolerations: []
  sessionQueue:
    annotations: {}
    imageName: session-queue
    imagePullPolicy: IfNotPresent
    imagePullSecret: ''
    nodePort: 30559
    nodeSelector: {}
    port: 5559
    priorityClassName: ''
    resources: {}
    securityContext: {}
    serviceAnnotations: {}
    serviceType: ClusterIP
    tolerations: []
  subPath: ''
customLabels: {}
distributorConfigMap:
  annotations: {}
  defaultMode: 493
  extraScripts:
    distributorProbe.sh: ''
  extraScriptsDirectory: /opt/selenium
  extraScriptsImportFrom: configs/distributor/**
  scriptVolumeMountName: null
edgeNode:
  annotations: {}
  deploymentEnabled: true
  deregisterLifecycle: null
  dshmVolumeSizeLimit: 1Gi
  enabled: true
  extraEnvFrom: null
  extraEnvironmentVariables:
    - name: SE_OPTS
      value: '--enable-managed-downloads true'
    - name: TZ
      value: Europe/Berlin
    - name: SCREEN_WIDTH
      value: '1920'
    - name: SCREEN_HEIGHT
      value: '1080'
    - name: SCREEN_DEPTH
      value: '24'
    - name: SE_NODE_SESSION_TIMEOUT
      value: '330'
    - name: SE_VNC_NO_PASSWORD
      value: '1'
    - name: SE_VNC_VIEW_ONLY
      value: '1'
  extraVolumeMounts: []
  extraVolumes: []
  hostAliases: null
  hpa:
    browserName: MicrosoftEdge
    platformName: linux
    sessionBrowserName: msedge
    unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}'
    url: '{{ template "seleniumGrid.graphqlURL" . }}'
  imageName: node-edge
  imagePullPolicy: IfNotPresent
  imagePullSecret: ''
  initContainers: []
  labels: {}
  lifecycle: {}
  livenessProbe:
    enabled: false
    failureThreshold: 6
    initialDelaySeconds: 30
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 60
  nodePort: null
  nodeSelector: {}
  port: 5555
  ports: []
  priorityClassName: ''
  readinessProbe:
    enabled: false
    failureThreshold: 10
    initialDelaySeconds: 10
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  replicas: 0
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  securityContext: {}
  service:
    annotations: {}
    enabled: true
    loadBalancerIP: ''
    ports: null
    type: ClusterIP
  sidecars: []
  startupProbe:
    enabled: true
    failureThreshold: 12
    initialDelaySeconds: 0
    path: /status
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 60
  terminationGracePeriodSeconds: 30
  tolerations: []
  scaledOptions:
    maxReplicaCount: 2
    minReplicaCount: 0
    pollingInterval: 20
firefoxNode:
  annotations: {}
  deploymentEnabled: true
  deregisterLifecycle: null
  dshmVolumeSizeLimit: 1Gi
  enabled: true
  extraEnvFrom: null
  extraEnvironmentVariables:
    - name: SE_OPTS
      value: '--enable-managed-downloads true'
    - name: TZ
      value: Europe/Berlin
    - name: SCREEN_WIDTH
      value: '1920'
    - name: SCREEN_HEIGHT
      value: '1080'
    - name: SCREEN_DEPTH
      value: '24'
    - name: SE_NODE_SESSION_TIMEOUT
      value: '300'
    - name: SE_VNC_NO_PASSWORD
      value: '1'
    - name: SE_VNC_VIEW_ONLY
      value: '1'
  extraVolumeMounts: []
  extraVolumes: []
  hostAliases: null
  hpa:
    browserName: firefox
    platformName: linux
    sessionBrowserName: firefox
    unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}'
    url: '{{ template "seleniumGrid.graphqlURL" . }}'
  imageName: node-firefox
  imagePullPolicy: IfNotPresent
  imagePullSecret: ''
  initContainers: []
  labels: {}
  lifecycle: {}
  livenessProbe:
    enabled: false
    failureThreshold: 6
    initialDelaySeconds: 30
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 60
  nodePort: null
  nodeSelector: {}
  port: 5555
  ports: []
  priorityClassName: ''
  readinessProbe:
    enabled: false
    failureThreshold: 10
    initialDelaySeconds: 10
    path: /status
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  replicas: 0
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  securityContext: {}
  service:
    annotations: {}
    enabled: true
    loadBalancerIP: ''
    ports: null
    type: ClusterIP
  sidecars: []
  startupProbe:
    enabled: true
    failureThreshold: 12
    initialDelaySeconds: 0
    path: /status
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 60
  terminationGracePeriodSeconds: 30
  tolerations: []
  scaledOptions:
    maxReplicaCount: 2
    minReplicaCount: 0
    pollingInterval: 20
global:
  K8S_PUBLIC_IP: ''
  seleniumGrid:
    defaultComponentLivenessProbe: exec
    defaultNodeLivenessProbe: exec
    defaultNodeStartupProbe: exec
    imagePullSecret: ''
    imageRegistry: selenium
    imageTag: 4.24.0
    kubectlImage: bitnami/kubectl:latest
    logLevel: INFO
    nodesImageTag: 4.24.0
    stdoutProbeLog: false
    videoImageTag: ffmpeg-6.1.1-20240621
  cattle:
    systemProjectId: p-rdbwg
hub:
  annotations: {}
  disableUI: false
  extraEnvFrom: null
  extraEnvironmentVariables:
    - name: SE_SESSION_REQUEST_TIMEOUT
      value: '3600'
  extraVolumeMounts: []
  extraVolumes: []
  imageName: hub
  imagePullPolicy: IfNotPresent
  imagePullSecret: ''
  labels: {}
  livenessProbe:
    enabled: true
    failureThreshold: 10
    initialDelaySeconds: 10
    path: /readyz
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  loadBalancerIP: ''
  newSessionThreadPoolSize: null
  nodePort: 31444
  nodeSelector: {}
  port: 4444
  priorityClassName: ''
  publishNodePort: 31442
  publishPort: 4442
  readinessProbe:
    enabled: true
    failureThreshold: 10
    initialDelaySeconds: 12
    path: /readyz
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  resources: {}
  securityContext: {}
  serviceAnnotations: {}
  serviceType: ClusterIP
  startupProbe:
    enabled: true
    failureThreshold: 10
    initialDelaySeconds: 5
    path: /readyz
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 10
  subPath: /
  subscribeNodePort: 31443
  subscribePort: 4443
  tolerations: []
ingress:
  annotations:
    haproxy-ingress.github.io/proxy-body-size: 40m
    haproxy-ingress.github.io/rewrite-target: /
    haproxy-ingress.github.io/timeout-client: 3600s
    haproxy-ingress.github.io/timeout-connect: 3600s
    haproxy-ingress.github.io/timeout-http-request: 3600s
    haproxy-ingress.github.io/timeout-keep-alive: 3600s
    haproxy-ingress.github.io/timeout-server: 3600s
  className: haproxydefault
  enabled: true
  hostname: our.hostname
  nginx:
    proxyBuffer:
      number: 4
      size: 512M
    proxyTimeout: 3600
  path: /
  pathType: Prefix
  paths: []
  ports:
    http: 80
    https: 443
  tls: []
ingress-nginx:
  controller:
    admissionWebhooks:
      enabled: false
  enabled: false
isolateComponents: true
jaeger:
  agent:
    enabled: false
  allInOne:
    enabled: true
    extraEnv:
      - name: QUERY_BASE_PATH
        value: /jaeger
  collector:
    enabled: false
  provisionDataStore:
    cassandra: false
  query:
    enabled: false
  storage:
    type: badger
keda:
  additionalAnnotations: null
  http:
    timeout: 60000
  webhooks:
    enabled: false
loggingConfigMap:
  annotations: {}
monitoring:
  enabled: false
nodeConfigMap:
  annotations: {}
  defaultMode: 493
  extraScripts:
    nodePreStop.sh: ''
    nodeProbe.sh: ''
  extraScriptsDirectory: /opt/selenium
  extraScriptsImportFrom: configs/node/**
  leftoversCleanup:
    browserElapsedTimeInSecs: 7200
    enabled: false
    jobIntervalInSecs: 3600
    tmpFilesAfterDays: 1
  scriptVolumeMountName: null
rbacRole:
  annotations: {}
  rules:
    - apiGroups:
        - keda.sh
      resources:
        - scaledjobs
      verbs:
        - get
        - list
        - patch
        - update
        - delete
    - apiGroups:
        - keda.sh
      resources:
        - scaledobjects
      verbs:
        - get
        - list
        - patch
        - update
        - delete
    - apiGroups:
        - autoscaling
      resources:
        - horizontalpodautoscalers
      verbs:
        - get
        - list
        - patch
        - update
        - delete
rbacRoleBinding:
  annotations: {}
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
recorderConfigMap:
  annotations: {}
  defaultMode: 493
  extraScripts: null
  extraScriptsDirectory: /opt/bin
  extraScriptsImportFrom: configs/recorder/**
  scriptVolumeMountName: null
  videoVolumeMountName: videos
routerConfigMap:
  annotations: {}
  defaultMode: 493
  extraScripts:
    routerProbe.sh: ''
  extraScriptsDirectory: /opt/selenium
  extraScriptsImportFrom: configs/router/**
  scriptVolumeMountName: null
secrets:
  annotations: {}
  create: true
  env:
    SE_VNC_PASSWORD: secret
serverConfigMap:
  annotations: {}
  certVolumeMountPath: /etc/ssl/certs/selenium
  certificateFile: selenium.pem
  disableHostnameVerification: true
  env:
    SE_JAVA_OPTS: '-XX:+UseZGC -Djdk.httpclient.keepalive.timeout=600'
  privateKeyFile: selenium.pkcs8
  trustStoreFile: selenium.jks
serviceAccount:
  annotations: {}
  create: true
tls:
  certificate: null
  defaultFile:
    certificate: certs/selenium.pem
    privateKey: certs/selenium.pkcs8.base64
    trustStore: certs/selenium.jks
  enabled: false
  ingress:
    defaultCN: www.selenium.dev
    defaultDays: 3650
    defaultIPList: []
    defaultName: SeleniumHQ
    defaultSANList: []
    generateTLS: false
  privateKey: null
  registrationSecret:
    enabled: false
    value: HappyTesting
  trustStore: null
  trustStorePassword: changeit
tracing:
  enabled: false
  enabledWithExistingEndpoint: false
  exporter: otlp
  exporterEndpoint: http://{{ .Release.Name }}-jaeger-collector.{{ .Release.Namespace }}:4317
  globalAutoConfigure: true
  ingress:
    annotations: null
    enabled: true
    paths:
      - backend:
          service:
            name: '{{ .Release.Name }}-jaeger-query'
            port:
              number: 16686
        path: /jaeger
        pathType: Prefix
uploaderConfigMap:
  annotations: {}
  defaultMode: 493
  extraScripts:
    upload.sh: ''
  extraScriptsDirectory: /opt/bin
  extraScriptsImportFrom: configs/uploader/**
  scriptVolumeMountName: null
  secretFiles:
    upload.conf: '[sample]'
  secretVolumeMountName: null
videoRecorder:
  enabled: false
  extraEnvFrom: null
  extraEnvironmentVariables: null
  extraVolumeMounts: []
  extraVolumes: []
  imageName: video
  imagePullPolicy: IfNotPresent
  lifecycle: {}
  livenessProbe: {}
  ports:
    - 9000
  resources:
    limits:
      cpu: '1'
      memory: 1Gi
    requests:
      cpu: '1'
      memory: 1Gi
  s3:
    args: []
    command: []
    extraEnvironmentVariables: null
    imageName: bitnami/aws-cli
    imagePullPolicy: IfNotPresent
    imageRegistry: public.ecr.aws
    imageTag: latest
    securityContext:
      runAsUser: 0
  securityContext: null
  startupProbe: {}
  targetFolder: /videos
  terminationGracePeriodSeconds: 30
  uploader:
    configFileName: upload.conf
    destinationPrefix: null
    enabled: false
    entryPointFileName: upload.sh
    name: null
    secrets: null

Relevant log output

Secret "helm-operation-xxxx" is invalid: data: Too long: must have at most 1048576 bytes

Operating System

Kubernetes v1.30.5+rke2r1

Docker Selenium version (image tag)

4.24.0

Selenium Grid chart version (chart version)

0.33.0

[github-actions]

@LQuartier, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

It looks like Helm v2.16.8-rancher2, which version is Helm v2. But we developed and tested chart against Helm v3. You can checkout the list Helm version in release notes.
Is there any dependency that blocking you update to Helm v3 ?

[LQuartier]

Thanks, that may be the piece i'm missing!
I will have to look into the possibility of Upgrading that as, while i'm maintaining the Grid, i'm not maintaining the Kubernetes Environment itself.

[VietND96]

I think Helm v3 is independent with K8s, since Helm v3 is the binary, so you just copy it to /usr/bin or /usr/local/bin, then it is execuable.

[LQuartier]

It turns out, the Helm Version that is shown in the Rancher Frontend is wrong and seems to be a known Bug ->
rancher/rancher#46536

The Actual Version, as shown by Web CLI command "helm version" is "v3.15.1-rancher2"

[LQuartier]

Okay so, after Further investigation:

Installing it via CLI works without issue. The Issue only persists, if i install the Charts via The "Apps" Section in the Rancher Webinterface. It appears that the System Pulls the tgz Files, tries to install those directly, and then runs into an issue like this:
helm/helm#9788
It breaks specifically with the 0.33 and higher (and, having now tried it, the 0.31 as well), because those tgz files are larger than 1048576 Bytes.

[VietND96]

Ok, I searched and saw that limitation in the chart package. Probably in recent versions, we added more sub-charts into the umbrella chart selenium-grid.
So, in your deployment, beside Selenium Grid, any else sub-chart will be installed together?
Actually, I also raised a ticket to split chart, selenium-grid will stay with Grid deploy alone, and selenium-grid-solution is umbrella chart with other sub-charts included #2388. However, I don't have time to complete this.

[emontou]

I am also getting the same error "Secret "helm-operation-xxxx" is invalid: data: Too long: must have at most 1048576 bytes" using rancher ui and upgrading from 0.32.0

I am trying to figure out how to upgrade via CLI to try and bypass it

Update: confirmed that upgrading through the cli and bypassing rancher fixes the issue. Thanks @LQuartier !