syn2mas migration failes on arm64

[bbieberle]

Describe the bug
The sy2mas migration job failed because the container "syn2mas-migrate" fails with this error message:

/tmp-mas-cli/mas-cli: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found

To Reproduce
Steps to reproduce the behavior:

1. enable non-dryrun migration job syn2mas
2. see job fail because the error occurs on the last container

Expected behavior
Migration succeeds

Desktop (please complete the following information):

• OS: Armbian v25.11.2 (Kernel 6.1.115-vendor-rk35xx)
• Chart version: 26.2.1
• image version: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2

Additional context
Migration was run on k3s and a node running Armbian v25.11.2 With Kernel version 6.1.115-vendor-rk35xx

After running it with the added nodeselector for amd64 it finished successfully

[benbz]

This looks and feels like the issue I fixed in #1042, but I believed that only impacted Matrix Authentication Service v1.11.0 which isn't in a release yet.

[benbz]

I'm a bit mystified about this. 26.2.1 uses syn2mas v1.10-debug

$ skopeo copy --multi-arch all docker://ghcr.io/element-hq/matrix-authentication-service:1.10-debug oci:images/mas-1.10
...

$ tar -zxvf images/mas-1.10/blobs/sha256/81784508f5ab00ac5830829d6f8746747275bfdcd503ee3b5e4ebdc2214e0dd5
usr/
usr/local/
usr/local/bin/
usr/local/bin/mas-cli

$ file usr/local/bin/mas-cli
usr/local/bin/mas-cli: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=77f7613be74b0612efdec3a8ec96c708e36e4297, for GNU/Linux 3.7.0, not stripped

$ objdump -T usr/local/bin/mas-cli | grep GLIBC_ | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu                       
2.0
2.17
2.18
2.25
2.28
2.29
2.32
2.33
2.34

Whereas the 1.11 binary has

$ skopeo copy --multi-arch all docker://ghcr.io/element-hq/matrix-authentication-service:1.11-debug oci-archive:mas-1.11
...

$ tar -zxvf images/mas-1.11/blobs/sha256/78da82d90ae927961a61d8f59f21f2b9bd817ea8b459e7f219ff38f14b1bbeb5              
usr/
usr/local/
usr/local/bin/
usr/local/bin/mas-cli

$ file usr/local/bin/mas-cli                                                                            
usr/local/bin/mas-cli: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=301a3363c839ffc19c9bd54e36bb776df1b63232, for GNU/Linux 3.7.0, not stripped

$ objdump -T usr/local/bin/mas-cli | grep GLIBC_ | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu        
2.0
2.17
2.18
2.25
2.28
2.29
2.30
2.32
2.33
2.34
2.38
2.39

You didn't happen to upgrade the syn2mas version to latest or something else did you?

[bbieberle]

I changed no versions in the values.yaml

This is the job template, that was rendered by the chart:
(I pulled it from my cluster after it ran)

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "0"
  creationTimestamp: "2026-02-17T13:27:15Z"
  generation: 1
  labels:
    app.kubernetes.io/component: matrix-authentication
    app.kubernetes.io/instance: ess-syn2mas
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: syn2mas
    app.kubernetes.io/part-of: matrix-stack
    app.kubernetes.io/version: 1.10.0-debug
    helm.sh/chart: matrix-stack-26.2.1_e962743851e2
    k8s.element.io/matrix-authentication-service-config-hash: 
    k8s.element.io/matrix-authentication-service-secret-hash: 
    k8s.element.io/postgres-password-matrixauthenticationserv-hash: 
    k8s.element.io/postgres-password-synapse-hash: 
    k8s.element.io/synapse-config-hash: 
    k8s.element.io/synapse-secret-hash: 
  name: ess-syn2mas
  namespace: ess
  resourceVersion: "335016415"
  uid: 8fd085c8-f56b-4857-92f8-2aa83e95a9d8
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  completions: 1
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: 8fd085c8-f56b-4857-92f8-2aa83e95a9d8
  suspend: false
  template:
    metadata:
      labels:
        app.kubernetes.io/component: matrix-authentication
        app.kubernetes.io/instance: ess-syn2mas
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: syn2mas
        app.kubernetes.io/part-of: matrix-stack
        app.kubernetes.io/version: 1.10.0-debug
        batch.kubernetes.io/controller-uid: 8fd085c8-f56b-4857-92f8-2aa83e95a9d8
        batch.kubernetes.io/job-name: ess-syn2mas
        controller-uid: 8fd085c8-f56b-4857-92f8-2aa83e95a9d8
        job-name: ess-syn2mas
        k8s.element.io/matrix-authentication-service-config-hash: 
        k8s.element.io/matrix-authentication-service-secret-hash: 
        k8s.element.io/postgres-password-matrixauthenticationserv-hash: 
        k8s.element.io/postgres-password-synapse-hash: 
        k8s.element.io/synapse-config-hash: 
        k8s.element.io/synapse-secret-hash: 39
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - syn2mas
        - --config
        - /conf/mas-config.yaml
        - --synapse-config
        - /conf/homeserver.yaml
        env:
        - name: NAMESPACE
          value: ess
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2
        imagePullPolicy: Always
        name: syn2mas-migrate
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /conf
          name: rendered-config
          readOnly: true
        - mountPath: /tmp-mas-cli
          name: tmp-mas-cli
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /secrets/ess-matrix-authentication-service-pre
          name: secret-5bb7f8561337
          readOnly: true
        - mountPath: /secrets/ess-secret
          name: secret-a4d9cd45d8a4
          readOnly: true
        - mountPath: /secrets/ess-synapse-hook
          name: secret-8d6c4f5d7d48
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - cp -v /usr/local/bin/mas-cli /tmp-mas-cli/mas-cli
        image: ghcr.io/element-hq/matrix-authentication-service:1.10.0-debug
        imagePullPolicy: Always
        name: copy-mas-cli
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp-mas-cli
          name: tmp-mas-cli
      - args:
        - render-config
        - -output
        - /conf/mas-config.yaml
        - /config-templates/mas-config-underrides.yaml
        - /secrets/ess-secret/synapse-mas-oidc-config.yaml
        - /secrets/ess-matrix-authentication-service-pre/user-1-passwords
        - /config-templates/mas-config-overrides.yaml
        env:
        - name: POSTGRES_PASSWORD
          value: '{{ readfile "/secrets/ess-secret/synapse-mas-db-password" | urlencode
            }}'
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2
        imagePullPolicy: Always
        name: render-config-mas
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config-templates
          name: plain-mas-config
          readOnly: true
        - mountPath: /secrets/ess-matrix-authentication-service-pre
          name: secret-5bb7f8561337
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /secrets/ess-secret
          name: secret-a4d9cd45d8a4
          readOnly: true
        - mountPath: /conf
          name: rendered-config
      - args:
        - render-config
        - -output
        - /conf/homeserver.yaml
        - /config-templates/01-homeserver-underrides.yaml
        - /secrets/ess-generated/SYNAPSE_EXTRA
        - /secrets/ess-synapse-hook/user-0-element-call
        - /secrets/ess-secret/oidc-config.yaml
        - /config-templates/04-homeserver-overrides.yaml
        - /config-templates/05-main.yaml
        env:
        - name: APPLICATION_NAME
          value: '{{ hostname }}'
        - name: SYNAPSE_POSTGRES_PASSWORD
          value: '{{ readfile "/secrets/ess-secret/synapse-db-password" | quote }}'
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2
        imagePullPolicy: Always
        name: render-config-syn
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config-templates
          name: plain-syn-config
          readOnly: true
        - mountPath: /secrets/ess-synapse-hook
          name: secret-8d6c4f5d7d48
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /secrets/ess-secret
          name: secret-a4d9cd45d8a4
          readOnly: true
        - mountPath: /conf
          name: rendered-config
      - args:
        - tcpwait
        - -address
        - truenas.lindbergh.local:5432
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2
        imagePullPolicy: Always
        name: db-wait-mas
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - tcpwait
        - -address
        - truenas.lindbergh.local:5432
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2
        imagePullPolicy: Always
        name: db-wait-syn
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - syn2mas
        - check
        - --config
        - /conf/mas-config.yaml
        - --synapse-config
        - /conf/homeserver.yaml
        image: ghcr.io/element-hq/matrix-authentication-service:1.10.0-debug
        imagePullPolicy: Always
        name: syn2mas-check
        resources:
          limits:
            memory: 350Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /conf
          name: rendered-config
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /secrets/ess-matrix-authentication-service-pre
          name: secret-5bb7f8561337
          readOnly: true
        - mountPath: /secrets/ess-secret
          name: secret-a4d9cd45d8a4
          readOnly: true
        - mountPath: /secrets/ess-synapse-hook
          name: secret-8d6c4f5d7d48
          readOnly: true
      nodeSelector:
        kubernetes.io/arch: amd64
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 10005
        runAsGroup: 10005
        runAsNonRoot: true
        runAsUser: 10005
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: ess-syn2mas
      serviceAccountName: ess-syn2mas
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: ess-syn2mas
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - configMap:
          defaultMode: 420
          name: ess-synapse-hook
        name: plain-syn-config
      - configMap:
          defaultMode: 420
          name: ess-matrix-authentication-service-pre
        name: plain-mas-config
      - name: secret-eda259c754a3
        secret:
          defaultMode: 420
          secretName: ess-generated
      - name: secret-5bb7f8561337
        secret:
          defaultMode: 420
          secretName: ess-matrix-authentication-service-pre
      - name: secret-a4d9cd45d8a4
        secret:
          defaultMode: 420
          secretName: ess-secret
      - name: secret-8d6c4f5d7d48
        secret:
          defaultMode: 420
          secretName: ess-synapse-hook
      - emptyDir:
          medium: Memory
        name: rendered-config
      - emptyDir:
          medium: Memory
        name: tmp-mas-cli
status:
  completionTime: "2026-02-17T13:27:52Z"
  conditions:
  - lastProbeTime: "2026-02-17T13:27:52Z"
    lastTransitionTime: "2026-02-17T13:27:52Z"
    message: Reached expected number of succeeded pods
    reason: CompletionsReached
    status: "True"
    type: SuccessCriteriaMet
  - lastProbeTime: "2026-02-17T13:27:52Z"
    lastTransitionTime: "2026-02-17T13:27:52Z"
    message: Reached expected number of succeeded pods
    reason: CompletionsReached
    status: "True"
    type: Complete
  ready: 0
  startTime: "2026-02-17T13:27:15Z"
  succeeded: 1
  terminating: 0
  uncountedTerminatedPods: {}

[bbieberle]

I think it uses the version 0.7.2 because right in the beginning of the helm values the version is set to 0.7.2 for chart version 26.2.1:
https://github.com/element-hq/ess-helm/blob/26.2.1/charts/matrix-stack/values.yaml#L24

this is totally unreleated to the sys2mas or mas version, isn't it?

maybe ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2 ships with an older version of mas-cli

[benbz]

I think it uses the version 0.7.2 because right in the beginning of the helm values the version is set to 0.7.2 for chart version 26.2.1: https://github.com/element-hq/ess-helm/blob/26.2.1/charts/matrix-stack/values.yaml#L24

this is totally unreleated to the sys2mas or mas version, isn't it?

maybe ghcr.io/element-hq/ess-helm/matrix-tools:0.7.2 ships with an older version of mas-cli

The 1st initContainer (copy-mas-cli) copies the mas-cli from MAS 1.10.0-debug into an emptyDir that's then used by the syn2mas-migrate container. This was done specifically to avoid having to embed the mas-cli into the matrix-tools image, allowing the 2 to be upgraded independently.