Only first user detected when adding local users in concourse helm chart

[mbarkley]

Summary

I am trying to configure a local user for a concourse deployment running in GKE, installed via the concourse helm chart. The first user in my list is correctly configured, but the second one does not get configured at all.

Steps to Reproduce

1. Install concourse in a Kubernetes cluster with helm
   1. Include this in your values:

      secrets:
        create: false

   2. Create a k8s secret (ours is called concourse-web) and add the attribute local-users with this content:

      concourse-admin:password123,other-user:otherpassword
      

   3. Run helm install
2. Install the Fly CLI and try to log in as the second local user

   fly -t test login --concourse-url 'https://concourse.example.com' --username other-user --password other-password
   

Expected Results

I should successfully sign in as the second local user when running fly login.

Actual Results

When I run fly login with the second user's credentials, I get this error:

error: oauth2: "access_denied" "Invalid username or password"

I confirmed that the problem was not with the second user's name or password by swapping the order and confirming that I could log in when the second user was put first in the list.

Additionally, when running fly active-users, I do not see the second user in the list, but I do see an "empty" user shown:

concourse-admin    local       2025-09-09
                                         2025-09-12

Web Node(s) configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "69"
  creationTimestamp: "2024-03-18T21:00:15Z"
  generation: 71
  labels:
    app: concourse-web
    argocd.argoproj.io/instance: concourse
    chart: concourse-18.4.1
    heritage: Helm
    release: concourse
  name: concourse-web
  namespace: concourse
  resourceVersion: "1757707611221823007"
  uid: 6088ee19-a206-4949-b60c-7ed9cbe7aebb
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: concourse-web
      release: concourse
  strategy:
    rollingUpdate:
      maxSurge: 26%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        checksum/secrets: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        kubectl.kubernetes.io/restartedAt: "2025-09-12T20:06:48Z"
      creationTimestamp: null
      labels:
        app: concourse-web
        release: concourse
    spec:
      containers:
      - args:
        - web
        env:
        - name: CONCOURSE_OIDC_SCOPE
          value: openid,email,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/userinfo.profile,openid,https://www.googleapis.com/auth/cloud-platform
        - name: CONCOURSE_CLUSTER_NAME
          value: dnastack-concourse
        - name: CONCOURSE_PAUSE_PIPELINES_AFTER
          value: "90"
        - name: CONCOURSE_ENABLE_GLOBAL_RESOURCES
          value: "true"
        - name: CONCOURSE_ENABLE_ACROSS_STEP
          value: "true"
        - name: CONCOURSE_SECRET_RETRY_ATTEMPTS
          value: "5"
        - name: CONCOURSE_SECRET_RETRY_INTERVAL
          value: 1s
        - name: CONCOURSE_LOG_LEVEL
          value: info
        - name: CONCOURSE_BIND_PORT
          value: "8080"
        - name: CONCOURSE_BIND_IP
          value: 0.0.0.0
        - name: CONCOURSE_ADD_LOCAL_USER
          valueFrom:
            secretKeyRef:
              key: local-users
              name: concourse-web
        - name: CONCOURSE_EXTERNAL_URL
          value: https://concourse.example.com # not actual URL
        - name: CONCOURSE_DEBUG_BIND_IP
          value: 127.0.0.1
        - name: CONCOURSE_DEBUG_BIND_PORT
          value: "8079"
        - name: CONCOURSE_INTERCEPT_IDLE_TIMEOUT
          value: 0m
        - name: CONCOURSE_GLOBAL_RESOURCE_CHECK_TIMEOUT
          value: 1h
        - name: CONCOURSE_RESOURCE_CHECKING_INTERVAL
          value: 1m
        - name: CONCOURSE_RESOURCE_TYPE_CHECKING_INTERVAL
          value: 1m
        - name: CONCOURSE_RESOURCE_WITH_WEBHOOK_CHECKING_INTERVAL
          value: 1m
        - name: CONCOURSE_CONTAINER_PLACEMENT_STRATEGY
          value: volume-locality
        - name: CONCOURSE_BAGGAGECLAIM_RESPONSE_HEADER_TIMEOUT
          value: 1m
        - name: CONCOURSE_BUILD_TRACKER_INTERVAL
          value: 10s
        - name: CONCOURSE_DB_NOTIFICATION_BUS_QUEUE_SIZE
          value: "10000"
        - name: CONCOURSE_POSTGRES_HOST
          value: postgres
        - name: CONCOURSE_POSTGRES_PORT
          value: "5432"
        - name: CONCOURSE_POSTGRES_USER
          valueFrom:
            secretKeyRef:
              key: postgresql-user
              name: concourse-web
        - name: CONCOURSE_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgresql-password
              name: concourse-web
        - name: CONCOURSE_POSTGRES_SSLMODE
          value: disable
        - name: CONCOURSE_POSTGRES_CONNECT_TIMEOUT
          value: 5m
        - name: CONCOURSE_POSTGRES_DATABASE
          value: atc
        - name: CONCOURSE_KUBERNETES_IN_CLUSTER
          value: "true"
        - name: CONCOURSE_KUBERNETES_NAMESPACE_PREFIX
          value: concourse-
        - name: CONCOURSE_DATADOG_AGENT_HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: CONCOURSE_DATADOG_AGENT_PORT
          value: "8125"
        - name: CONCOURSE_DATADOG_PREFIX
          value: concourse.ci
        - name: CONCOURSE_GC_INTERVAL
          value: 30s
        - name: CONCOURSE_GC_ONE_OFF_GRACE_PERIOD
          value: 5m
        - name: CONCOURSE_GC_MISSING_GRACE_PERIOD
          value: 5m
        - name: CONCOURSE_AUTH_DURATION
          value: 24h
        - name: CONCOURSE_SESSION_SIGNING_KEY
          value: /concourse-keys/session_signing_key
        - name: CONCOURSE_MAIN_TEAM_LOCAL_USER
          value: concourse-admin
        - name: CONCOURSE_MAIN_TEAM_OIDC_USER
          value: some-emails@example.com # not actual list
        - name: CONCOURSE_OIDC_DISPLAY_NAME
          value: Google
        - name: CONCOURSE_OIDC_ISSUER
          value: https://accounts.google.com
        - name: CONCOURSE_OIDC_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: oidc-client-id
              name: concourse-web
        - name: CONCOURSE_OIDC_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: oidc-client-secret
              name: concourse-web
        - name: CONCOURSE_OIDC_GROUPS_KEY
          value: hd
        - name: CONCOURSE_OIDC_USER_NAME_KEY
          value: email
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: CONCOURSE_PEER_ADDRESS
          value: $(POD_IP)
        - name: CONCOURSE_TSA_LOG_LEVEL
          value: info
        - name: CONCOURSE_TSA_BIND_IP
          value: 0.0.0.0
        - name: CONCOURSE_TSA_BIND_PORT
          value: "2222"
        - name: CONCOURSE_TSA_DEBUG_BIND_IP
          value: 127.0.0.1
        - name: CONCOURSE_TSA_DEBUG_BIND_PORT
          value: "2221"
        - name: CONCOURSE_TSA_HOST_KEY
          value: /concourse-keys/host_key
        - name: CONCOURSE_TSA_AUTHORIZED_KEYS
          value: /concourse-keys/worker_key.pub
        - name: CONCOURSE_TSA_HEARTBEAT_INTERVAL
          value: 30s
        image: concourse/concourse:7.14.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /api/v1/info
            port: atc
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 3
        name: concourse-web
        ports:
        - containerPort: 8080
          name: atc
          protocol: TCP
        - containerPort: 2222
          name: tsa
          protocol: TCP
        - containerPort: 8079
          name: atc-debug
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/v1/info
            port: atc
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: "1"
            memory: 2Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /concourse-keys
          name: concourse-keys
          readOnly: true
        - mountPath: /concourse-auth
          name: auth-keys
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - migrate
        - --migrate-to-latest-version
        env:
        - name: CONCOURSE_POSTGRES_HOST
          value: postgres
        - name: CONCOURSE_POSTGRES_PORT
          value: "5432"
        - name: CONCOURSE_POSTGRES_USER
          valueFrom:
            secretKeyRef:
              key: postgresql-user
              name: concourse-web
        - name: CONCOURSE_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgresql-password
              name: concourse-web
        - name: CONCOURSE_POSTGRES_SSLMODE
          value: disable
        - name: CONCOURSE_POSTGRES_CONNECT_TIMEOUT
          value: 5m
        - name: CONCOURSE_POSTGRES_DATABASE
          value: atc
        image: concourse/concourse:7.14.1
        imagePullPolicy: IfNotPresent
        name: concourse-migration
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: concourse-web
      serviceAccountName: concourse-web
      terminationGracePeriodSeconds: 30
      volumes:
      - name: concourse-keys
        secret:
          defaultMode: 256
          items:
          - key: host-key
            path: host_key
          - key: session-signing-key
            path: session_signing_key
          - key: worker-key-pub
            path: worker_key.pub
          secretName: concourse-web
      - name: auth-keys
        secret:
          defaultMode: 256
          secretName: concourse-web
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2025-09-05T08:07:56Z"
    lastUpdateTime: "2025-09-05T08:07:56Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-08-21T20:37:36Z"
    lastUpdateTime: "2025-09-12T20:06:51Z"
    message: ReplicaSet "concourse-web-7845bb58dc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 71
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Worker(s) configuration

No response

Concourse Version

7.14.1

Browser (if applicable)

No response

Did this use to work?

Not sure, never tried it before

[taylorsilva]

This is a configuration issue. I know you made your secrets manually, but I first tried reproducing with secrets.create set to true and settings secrets.localUsers to concourse-admin:password123,other-user:otherpassword. I was able to login as both users.

I decided to look at how Concourse creates the localUsers secret. We can see that here:

concourse-chart/templates/web-secrets.yaml

Line 41 in 23d4341

local-users: {{ .Values.secrets.localUsers | b64enc | quote }}

Nothing unusual there. String gets base64 encoded, no other parsing done to it.

The only thing I can guess is that there's an issue with the created secret. Can you share the yaml you used to create the secret?

[mbarkley]

Hi @taylorsilva, thanks for the quick response. I don't think there's any issue with our secret, but just in case here are the values of the relevant environment variables in my concourse-web pod (modulo passwords):

CONCOURSE_MAIN_TEAM_LOCAL_USER=concourse-admin
CONCOURSE_ADD_LOCAL_USER=concourse-admin:hex_password,platformAutomation:hex_password

I will ask around to find out how this secret was generated, but I unfortunately don't have that information at hand.

[taylorsilva]

Are you bcrypt'ing the passwords? Wondering what you mean by hex_password.

[mbarkley]

Are you bcrypt'ing the passwords? Wondering what you mean by hex_password.

No we're not bcrypting them. I actually misspoke in that they're not both generated in the same way. The concourse-admin password looks like it might have come from a password manager (just a random alpha-numeric string). I'm guessing it's not important but the password for the second user that's not being picked up correctly is a random string of hex digits generated with openssl rand -hex 16.

[taylorsilva]

Interesting. That should result in a safe string for parsing (no commas).

I'm not sure what else it could be at this point. Sorry!