Login (LDAP) throws error (panic - securecookie: hash key is not set) after upgrade from v2.9.45 to v2.9.58

[gschafra]

Running in docker environment (docker-compose)

semaphore-1  | 2024/03/26 11:12:11 http: panic serving 172.19.0.1:52638: securecookie: hash key is not set
semaphore-1  | goroutine 49 [running]:
semaphore-1  | net/http.(*conn).serve.func1()
semaphore-1  |  /usr/local/go/src/net/http/server.go:1868 +0xb9
semaphore-1  | panic({0xbfd520?, 0xc0001d3b00?})
semaphore-1  |  /usr/local/go/src/runtime/panic.go:920 +0x270
semaphore-1  | github.com/ansible-semaphore/semaphore/api.createSession({0x1e6cd80, 0xc0001ec0e0}, 0xc0001a3800, {0x2, {0x0, 0xedcb21188, 0x0}, {0xc000290cd8, 0x3}, {0xc000038210, ...}, ...})
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/login.go:151 +0x491
semaphore-1  | github.com/ansible-semaphore/semaphore/api.login({0x1e6cd80, 0xc0001ec0e0}, 0xc0001a3800)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/login.go:294 +0x5d4
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xbdc580?, {0x1e6cd80?, 0xc0001ec0e0?}, 0xc?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/ansible-semaphore/semaphore/api.JSONMiddleware.func1({0x1e6cd80, 0xc0001ec0e0}, 0x45442b?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:44 +0xf6
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xb2de00?, {0x1e6cd80?, 0xc0001ec0e0?}, 0x774ccc?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1.1()
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:35 +0x28
semaphore-1  | github.com/ansible-semaphore/semaphore/db.StoreSession({0x1e79110, 0xc000124690}, {0xc0004d6330, 0xc}, 0xc0004f57b8)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/db/Store.go:432 +0x5f
semaphore-1  | github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1({0x1e6cd80?, 0xc0001ec0e0}, 0xc0001a3800)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:34 +0xf5
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc0001a3800?, {0x1e6cd80?, 0xc0001ec0e0?}, 0xbe6f20?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/ansible-semaphore/semaphore/cli/cmd.runService.func1.1({0x1e6cd80, 0xc0001ec0e0}, 0xc0004b0900?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/root.go:74 +0xea
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc000580000?, {0x1e6cd80?, 0xc0001ec0e0?}, 0xc0004f58e8?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/ansible-semaphore/semaphore/api.Route.CORSMethodMiddleware.func1.1({0x1e6cd80, 0xc0001ec0e0}, 0xc0004b0870?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/middleware.go:51 +0x88
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc0001a3700?, {0x1e6cd80?, 0xc0001ec0e0?}, 0xc0004f5990?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001600c0, {0x1e6cd80, 0xc0001ec0e0}, 0xc0001a3500)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/mux.go:212 +0x1c5
semaphore-1  | github.com/ansible-semaphore/semaphore/cli/cmd.runService.ProxyHeaders.func2({0x1e6cd80, 0xc0001ec0e0}, 0xc0001a3500)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/handlers/proxy_headers.go:59 +0x143
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc000298005?, {0x1e6cd80?, 0xc0001ec0e0?}, 0x10?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | github.com/ansible-semaphore/semaphore/cli/cmd.runService.cropTrailingSlashMiddleware.func3({0x1e6cd80, 0xc0001ec0e0}, 0xc0001a3500)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/server.go:27 +0xbd
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0x4107c5?, {0x1e6cd80?, 0xc0001ec0e0?}, 0xc0001ec001?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2136 +0x29
semaphore-1  | net/http.serverHandler.ServeHTTP({0x1e6a850?}, {0x1e6cd80?, 0xc0001ec0e0?}, 0x6?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2938 +0x8e
semaphore-1  | net/http.(*conn).serve(0xc000090000, {0x1e6eb78, 0xc000293a70})
semaphore-1  |  /usr/local/go/src/net/http/server.go:2009 +0x5f4
semaphore-1  | created by net/http.(*Server).Serve in goroutine 1
semaphore-1  |  /usr/local/go/src/net/http/server.go:3086 +0x5cb

docker-compose.yml:

services:
  db:
    restart: unless-stopped
    image: mariadb:10.11
    ports:
      - 3307:3306
    volumes:
      - semaphore-db:/var/lib/mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: semaphore
      MYSQL_USER: semaphore
      MYSQL_PASSWORD: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  semaphore:
    restart: unless-stopped
    image: registry-gitlab.itx.de/docker-prod/semaphore:2.9.58
    ports:
      - 3000:3000
    environment:
      SEMAPHORE_WEB_ROOT: http://localhost:3000
      SEMAPHORE_DB_USER: semaphore
      SEMAPHORE_DB_PASS: XXXXXXXXXXXXXXXXXXXXXXX
      SEMAPHORE_DB_HOST: db
      SEMAPHORE_DB_PORT: 3306
      SEMAPHORE_DB_DIALECT: mysql
      SEMAPHORE_DB: semaphore
      SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
      SEMAPHORE_EMAIL_HOST: 10.0.0.20
      SEMAPHORE_EMAIL_PORT: 25
      SEMAPHORE_EMAIL_SENDER: [email protected]
      SEMAPHORE_ADMIN_PASSWORD: XXXXXXXXXXXXXXXXXXXXXX
      SEMAPHORE_ADMIN_NAME: admin
      SEMAPHORE_ADMIN_EMAIL: [email protected]
      SEMAPHORE_ADMIN: admin
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: XXXXXXXXXXXXXXXXXXXXXXXXX # command: head -c32 /dev/urandom | base64
      # Deactivated due to bug https://github.com/ansible-semaphore/semaphore/issues/419
      # see https://github.com/ansible-semaphore/semaphore/discussions/1328
      # see https://github.com/ansible-semaphore/semaphore/issues/1276
      # SEMAPHORE_LDAP_ACTIVATED: 'yes'
      # SEMAPHORE_LDAP_HOST: dc1.XXXX.XX
      # SEMAPHORE_LDAP_PORT: '389'
      # SEMAPHORE_LDAP_NEEDTLS: 'no'
      # SEMAPHORE_LDAP_DN_BIND: 'CN=Service Semaphore,OU=XXXX,OU=XXX,DC=XXX,DC=XXX'
      # SEMAPHORE_LDAP_PASSWORD: 'XXXXXXX'
      # SEMAPHORE_LDAP_DN_SEARCH: 'OU=Users,OU=XXXX,DC=XXXXX,DC=XXXXX'
      # SEMAPHORE_LDAP_SEARCH_FILTER: "(\u0026(sAMAccountName=%s)(memberOf=cn=XXXXX,cn=XXXX,cn=XXXX,cn=local))"
      # SEMAPHORE_LDAP_MAPPING_USERNAME: 'sAMAccountName'
      ANSIBLE_HOST_KEY_CHECKING: 'false'
    volumes:
      - type: bind
        source: ./semaphore/config.json
        target: /etc/semaphore/config.json
    depends_on:
      - db
    labels:
      - "diun.enable=true"
      - "diun.hub_link=https://hub.docker.com/r/semaphoreui/semaphore"
      - "diun.platform=linux/amd64"
volumes:
  semaphore-db:

config.json

{
    "tmp_path": "/tmp/semaphore",
    "access_key_encryption": "ENCRYPTION_KEY",
    "mysql": {
        "host": "db",
        "user": "semaphore",
        "pass": "XXXXXXXXXXXXXXXXXXXXX",
        "name": "semaphore",
        "options": {}
    },
    "dialect": "mysql",
    "ldap_enable": true,
    "ldap_needtls": false,
    "ldap_binddn": "CN=Service Semaphore,OU=XXXXX,OU=XXXXX,DC=XXXX,DC=local",
    "ldap_bindpassword": "XXXXXX",
    "ldap_server": "dc1.XXXXXXX:389",
    "ldap_searchdn": "OU=Users,OU=XXXX,DC=XXXX,DC=XXXXX",
    "ldap_searchfilter": "(&(sAMAccountName=%s)(memberOf=CN=XXXXXX,CN=Users,DC=XXXXX,DC=XXXXXX))",
    "ldap_mappings": {
      "dn": "dn",
      "mail": "mail",
      "uid": "sAMAccountName",
      "cn": "cn"
    },
    "email_sender": "[email protected]",
    "email_host": "10.0.0.20",
    "email_port": "25",
    "email_alert": true,
    "web_host": "http://localhost:3000"
}

[gschafra]

May be related to https://stackoverflow.com/questions/78020837/gorilla-session-package-error-securecookie-hash-key-is-not-set:

This line is getting a key from ENV variable named SESSION_KEY:

var Store = sessions.NewCookieStore([]byte(os.Getenv("SESSION_KEY")))
Most probably this variable is not set and that's why you get the error.

If you're running the program from terminal, check if it's there:

env | grep SESSION_KEY
If nothing comes out, then it's not and you should set it:

export SESSION_KEY="something"

But setting this env variable does NOT fix the issue

[fiftin]

Hi @gschafra ,

It is because SEMAPHORE_COOKIE_ENCRYPTION has been updated. Can you sign out from the account?

SEMAPHORE_COOKIE_ENCRYPTION should be provided via environment variables like SEMAPHORE_ACCESS_KEY_ENCRYPTION. Otherwise it will be regenerated each time when you update docker container.

[gschafra]

After logging out (wasn't even logged in any more), clearing the browser data, restarting the docker compose services and trying to log in again I'm still getting a 502/500 (502 because behind reverse proxy) with the known error (semaphore-1 | 2024/03/27 12:04:23 http: panic serving 172.25.0.2:56110: securecookie: hash key is not set). How is the SEMAPHORE_COOKIE_ENCRYPTION environment variable persisted after regenaration on docker container update? Is it written into the config.json? Since I'm mounting my own config file via volume, this might be the problem?

[gschafra]

Even providing it as environment variable in the docker-compose.yml

services:
  # ...
  semaphore:
    restart: unless-stopped
    image: registry-gitlab.itx.de/docker-prod/semaphore:2.9.58
    ports:
      - 3000:3000
    environment:
      # ...
      ANSIBLE_HOST_KEY_CHECKING: 'false'
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: vrornXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
     ANSIBLE_HOST_KEY_CHECKING: 'false'
      SESSION_KEY: rU8cl6gH+XXXXXXXXXXXXXXXXXXXXXXXXX
      SEMAPHORE_COOKIE_ENCRYPTION: rUXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    volumes:
      - type: bind
        source: ./semaphore/config.json
        target: /etc/semaphore/config.json
    depends_on:
      - db

as well as putting it directly into the mounted config.json

{
    "tmp_path": "/tmp/semaphore",
    "access_key_encryption": "vrornXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "cookie_encryption": "rUXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
}

doesn't help 😞 .

[fiftin]

@gschafra I found the issue. You need to provide cookie_hash too like cookie_encryption :)

[fiftin]

Can you check your config.json right now? I can't understand why it worked before.

[gschafra]

I'll try... what should the value of cookie_hash and cookie_encryption look like? Can I use head -c32 /dev/urandom | base64 output?

[fiftin]

Yes

[fiftin]

I see this block in your docker file:

It means that Semaphore stores config in your host file system. So recreating of the continuer shouldn't break your configuration.

[gschafra]

O.k. now it works. TXL