Protocal authentication error/1007

[madamowicz]

Hi there!

I have sucessfully set up slurm-web-gateway (with ldap) and slurm-web-agent with jwt.
I can log in and pick up the slurm cluster from the list.
But it looks as though there is a problem between slurm-web-agent and slurmrestd.

These are the logs from slurm-web-agent:

[DEBUG] ⸬slurmweb.slurmrestd.unix:24    ↦ Connecting to unix socket /run/slurmrestd/slurmrestd.socket                                                                                                                                                                                                                                                                                                                                                              
[DEBUG] ⸬urllib3.connectionpool:544     ↦ http://localhost:None "GET /slurm/v0.0.40/ping HTTP/1.1" 200 742                                                                                                                       
[DEBUG] ⸬slurmweb.slurmrestd.unix:24    ↦ Connecting to unix socket /run/slurmrestd/slurmrestd.socket                                                                                                                            
[DEBUG] ⸬urllib3.connectionpool:544     ↦ http://localhost:None "GET /slurm/v0.0.40/jobs HTTP/1.1" 511 947                                                                                                                       
[ERROR] ⸬slurmweb.views.agent:78        ↦ slurmrestd error: Unable to query jobs (slurm_load_jobs()) [Protocol authentication error/1007]                                                                                        
[INFO]  ⸬werkzeug:97                    ↦ 10.79.2.111 - - [11/Mar/2025 16:07:03] "GET /v4.1.0/stats HTTP/1.1" 500 -

Logs from slurmrestd:

slurmrestd[398665]: slurmrestd: error: slurm_load_jobs(): [unix:/run/slurmrestd/slurmrestd.socket(fd:11)] parser=data_parser/v0.0.40 rc[1007]=Protocol authentication error -> Unable to query jobs
slurmrestd[398665]: error: slurm_load_jobs(): [unix:/run/slurmrestd/slurmrestd.socket(fd:11)] parser=data_parser/v0.0.40 rc[1007]=Protocol authentication error -> Unable to query jobs

I've even tried to check with curl as slumrestd user:

$ curl -s --unix-socket /run/slurmrestd/slurmrestd.socket  -X GET http://localhost/slurm/v0.0.40/jobs/state/
{
  "jobs": [
  ],
  "meta": {
    "plugin": {
      "type": "openapi\/slurmctld",
      "name": "Slurm OpenAPI slurmctld",
      "data_parser": "data_parser\/v0.0.40",
      "accounting_storage": "accounting_storage\/slurmdbd"
    },
    "client": {
      "source": "unix:\/run\/slurmrestd\/slurmrestd.socket(fd:11)",
      "user": "root",
      "group": "root"
    },
    "command": [
    ],
    "slurm": {
      "version": {
        "major": "24",
        "micro": "0",
        "minor": "11"
      },
      "release": "24.11.0",
      "cluster": "testcluster01"
    }
  },
  "errors": [
    {
      "description": "Unable to query job states",
      "error_number": 1007,
      "error": "Protocol authentication error",
      "source": "slurm_load_job_state()"
    }
  ],
  "warnings": [
  ]
}

What's very interesting is the client part. Why does it say root?
Non of my slurm services runs as root.

My slurm auth is:

AuthType=auth/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/var/spool/slurm/secrets/jwt_hs256.key

The same jwt key is used across the board.

User slurmrestd can generate JWT token and everything seems to be working fine via JWT:

$ curl  -k --unix-socket /run/slurmrestd/slurmrestd.socket -H X-SLURM-USER-TOKEN:$SLURM_JWT -X GET http://localhost:6820/slurm/v0.0.42/jobs/state/
{
  "jobs": [
  ],
  "meta": {
    "plugin": {
      "type": "openapi\/slurmctld",
      "name": "Slurm OpenAPI slurmctld",
      "data_parser": "data_parser\/v0.0.42",
      "accounting_storage": "accounting_storage\/slurmdbd"
    },
    "client": {
      "source": "unix:\/run\/slurmrestd\/slurmrestd.socket(fd:11)",
      "user": "root",
      "group": "root"
    },
    "command": [
    ],
    "slurm": {
      "version": {
        "major": "24",
        "micro": "0",
        "minor": "11"
      },
      "release": "24.11.0",
      "cluster": "testcluster01"
    }
  },
  "errors": [
  ],
  "warnings": [
  ]
}

But I know that acording to docs slurm-web-agent only supports auth/local atm.

I believe that munge works fine cause rest of the components of the cluster can talk to each other.

Please, help me understand what might be wrong here.

[madamowicz]

I have found the root cause of the problem.
The slurmrestd (DEB pkg) comes with systemd service file which has some defaults. The one which was causing problems is:

Environment=SLURM_JWT=daemon

When I commented this env variable definition and restarted slurmrestd the slurm-web-agent finally started working fine.