HTTP /healthz check in Nomad GET /healthz

[theronic]

I'm trying to get the SpiceDB GET /healthz HTTP endpoint working in a Nomad job, but I can't get it to show healthy in Nomad status:

job "spicedb" {
  datacenters = ["dc1"]
  type = "service"

  group "spicedb" {
    count = 1

    network {
      port "grpc" {
        static = 50051
        to     = 50051
      }
      port "http" {
        static = 8443
      }
    }

    task "spicedb" {
      driver = "docker"

      config {
        image = "authzed/spicedb:latest"
        ports = ["grpc"]

        args = [
          "serve",
          "--log-level=debug",
          "--grpc-preshared-key", "spicedb",
          "--http-enabled",
          "--datastore-engine", "memory"
        ]
      }

      resources {
        cpu = 500 # MHz
        memory = 256 # MB
      }

      service {
        name = "spicedb"
        port = "grpc"

        tags = ["spicedb"]

        check {
          type     = "http"
          name     = "spicedb_healthz"
          port     = "http"
          path     = "/healthz"
          interval = "10s"
          timeout  = "5s"
        }
      }
    }
  }
}

Output of nomad status spicedb:

ID            = spicedb
Name          = spicedb
Submit Date   = 2024-07-05T15:18:01+02:00
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Node Pool     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
spicedb     0       0         1        0       0         0     0

Latest Deployment
ID          = 6c837aaa
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
spicedb     1        1       0        0          2024-07-05T15:28:01+02:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
c94f5328  a72bbf2d  spicedb     0        run      running  1m53s ago  1m51s ago

What am I doing wring? HTTP seems to be enabled.

[theronic]

I also tried moving the HTTP check to the "http" service and am able to run curl -v 127.00.1/healthz and get back a 200 with SERVING response, but Nomad still shows unhealthy:

job "spicedb" {
  datacenters = ["dc1"]
  type = "service"

  group "spicedb" {
    count = 1

    network {
      port "grpc" {
        static = 50051
        to     = 50051
      }
      port "http" {
        static = 8443
        to     = 8443
      }
    }

    task "spicedb" {
      driver = "docker"

      config {
        image = "authzed/spicedb:latest"
        ports = ["grpc"]

        args = [
          "serve",
          "--log-level=debug",
          "--grpc-preshared-key", "spicedb",
          "--http-enabled",
          "--datastore-engine", "memory"
        ]
      }

      resources {
        cpu = 500 # MHz
        memory = 256 # MB
      }

      service {
        name = "spicedb-http"
        port = "http"

        check {
          type     = "http"
          name     = "spicedb_healthz"
          port     = "http"
          path     = "/healthz"
          interval = "10s"
          timeout  = "5s"
        }
      }

      service {
        name = "spicedb"
        port = "grpc"

        tags = ["spicedb"]

        #         check {
        #           type     = "grpc"
        #           port     = "grpc"
        #           interval = "5s"
        #           timeout  = "2s"
        #           tls_skip_verify = true
        #         }

        #         check {
        #           type     = "script"
        #           name     = "grpc-health-check"
        #           task     = "spicedb"
        #           command  = "/usr/local/bin/grpc-health-check"
        #           args     = ["-addr", "127.0.0.1:50051"]
        #           interval = "10s"
        #           timeout  = "2s"
        #         }
      }
    }
  }
}

(edit: I was running a separate Docker image that was responding to the HTTP /healthz check. Can't get the Nomad job to respond on /healthz)

[theronic]

I can see the ports allocated correctly in the Nomad web UI, but cannot access the health check at http://127.0.0.1/healthz even though grpc-health-probe works against 127.0.0.1:50051:

[theronic]

I can see the SpiceDB logs in the Nomad web UI showing that HTTP server has been started at 0.0.0.0:8443 and gRPC on port 50051:

{"level":"info","addr":":50051","network":"tcp","service":"grpc","workers":0,"insecure":true,"time":"2024-07-05T13:44:15Z","message":"grpc server started serving"}
{"level":"info","upstream":":50051","time":"2024-07-05T13:44:15Z","message":"starting REST gateway"}
{"level":"debug","time":"2024-07-05T13:44:15Z","message":"initializing telemetry collector"}
{"level":"info","datastore":"*schemacaching.definitionCachingProxy","time":"2024-07-05T13:44:15Z","message":"running server"}
{"level":"info","addr":":9090","service":"metrics","insecure":true,"time":"2024-07-05T13:44:15Z","message":"http server started serving"}
{"level":"info","addr":"0.0.0.0:8443","service":"http","insecure":true,"time":"2024-07-05T13:44:15Z","message":"http server started serving"}
{"level":"info","interval":"1h0m0s","endpoint":"https://telemetry.authzed.com","next":"35s","time":"2024-07-05T13:44:15Z","message":"telemetry reporter scheduled"}
{"level":"debug","time":"2024-07-05T13:44:15Z","message":"checking if datastore and dispatcher are ready"}
{"level":"debug","datastoreReady":true,"dispatchReady":true,"time":"2024-07-05T13:44:15Z","message":"completed dispatcher and datastore readiness checks"}
{"level":"debug","protocol":"grpc","grpc.component":"server","grpc.service":"grpc.health.v1.Health","grpc.method":"Check","grpc.method_type":"unary","requestID":"cq3vhcr8hgds73f6ja70","peer.address":"172.17.0.1:60498","grpc.start_time":"2024-07-05T13:44:19Z","grpc.request.deadline":"2024-07-05T13:44:20Z","grpc.code":"OK","grpc.time_ms":0,"time":"2024-07-05T13:44:19Z","message":"finished call"}
{"level":"debug","protocol":"grpc","grpc.component":"server","grpc.service":"grpc.health.v1.Health","grpc.method":"Check","grpc.method_type":"unary","requestID":"cq3vhd38hgds73f6ja7g","peer.address":"172.17.0.1:60504","grpc.start_time":"2024-07-05T13:44:20Z","grpc.request.deadline":"2024-07-05T13:44:21Z","grpc.code":"OK","grpc.time_ms":0,"time":"2024-07-05T13:44:20Z","message":"finished call"}
{"level":"debug","endpoint":"https://telemetry.authzed.com","next":"1h0m0s","time":"2024-07-05T13:44:51Z","message":"reported telemetry"}

[vroldanbet]

A difference I see is that the HTTP port is not listening over all interfaces, like gRPC. Note:
0.0.0.0:8443
vs
:50051

Not sure if that would matter in Nomad, but it's worth double-checking. You can change the HTTP address with the corresponding flag.