Specify port for healthcheck

[nickhammond]

I'm working on setting up otel/opentelemetry-collector-contrib:0.96.0 with kamal and having an issue with the healthcheck passing, does health-check-path support a port?

The main port that everything runs on for the container is 4318, but the internal healthcheck is exposed on 13133. So the deploy command needs to map the container to 4318 but internally the healthcheck should be over 13133. This is so that you don't have to publicly expose that port since there are other internal metrics and data you can find on that port if publicly exposed, it's just meant for internal use.

docker exec kamal-proxy kamal-proxy deploy hey-otel --target="15eb532f3dc1:4318" --host="otel.hey.io" --tls --deploy-timeout="30s" --drain-timeout="30s" --health-check-interval="3s" --health-check-timeout="5s" --health-check-path="localhost:13133/healthcheck" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent"

Looking at the logs for the container, it's running fine and not reporting errors, it's continually collecting metrics, I just can't get kamal proxy to see it as healthy and start sending it traffic. It's just reporting the regular docker stderr: Error: target failed to become healthy within configured timeout (30s) error, nothing more specific beyond that.

I also added a port mapping so that I could check the status of the container locally on the machine and it's returning a 200.

curl localhost:13133 -I
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 30 Apr 2025 16:57:02 GMT
Content-Length: 98

[kevinmcconnell]

@nickhammond the health check path doesn't accept a port. It's purely a path portion, and will always be fetched using the target's host & port. So in this case, is the problem that there's no suitable path on the target's main port that could signify whether that target is healthy or not?

[nickhammond]

@kevinmcconnell Yes, because the only endpoints available on the port that needs to be open(4318) process POST requests only. https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md#writing-with-httpjson

[nickhammond]

@kevinmcconnell Another potential option would be running a health check command that could check the status but I know that's not currently supported.

[nickhammond]

@kevinmcconnell What's the environment that the healthcheck runs in? Can I publish another port locally and use that for the healthcheck instead of the primary one? Just trying to think of alternative options.

[nickhammond]

@kevinmcconnell Any suggested workarounds for this?

[giosakti]

Hi @nickhammond & @kevinmcconnell I'm facing similar issue and I also need kamal to allow port customization for health check.

I just raised a PR: #152
Let me know if there's any feedback.

[nickhammond]

@giosakti Nice! Just posted a question but this would be great to get added in.

[kevinmcconnell]

I've merged the associated PR for this (#152), and we'll get it into the next release.