sentry-self-hosted-relay-1 is unhealthy #3983
Description
Self-Hosted Version
25.9.0 (master branch)
CPU Architecture
x86_64
Docker Version
Docker version 28.4.0, build d8eb465
Docker Compose Version
Docker Compose version v2.39.4
Machine Specification
- My system meets the minimum system requirements of Sentry
Steps to Reproduce
docker compose up -d --wait after installing.
Also note, this is an airgapped VM.
Expected Result
Relay being healthy or atleast, verbose healthcheck that lets me know where its failing.
Actual Result
The healthcheck that is failing
docker compose exec -it relay /bin/relay --config /work/.relay healthcheck
error: Relay is unhealthy. Status code: 503 Service Unavailable
relay logs:
relay-1 | 2025-10-04T11:27:51.075770Z INFO relay::setup: launching relay from config folder /work/.relay
relay-1 | 2025-10-04T11:27:51.122966Z INFO relay::setup: relay mode: proxy
relay-1 | 2025-10-04T11:27:51.122993Z INFO relay::setup: relay id: 94d45cba-46f9-42b8-b309-30dcbe444bfa
relay-1 | 2025-10-04T11:27:51.123003Z INFO relay::setup: public key: [relay_public_key_here]
relay-1 | 2025-10-04T11:27:51.123014Z INFO relay::setup: log level: trace
relay-1 | 2025-10-04T11:27:51.340404Z INFO relay_server: relay server starting
relay-1 | 2025-10-04T11:27:51.666802Z INFO relay_server::services::outcome: Configured to emit outcomes as client reports
relay-1 | 2025-10-04T11:27:51.688968Z INFO relay_server::services::outcome: OutcomeProducer started.
relay-1 | 2025-10-04T11:27:51.688857Z INFO relay_server::services::outcome_aggregator: outcome aggregator started
relay-1 | 2025-10-04T11:27:51.718974Z INFO relay_server::services::global_config: global config service starting
relay-1 | 2025-10-04T11:27:51.728283Z INFO relay_server::services::projects::source::upstream: project upstream cache started
relay-1 | 2025-10-04T11:27:51.750826Z INFO relay_server::services::global_config: serving default global configs due to lacking static global config file
relay-1 | 2025-10-04T11:27:51.797168Z INFO relay_server::service: starting 1 store workers
relay-1 | 2025-10-04T11:27:51.797656Z INFO relay_server::services::metrics::router: metrics router started
relay-1 | 2025-10-04T11:27:51.798552Z INFO relay_server::service: starting 6 envelope processing workers
relay-1 | 2025-10-04T11:27:51.825918Z TRACE relay_server::services::buffer::envelope_buffer: PolymorphicEnvelopeBuffer: initializing memory envelope buffer
relay-1 | 2025-10-04T11:27:51.837546Z INFO relay_server::services::relays: key cache started
relay-1 | 2025-10-04T11:27:51.837654Z INFO relay_server::services::server: spawning http server
relay-1 | 2025-10-04T11:27:51.837684Z INFO relay_server::services::server: listening on http://0.0.0.0:3000/
relay-1 | 2025-10-04T11:27:51.876868Z INFO relay_server::services::buffer: EnvelopeBufferService 0: starting
Web logs:
web-1 | Updating certificates in /etc/ssl/certs...
web-1 | 0 added, 0 removed; done.
web-1 | Running hooks in /etc/ca-certificates/update.d...
web-1 | done.
web-1 | sentry/requirements.txt is deprecated, use sentry/enhance-image.sh - see https://develop.sentry.dev/self-hosted/#enhance-sentry-image
web-1 | error : Unknown IO error
web-1 | *** Starting uWSGI 2.0.28 (64bit) on [Sat Oct 4 11:26:42 2025] ***
web-1 | compiled with version: 10.2.1 20210130 (Red Hat 10.2.1-11) on 31 October 2024 18:21:41
web-1 | os: Linux-6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025
web-1 | nodename: acc7ebf9cfb8
web-1 | machine: x86_64
web-1 | clock source: unix
web-1 | pcre jit disabled
web-1 | detected number of CPU cores: 8
web-1 | current working directory: /usr/src/sentry
web-1 | detected binary path: /.venv/bin/python3
web-1 | your memory page size is 4096 bytes
web-1 | detected max file descriptor number: 4096
web-1 | lock engine: pthread robust mutexes
web-1 | thunder lock: enabled
web-1 | uWSGI http bound on 0.0.0.0:9000 fd 4
web-1 | uwsgi socket 0 bound to TCP address 127.0.0.1:41635 (port auto-assigned) fd 3
web-1 | Python version: 3.13.1 (main, Feb 4 2025, 04:56:26) [GCC 12.2.0]
web-1 | --- Python VM already initialized ---
web-1 | Python main interpreter initialized at 0x78288427ca70
web-1 | python threads support enabled
web-1 | your server socket listen backlog is limited to 100 connections
web-1 | your mercy for graceful operations on workers is 60 seconds
web-1 | setting request body buffering size to 65536 bytes
web-1 | mapped 1924224 bytes (1879 KB) for 12 cores
web-1 | *** Operational MODE: preforking+threaded ***
web-1 | spawned uWSGI master process (pid: 739)
web-1 | spawned uWSGI worker 1 (pid: 741, cores: 4)
web-1 | spawned uWSGI worker 2 (pid: 742, cores: 4)
web-1 | spawned uWSGI worker 3 (pid: 743, cores: 4)
web-1 | spawned uWSGI http 1 (pid: 744)
web-1 | error : Unknown IO error
web-1 | error : Unknown IO error
web-1 | error : Unknown IO error
web-1 | WSGI app 0 (mountpoint='') ready in 51 seconds on interpreter 0x78288427ca70 pid: 743 (default app)
web-1 | WSGI app 0 (mountpoint='') ready in 51 seconds on interpreter 0x78288427ca70 pid: 741 (default app)
web-1 | WSGI app 0 (mountpoint='') ready in 51 seconds on interpreter 0x78288427ca70 pid: 742 (default app)
The config file (I updated it to change mode to proxy and disabled the processing)
relay:
upstream: "http://web:9000/"
host: 0.0.0.0
port: 3000
mode: proxy
logging:
level: TRACE
processing:
enabled: false
kafka_config:
- {name: "bootstrap.servers", value: "kafka:9092"}
- {name: "message.max.bytes", value: 50000000} # 50MB
redis: redis://redis:6379
geoip_path: "/geoip/GeoLite2-City.mmdb"
Is there a way I can make the healthcheck verbose to see what exactly is going wrong? The relay logs dont seem to be showing anything even at trace. I dont't think(?) this is an resource constraint issue as the logs don't show that either.
Event ID
No response