docker UDP log issues - conntrack

[YuryHrytsuk]

Original title: sim4life: director-v2 missing logs in Graylog

[YuryHrytsuk]

Observations

• both v0 and v2 directors are missing logs.
• all other containers on the node where directors run have logs in Graylog
• director v0 has replicas on 3 nodes. This should not be node-related. On these nodes other containers have their logs sent to Graylog

Note: on NIH PROD and on Dalco PROD logs are present. The is unlikely to be backend code related

[YuryHrytsuk]

Errors in graylog container logs

2025-08-18 08:34:18,223 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=2184..., messageQueueId=945..., codec=gelf, payloadSize=397, timestamp=2025-08-18T08:34:18.222Z, seqenceNr=10526263, remoteAddress=/172.16.0.145:42719} on input <689...>.

This error log could be found on NIH PROD and Dalco PROD too

Graylog2/graylog2-server#5793

[YuryHrytsuk]

Send logs manually (does not work)

https://serverfault.com/questions/591758/send-echo-message-to-graylog2-via-gelf-tcp-12201-port#comment730337_591758

[YuryHrytsuk]

Enabled debug log level in daemon.json (NIH STAG) but it does not show anything related to logging. It looks like it won't help gettings insights in logs send via / with GELF.

However, we don't have missing logs in this deployment. So, debug logs might be still useful

Also, source code of GELF logger in daemon. Here we can find some logs to be printed

[YuryHrytsuk]

I managed to reproduce it.

Disable graylog. Restart director service. Wait 2 minutes and enable graylog. Logs from director containers won't end up in graylog anymore

Created an issue in moby/moby#50766

[YuryHrytsuk]

Docker daemon codebase

daemons log initialization https://github.com/moby/moby/blob/master/daemon/logs.go

• creates a logger for container?

gelf logger implementation https://github.com/moby/moby/blob/master/daemon/logger/gelf/gelf.go

• defines classes to send gelf logs

copier https://github.com/moby/moby/blob/master/daemon/logger/copier.go#L129

• captures stdout / stderr and feed it to docker logger?
• Copilot says The copier.go module defines the Copier type, which is responsible for reading logs from multiple sources (typically a container’s stdout and stderr streams) and forwarding those logs to a specified log driver (a Logger).

Workflow:

• container startLogging() --> container.StartLogger() + copier.Run()
  • container StartLogger() --> logger getLogDriver()
  • copier Run() --> copier copySrc() --> copier dst.Log(msg) # dst is set to container.startLogger() inside container startLogging

Docker Swarm Networking

• https://www.suse.com/c/docker-swarm-container-networking/

Docker gelf logging driver

• https://jpetazzo.github.io/2017/01/20/docker-logging-gelf/

[YuryHrytsuk]

The problem might be tight to Graylog.

When running custom udp 12201 listener, I keep always getting logs (no matter when I start / restart traefik).

With graylog, if we start it after traefik was started, no logs anywhere and no complains from daemon about cannot send gelf logs

[YuryHrytsuk]

Root problem / cause

It looks to be related to docker swarm networking. When this problem occurs, traffic reaches localhost port 12201 (we run python app listening on this port directly on machine without containers) but it does not reach if the same python app is deployed as docker swarm service.

Read more in comment and issues mentioned in this comment: moby/moby#50766 (comment)

[YuryHrytsuk]

Workaround

Go to node on which container with missing logs run and execute sudo conntrack -D -p udp (if tool is missing dosudo apt-get install conntrack)

[YuryHrytsuk]

Solutions / Fixes

Switch to TCP

Source: moby/moby#35135 (comment)

If endpoint is down, new containers do not start moby/moby#21966. Can this be worked around --> with fluentd it is possible moby/moby#21966 (comment)

[YuryHrytsuk]

It is fixed since we switched to Logging Driver using TCP (this fixes stale connections issue to the best of my knowledge + testing) with Fluentd with async setting (that fixes moby/moby#21966) 🎉 🥳

PR with changes fixing this iisue

• Introduce Logging Stack: Add fluentd, add loki (🚧🚧 DEVOPS 🚧🚧) #1058