Webserver generated 20 million of logs in 1 day on dalco stag

[YuryHrytsuk]

log level: info

[YuryHrytsuk]

Total messages

Total message with log level warning

Pinpoint the source

log_level=WARNING | log_timestamp=2025-08-07 09:23:43,820 | log_source=engineio.server:send_packet(118) | log_uid=None | log_oec=None | log_trace_id=... | log_span_id=... | log_resource.service.name= | log_trace_sampled=True | log_msg=Cannot send to sid ...```

[YuryHrytsuk]

Same happends on internal master

[YuryHrytsuk]

This does not happen in production (screenshot from nih prod)

[matusdrobuliak66]

Matus notes:

• maybe mute also this?

log_level=INFO | log_timestamp=2025-08-06 18:36:46,416 | log_source=gunicorn.access:log(214) | log_uid=None | log_oec=None | log_trace_id=0 | log_span_id=0 | log_resource.service.name= | log_trace_sampled=False | log_msg=172.16.119.229 [06/Aug/2025:18:36:46 +0000] "GET / HTTP/1.1"

[YuryHrytsuk]

We tried to identify the root cause to no avail. The plan is to silent these warning messages as of now

[YuryHrytsuk]

Just observed similar behavior in sim4life (3'000 messages in 5 minutes)

log_level=WARNING | log_timestamp=2025-08-18 11:29:26,563 | log_source=engineio.server:send_packet(118) | log_uid=None | log_oec=None | log_trace_id=None | log_span_id=None | log_msg=Cannot send to sid ...