Message delivery to matrix.org is extremely unreliable (messages do not send, appear as empty on clients)

[c-x-berger]

Description

My homesever (boiler.social) cannot reliably send messages to matrix.org, even though federation seems to otherwise work OK. This happens in both E2EE rooms as well as plain-text channels (we have one channel that's all boiler.social users and one very confused matrix.org user...)

In a few cases, messages from boiler.social arrived to a matrix.org user as empty messages, even though other boiler.social -> matrix.org messages from the same sender eventually arrived OK. Seriously - this screenshot is from a matrix.org user, all messages are from boiler.social: https://imgur.com/tDQyWsT

(The second blank message here has an event ID of $eCbjzJ2a7jxz_e1SUtxI0M-YHOAZTD6kzatCd9hO4OA, if anyone with matrix.org access wants to check in more detail.)

Strangely, messages from matrix.org can (usually?) be seen on boiler.social, though there's often a delay. Sometimes. I'm not totally clear when or why this works.

I thought this was a version of #17035, but running

delete from device_federation_outbox where length(messages_json) > 65536;

and restarting Synapse hasn't moved the needle.

Steps to reproduce

• be on boiler.social
• try to message friends
• :(

Homeserver

boiler.social

Synapse Version

1.145.0

Installation Method

Debian packages from packages.matrix.org

Database

Postgresql, single server, no porting or restoring

Workers

Multiple workers

Platform

Debian Bookworm on physical hardware

Configuration

No response

Relevant log output

# all seen both before and after running DELETE FROM
2026-01-17 18:25:06,845 - synapse.federation.sender.per_destination_queue - 453 - WARNING - federation_transaction_transmission_loop-3057 - TX [matrix.org] Failed to send transaction: Failed to send request: ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
2026-01-17 18:40:39,046 - synapse.crypto.keyring - 702 - WARNING - PerspectivesKeyFetcher-1 - Key lookup failed from 'matrix.org': Not retrying server matrix.org because we tried it recently retry_last_ts=1768693231502 and we won't check for another retry_interval=600000ms.
2026-01-17 18:44:41,949 - synapse.handlers.device - 1762 - WARNING - _maybe_retry_device_resync-14 - Failed to handle device list update for @travltux:matrix.org: Failed to send request: ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.defer.CancelledError: >]

Anything else that would be useful to know?

No response