Event got sent but not synced back down everywhere

[bwindels]

The event in question is https://app.element.io/#/room/!RbKRuLTvHIhArQcvjr:matrix.org/$1613561441289661rpDpF:matrix.org?via=matrix.org&via=vector.modular.im&via=t2l.io

The client that sent the event (element desktop) and my dekstop hydrogen running alongside did not get the event from sync.

On element desktop the event was black (e.g. sent) but stuck to the bottom of the room, and on hydrogen I check the storage and it showed that there was no limited sync for that room around those messages, and there is no trace of that event anywhere.

When opening my phone hydrogen an hour or so later, the event was synced down to it in the correct order.

At least T3chguy and Ryan did receive the event in the correct order.

[BillCarsonFr]

we have also several report of that on android
https://github.com/matrix-org/element-android-rageshakes/issues/20724
https://github.com/matrix-org/element-android-rageshakes/issues/20672
https://github.com/matrix-org/element-android-rageshakes/issues/20578

I also experienced it on other client (web)

Also this issue element-hq/element-android#516

[BillCarsonFr]

#9624

[bwindels]

@callahad @anoadragon453 I renamed this ticket as #9624 (which was marked as a dupe) is about the recipient and not the sender not receiving the message. FYI if this changes the severity of the bug.

[callahad]

Just an update: This is on our radar, but we expect it to be a pretty big yak to shave, so it's currently deferred until we can wrap up some other hairy problems 🐃

[richvdh]

Matthew reported another instance of this today. Link and details for internal users at https://github.com/matrix-org/element-web-rageshakes/issues/5808

[anoadragon453]

And another set by Matthew on Aug 10:

Room ID: !JjwYSGirOwIXEpJgzg:matrix.org
Event ID: $ZrcrvSAESv5PBzszSXpT5EahA2eQR-raFbFfSY314LQ

[erikjohnston]

We should add some more logs to try and track down where the event is going missing. I'd probably do something like:

• Log to the span how many rooms have changed (i.e. what's returned by _get_rooms_changed)
• Add a span for each _generate_room_entry call, tagging it with room ID
• Log to the span the number of events returned from the DB, and whether we hit a cache, and what tokens are used in the query
• Log to the span how many events there are after filtering.
• Log to the span the number of room entries and events in the response JSON just before returning

etc. The idea being that hopefully we'll be able to see if the missing event is pulled out of the DB and if so where we drop it (if we ever do).