Messages from certain remote users missing / being rejected

[tonkku107]

Description

I've noticed people replying to events that don't exist for me and looking at the event source they are replying to specific users from the matrix.org homeserver. I am not receiving messages from those members but I do see them typing and their read receipts.
In the case of the log output below, they have been in the room for close to a month and the referenced m.room.member event can be found via element's dev tools, yet at the time of receiving the event synapse fails to calculate that as an auth_event.

Steps to reproduce

• (it is currently unknown what causes users to get in this state)
• receive a message from an affected user through federation
• be confused when people are talking to others that you cannot see

Homeserver

tonkku.me

Synapse Version

1.66.0

Installation Method

Docker (matrixdotorg/synapse)

Platform

Running in docker containers on a server with Ubuntu 20.04.3, AMD Ryzen 5 3600, 64GB RAM.

Relevant log output

2022-09-04 16:12:40,328 - synapse.federation.federation_server - 1197 - INFO - _process_incoming_pdus_in_room_inner-1875 - handling received PDU in room !GtIfdsfQtQIgbQSxwJ:archlinux.org: <FrozenEventV3 event_id=$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU, type=m.room.message, state_key=None, outlier=False>
2022-09-04 16:12:40,350 - synapse.handlers.federation_event - 1806 - INFO - _process_incoming_pdus_in_room_inner-1875-$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU - event's auth_events are different to our calculated auth_events. Claimed but not calculated: [<FrozenEventV3 event_id=$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU, type=m.room.member, state_key=@rajabinkz:matrix.org, outlier=False>]. Calculated but not claimed: []
2022-09-04 16:12:40,350 - synapse.handlers.federation_event - 1826 - WARNING - _process_incoming_pdus_in_room_inner-1875-$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU - While checking auth of <FrozenEventV3 event_id=$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU, type=m.room.message, state_key=None, outlier=False> against room state before the event: 403: User @rajabinkz:matrix.org not in room !GtIfdsfQtQIgbQSxwJ:archlinux.org (None)
2022-09-04 16:12:40,368 - synapse.http.client - 446 - INFO - _process_incoming_pdus_in_room_inner-1875-$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU - Received response to POST http://generic-worker4:9114/_synapse/replication/fed_send_events/XtTHDlfEIH: 200

Anything else that would be useful to know?

No response

[DMRobertson]

The key log line:

2022-09-04 16:12:40,350 - synapse.handlers.federation_event - 1826 - WARNING - _process_incoming_pdus_in_room_inner-1875-$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU - While checking auth of <FrozenEventV3 event_id=$9nEfMJ48sVmR_5zO1fjmNyaXWNVHSw1Go6CwAtSH3EU, type=m.room.message, state_key=None, outlier=False> against room state before the event: 403: User @rajabinkz:matrix.org not in room !GtIfdsfQtQIgbQSxwJ:archlinux.org (None)

Can you see any logs that mention $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU? (That event was created on the 12th of August this year.)

Additionally, what do the following database queries return?

SELECT COUNT(*) FROM events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
SELECT COUNT(*) FROM event_json WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';

[tonkku107]

Can you see any logs that mention $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU? (That event was created on the 12th of August this year.)

No, only that one log line above. I believe my log history doesn't go that far back since the containers have been recreated to update synapse.

Additionally, what do the following database queries return?

synapse=# SELECT COUNT(*) FROM events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
 count 
-------
     1
(1 row)

synapse=# SELECT COUNT(*) FROM event_json WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
 count 
-------
     1
(1 row)

[squahtx]

It's possible that the user's membership, $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU, has been marked as rejected for some reason or another, and so their messages are being marked as rejected too.

Can you run the following query to confirm?

SELECT event_id, type, outlier, rejection_reason FROM events WHERE event_id IN (SELECT '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU' UNION SELECT auth_id FROM event_auth WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU');

[tonkku107]

It's possible that the user's membership, $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU, has been marked as rejected for some reason or another, and so their messages are being marked as rejected too.

Can you run the following query to confirm?

synapse=# SELECT event_id, type, outlier, rejection_reason FROM events WHERE event_id IN (SELECT '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU' UNION SELECT auth_id FROM event_auth WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU');
                   event_id                   |        type         | outlier | rejection_reason 
----------------------------------------------+---------------------+---------+------------------
 $XhK6dzOmcZ3bWdeIY5dgndPYynv81kV1KC3tAc4Mwqc | m.room.power_levels | f       | 
 $lYTZFGTzuDxSU0Hr6Jn6UhzA2UaBja2fiCbR7Y_YpNg | m.room.join_rules   | f       | 
 $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU | m.room.member       | f       | 
 $o-qWJk4LlSnwSr_N0WXu9z-kE4-je1H0i7p1I2scPP0 | m.room.create       | t       | 
(4 rows)

There seems to be no rejection reason

[DMRobertson]

Odd. I wonder if you've been hit by a state reset (#8629) due to the weird corner (#12943). I'd like to do a little more investigation---let me concoct some queries for you to run.

In the worst case scenario, I'd suggest purging the room with the admin API and rejoining it.

[DMRobertson]

@tonkku107, could you run this query and paste its output please?

SELECT current_state_delta_stream.* FROM events JOIN current_state_delta_stream ON events.stream_ordering = current_state_delta_stream.stream_id WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
SELECT * FROM current_state_events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';

[tonkku107]

synapse=# SELECT current_state_delta_stream.* FROM events JOIN current_state_delta_stream ON events.stream_ordering = current_state_delta_stream.stream_id WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
 stream_id |              room_id              |     type      |       state_key       |                   event_id                   | prev_event_id | instance_name 
-----------+-----------------------------------+---------------+-----------------------+----------------------------------------------+---------------+---------------
    628852 | !GtIfdsfQtQIgbQSxwJ:archlinux.org | m.room.member | @rajabinkz:matrix.org | $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU |               | master
(1 row)

synapse=# SELECT * FROM current_state_events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
 event_id | room_id | type | state_key | membership 
----------+---------+------+-----------+------------
(0 rows)

I'm guessing that last one isn't supposed to return nothing

[DMRobertson]

I'm guessing that last one isn't supposed to return nothing

Yeah. To help understand why, can you run this query?

WITH groups AS (
    SELECT state_group_edges.*
    FROM state_group_edges
        JOIN event_to_state_groups USING(state_group)
        JOIN events USING(event_id)
    WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'
)
SELECT state_groups_state.*
FROM state_groups_state
    JOIN groups ON state_groups_state.state_group IN (groups.state_group, groups.prev_state_group)
;

[tonkku107]

synapse=# WITH groups AS (
synapse(#     SELECT state_group_edges.*
synapse(#     FROM state_group_edges
synapse(#         JOIN event_to_state_groups USING(state_group)
synapse(#         JOIN events USING(event_id)
synapse(#     WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'
synapse(# )
synapse-# SELECT state_groups_state.*
synapse-# FROM state_groups_state
synapse-#     JOIN groups ON state_groups_state.state_group IN (groups.state_group, groups.prev_state_group)
synapse-# ;
 state_group |              room_id              |     type      |        state_key        |                   event_id                   
-------------+-----------------------------------+---------------+-------------------------+----------------------------------------------
       64427 | !GtIfdsfQtQIgbQSxwJ:archlinux.org | m.room.member | @codingotaku:matrix.org | $LpwIBBp8Nnz7rXcsDno9cJztUuE13MtNT65JMV7TCPg
       64439 | !GtIfdsfQtQIgbQSxwJ:archlinux.org | m.room.member | @rajabinkz:matrix.org   | $jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU
(2 rows)

[DMRobertson]

Okay, looks like the event was accepted when it was received. Can you run this query to see if there any changes since then? (May take a while)

SELECT * FROM current_state_delta_stream WHERE room_id='!GtIfdsfQtQIgbQSxwJ:archlinux.org' AND type=' m.room.member' AND '@rajabinkz:matrix.org' ORDER BY stream_id ASC;