Store sender data for Megolm sessions as per Invisible crypto

[andybalaam]

Part of Invisible Crypto.

Currently when we store an Inbound group session, we associate some SessionCreatorInfo to it. Currently this info mainly contains the curve25519 identity key of the session creator.

We need now to add more info:

• user_id - the associated mxid of the session creator (rooted in Master Signing Key authenticity).
• master_key - the Master Signing Key signing (via the SSK) this device at time of reception.
• master_key_verified - true if at time of reception the user was verified (see here)

We are currently computing the "authenticity" of a megolm session but we are doing it at decryption time.
The info we get at decryption time are sender: OwnedUserId and sender_device: OwnedDeviceId.
Both related to a VerificationState. This owner info is untrusted (claimed), unless the VerificationState is Trusted.

The VerificationState variants are:

• Verified
• Unverified::UnverifiedIdentity
• Unverified::UnsignedDevice
• Unverified::MissingDevice
• Unverified::InsecureSource

We are basically trying to get this same info but at reception time.

Three places we populate SenderData in an InboundGroupSession

1. When we receive the room keys via a to-device message. (This is already done via SenderDataFinder.)

2. When we get new or updated device info from /keys/query. To do this we need to be able to look up InboundGroupSessions that don't have SenderData by device curve key. This will require a new index on the inboundgroupsessions table. Question: if there are lots of sessions for a particular device, will we break everything by working through batches of them to update them?

3. When we decrypt a message for a session. In this case we have the session and device already, so it's just a case of persisting (into inboundgroupsessions) the VerificationState that we already look up at this moment.

The plan

In order to complete this we need to do these tasks:

• Sender Data: Support sender data in InboundGroupSession and InboundGroupSessionPickle #3542
• Sender Data: Fetch and store sender data when we receive a new megolm session #3543
• Sender Data: When we decrypt a message in an InboundGroupSession that is missing SenderData, populate it with what we know #3751
• Populate missing SenderData when we receive device info:
  • Sender Data: Allow looking up InboundGroupSessions with missing SenderData by device curve key #3752
  • Sender Data: Populate missing SenderData when we receive device info via /keys/query #3753
• Sender Data: Validate SenderData using Device::is_owner_of_session #3754
• Sender Data: Remove retry data from SenderData enum #3755
• crypto: Mark all new SenderData info as non-legacy #3785

Old tasks that are no longer relevant (see old plan in a comment below):

• Sender Data: Change DB schemata to add next_retry_time_ms property/column #3545
• Sender Data: Background task to retry fetching sender data for megolm sessions #3546
• Sender Data: Async migration task to prepare old sessions for fetching sender data #3547 (essentially just adds a next_retry_time so they get retried)

Out of scope

• finish MSC4147
• Store and retrieve sender data to/from backed up (and exported) sessions #3548 . Requires an MSC.
• Update message sender trust status when user's verification state improves #3890

---

[Moved from https://github.com/element-hq/crypto-internal/issues/310]

[andybalaam]

Old description for reference:

Part of Invisible Crypto.

Currently when we store an Inbound group session, we associate some SessionCreatorInfo to it. Currently this info mainly contains the curve25519 identity key of the session creator.

We need now to add more info:

• mxid The associated mxid of the session creator (rooted in MSK authenticity).
• MSK The msk signing (via SSK) this device at time of reception.
• MSK Trust Level at time of reception (TOFU|Verified, see here)

We are currently computing the "authenticity" of a megolm session but we are doing it at decryption time.
The info we get at decryption time are sender: OwnedUserId and sender_device: OwnedDeviceId.
Both related to a VerificationState. This owner info is untrusted (claimed), unless the VerificationState is Trusted.

The VerificationState variants are:

• Verified
• Unverified::UnverifiedIdentity
• Unverified::UnsignedDevice
• Unverified::MissingDevice
• Unverified::InsecureSource

We are basically trying to get this same info but at reception time.

Three speeds for handling a new session

When we receive a to-device message establishing a megolm session, we want to process it as quickly as possible, so there are 3 speeds:

• Fast lane: if we already have the information we need in memory or the store, process it during the sync loop.
• Slow lane: if we need to wait for Matrix API responses, don't block the sync loop. Instead, store the session in the store (in case we get restarted before we get a response) and kick off an async task that waits for the API response and processes the session later.
• Background retry task: if the information we need is not immediately available from the Matrix API, store the session in the store and the periodic background/retry task will pick it up and retry it later.

"Quarantine"

(These ideas will be used by e.g. https://github.com/element-hq/crypto-internal/issues/308 - not really for this issue.)

In previous documents, we talked about "quarantine", but this is a difficult subject because different sessions will be acceptable to different clients at different times. We prefer to talk about what messages will be shown in what modes. These are some possible policies:

• Show everything: show messages from all sessions, even if the device was not signed. This is likely to be the initial situation. Note: this excludes sessions where the cross-signing signature is incorrect: this is evidence of tampering so we discard those sessions.
• Show only from cross-signed device + legacy: don't show messages that are non_legacy but have no verified sender info. Include session if:
  • sender_data is of type SenderKnown, or
  • legacy_session is true
• Show only from cross-signed device: for environments where we expect no legacy sessions, we show only verified. Include session if:
  • sender_data is of type SenderKnown
• Top security ("paranoid"): for environments where TOFU trust is not good enough, we show only sessions from senders we have manually verified. Include session if:
  • sender_data is of type SenderKnown, and
  • msk_verified is true

Importing session keys

If we import a session from backup or from an exported file, we mark it as legacy and let the background task handle it.

The plan

In order to complete this we need to do these tasks:

• Sender Data: Support sender data in InboundGroupSession and InboundGroupSessionPickle #3542
• Sender Data: Fetch and store sender data when we receive a new megolm session #3543 (including adding a SessionManager to prevent clashes between concurrent tasks)
• Sender Data: Change DB schemata to add next_retry_time_ms property/column #3545
• Sender Data: Background task to retry fetching sender data for megolm sessions #3546
• Sender Data: Async migration task to prepare old sessions for fetching sender data #3547 (essentially just adds a next_retry_time so they get retried)

In addition, as a separate story, we need to this:

• Store and retrieve sender data to/from backed up (and exported) sessions #3548 . Requires an MSC.

---

[Moved from https://github.com/element-hq/crypto-internal/issues/310]