OCM Journaling

[mickenordin]

If we keep a sequential id, with number series per remote OCM server, and incorporate that in to any OCM message, it would be possible to recover state after an outage by requesting previous messages, and also to detect if message have been send that have been lost.

[mickenordin]

Journal for OCM provider A:

1. Share resource β with person X
2. Share resource γ with person Y
3. Unshare resource β with person X
4. Share resource ∆ with person X

This can then be compressed to:
2. Share resource γ with person Y
4. Share resource ∆ with person X

Which is to be understood as:

1. NOOP
2. Share resource γ with person Y
3. NOOP
4. Share resource ∆ with person X

upon replay request.

[mickenordin]

Instead of using one journal per provider, it would probably make sense to have one journal per message type (so shares and invites), and also each server needs to keep track of both outgoing ids as well as incomming ids.

This could be an algorithm for implementing this at an ocm provider:

Populating outgoing ids

1. Add sequential id column to both outgoing shares and outgoing invites
2. Foreach provider find all outgoing shares and invites and order by timestamp if possible
3. Assign a sequential id to each entry (again from individual journal per provider and type only for outgoing messages)
4. When sending next message, add next sequential id to outgoing message.

Populating incomming ids

A server MUST not care about lack of sequential id in an incomming message, but when it sees a provider use a sequential id for the first time and the sequence id != 1 the client SHOULD request all messages of the type from id 1 to to the id it has seen, and make sure there are no shares or invites (depending on type) that can not be tagged with an incomming sequential id. If it finds an orphan object in the database, it SHOULD be removed, as the remote server does not consider this a valid entry. This replay would populate incomming id for that provider, so now both incomming and outgoing ids are populated.

Replay

To facilitate replay we would need to introduce a mechanism to request all messages from a certain id, this could be done by adding a new tyoe of request to the /share and /invite-accepted endpoint respectively or by introducing one or more endpoints for this type of replay request.

In any case, the response to a replay request MUST be a JSON array with all OCM messages potentially with a new noop message.

We should also have a journal capability advertised at the discovery endpoint.

[glpatcern]

For the record, this would also address #107

[mickenordin]

What do you guys think about this? Should I make a pr to implement this?

[KrausMatthias]

I don't really understand the benefit of the event Ids. Couldn't the Sending Server just keep track of successful requests / notifications to the Receiving Server (based on a 2xx response)? The Sending Server might keep a FIFO queue with outgoing requests (which it might compress as described), and just not send any further outgoing request until that FIFO queue is emptied?

When would there be a situation where the Sending Server would skip an outgoing message/id? I could imagine in a "invite-then-share" situation that the share might race the invite acceptance, but that wouldn't be solved by separate journals for invites and shares. I'm also not sure it would solve "invite-share-races" with a common journal, as invite-acceptance and share-creation are originating from different servers?

Do I miss something? 🤔

[mickenordin]

The one time it will happen is if there is a share followed by an unshare, or an invite that will later be removed.