libp2p based network layer rollout plan

[da-kami]

libp2p network layer rollout plan

Problem: In order to use a new protocol the CFD on both sides has to be aware of the counterparty peer-id to send messages.

We have to ensure that a CFD either knows the peer-id, or does not know the peer-id. A CFD that does not know the peer-id is a legacy CFD that was created before we stored peer-ids and only supports old legacy connection protocols.

Note: Prior to storing peer-ids in the CFDs, CFDs cannot know the peer-id on the maker side (because we don't have a mapping from legacy network-id to peer-id for the takers).

Rollout plan constraints for taker and maker:

1. The taker decides to use either the new or the old protocol.
   • The taker decides based on "is the peer ID known for this CFD"
   • The decision on the taker is done per protocol, if we roll out collab settlement then the taker will decide to trigger collab-settlement-old or collab-settlement-new.
2. The maker does not decide anything, if the taker comes with a new protocol request if will assume that the peer-id is available and just go. If the protocol fails we accept that a CFD may get force closed.
   • As long as the taker's decision which protocol to use is sane this should be fine.
3. Each newly created CFD will store the peer-id on both sides
   • The taker stores the peer-id that it knows from the maker to make a connection.
   • The maker stores the peer-id that is sent over by the taker as part of the old connection initially
     • This is necessary because taking an order triggers the creation of the CFD, and at that point we need to know the peer-id, but we are on the legacy connection until we roll out taking an order via the libp2p connection.
   • In the database we keep a "dummy" entry for legacy CFDs
     • That way the peer-id field in cfds can be NOT NULL
     • We map to an option in the model based on the dummy.
     • Peer-ids are not updated in the database. A legacy CFD cannot become a CFD that supports libp2p connections. Eventually all legacy CFDs will be closed, then we can remove the old network code.

[da-kami]

Note on the maker's accept / reject:

When accepting / rejecting within a protocol (e.g. for rollover) the maker cannot (easily) decide how to dispatch (to libp2p actor or rollover) based on a peer_idbecause:

• If the taker opened the CFD prior to upgrading the maker will not have a peer_id in the database, so checking for a peer_id in the db is not reliable if we want to use the new rollover on existing positions.
• We could pass on peer_id information from the connection to the cfd actor, but moving this state around is not a good option - it just makes things complicated. We would have to store the information in the cfd actor, because accept/reject does not involve the connection actor (besides sending a a message from maker to taker but that is beyond the mentioned decision).

That's why we decide to just try libp2p and if that does not work fall back to dispatch to a potential legacy actor. See #2122

[klochowicz]

That would work if we did a single release for all the libp2p protocols.... Unfortunately, with anything other than rollover we have to fallback anyway (which is OK, I just wanted to make sure everyone is aware of this).

[da-kami]

To gt an overview what is still missing until the migration is done here is a list of thing from the top of my mind.
@klochowicz please add if I am missing something.

Todo:

• Stabilize connection lifecycle (currently being tackled, with the recent fixes this should be done)
• Identify protocol (to be finished, almost done)
• Contract Setup protocol
• Stats/Monitoring: Ensure that we are reporting the same / similar metrics so we can maintain similar stats in Grafana

Done:

• Rollover protocol
• Collab settlement protocol
• Offers protocol