Added broadcast delay for snarks and transactions until they are fully vertified

[0xMimir]

Resolves #1055, continuation of #1001

[tizoc]

@0xMimir this is not correct, because the full diff may have been rejected because of one particular transaction, but here we are reacting to every rejected transaction.

I have to think about it more, but I am unsure about the approach in general. This probably should work on the diff level, not per transaction.

[tizoc]

To add a bit more context, diffs will only be rejected if one of the transactions is rejected because of one of these reasons:

https://github.com/openmina/openmina/blob/b152745fce013d2cee65f78f9a51ae1e776d0726/ledger/src/transaction_pool.rs#L239

[0xMimir]

I could filter by error if tx error is grounds_for_diff_rejection, reject it, otherwise continue

[tizoc]

The "diff" is lost here:

https://github.com/openmina/openmina/blob/develop/p2p/src/network/pubsub/p2p_network_pubsub_reducer.rs#L577-L586

I guess it was implemented that way to make it more compatible with the way the webrtc layer works, but that makes it harder to work at the diff level. It also means that the transactions will be verified one by one instead of batched (in webrtc they are batched [here](https://github.com/openmina/openmina/blob/44ba15084f97b13f360ba4b88145ba07a78e638c/node/src/transaction_pool/candidate/transaction_pool_candidate_reducer.rs#L98, but that will not help the libp2p part).

You can also see here that the action dispatched to verify the transactions also supports multiple transactions (even if the callback always constructs the call using a single transaction):

https://github.com/openmina/openmina/blob/44ba15084f97b13f360ba4b88145ba07a78e638c/node/src/state.rs#L535-L538

I don't know if in practice we receive more than one transaction per diff from libp2p's gossip btw (you could add some logging and run the node for some time to check).

So:

• P2pChannelsTransactionAction::Libp2pReceived should contain all the transactions, not just one
• on_p2p_channels_transaction_libp2p_received callback should receive the full list of transactions, not just one. And then dispatch StartVerify with all the transactions at once.
• StartVerify and the related actions should probably contain the message id (as an Option similar to how it contains a reference to the rpc from where the transaction may have come from if it was sent by the user directly using the API)
• Then at the end of the sequence when handling ApplyVerifiedDiffWithAccounts you will have access to the actual message id (if any) from which those transactions came from.

Btw those actions probably need to be split in more, but lets leave that for later

[tizoc]

Another idea is that instead of having from_rpc and from_gossip separately (because you cannot have both), there could be a single from_origin that is either FromRpc(RpcId) | FromGossip(MessageId) | Other (or WebRtc instead of Other, I don't think there is another possibility) so that you can react accordingly depending on the origin.

[tizoc]

One last thing, and this should be left for later. But transactions should first go into the TransactionPoolState::candidates and with TransactionPoolCandidatesState extended to support storing the libp2p received messages grouped by message id. The logic should also be extended to handle libp2p gossip as an origin. This will make it much easier to later fix the issue of us trying to validate the pool before the node is even synced.

With that approach we would still receive and store the gossip messages but wait until we are synced to validate them.

Anyway, this is for later, first thing to solve is what I mentioned in the above comments.

[0xMimir]

I have updated logic in pubsub and transaction pool, to use message id instead of transaction hashes, also I have been running node for almost a day now and every time transaction diff, had only one transaction.

[tizoc]

Ok, thats useful to know (still we need to consider multiple transactions because the protocol allows it)

[tizoc]

LGTM (left some comments re: comments to add, please add them before merging)