Explanation for Mempool test failures and next steps

[jasagredo]

After investigating the error in a commit in main I found the cause. TryAddTxs will re-sync with the LedgerDB if the old ledger state is inaccesible. This is a known shortcoming of UTxO-HD that was documented in the UTxO-HD docs: https://ouroboros-consensus.cardano.intersectmbo.org/docs/for-developers/utxo-hd/utxo-hd-in-depth#the-mempool.

Essentially this means that there are executions in the SUT that cannot be linearized by the model, thus a parallel test will not work for the mempool.

To solve this problem, we could do a rework of the mempool such that it holds a LedgerTablesHandle of the synced state, meaning that tryAddTxs will not need to revalidate ever and we can switch back to the behavior from before UTxO-HD. This however can result in race conditions if the handle is closed.

Discussing it with @amesgen, we came up with a possible design that would implement kind of reference counting for the handles, ensuring that used handles are not deallocated prematurely. More info

However to minimize the wasted effort, we might delay this until LedgerDB V1 is gone. So for now the parallel.atomic test is disabled.