LedgerDB: implement predictable snapshotting

[amesgen]

Closes #1424

Based on top of #1513, see there for the relation to this change.

Note that we need #1573 before this can be released in a node.

This PR is intended to be reviewed commit-by-commit.

---

This PR replaces the previous logic for when to create snapshots (It would be possible to preserve it, but I don't see a big motivation). Concretely, SnapshotFrequencyArgs contains

• sfaInterval :: SlotNo: Create snapshots every sfaInterval many slots.

  Default: 2*k = 4320 slots, so 72 min on mainnet as before.

• sfaOffset :: SlotNo: Allows to determine the offset of where snapshots are taken, see below.

  Default: 0

• sfaRateLimit :: DiffTime: A minimum duration between snapshots (used to avoid excessive snapshots while syncing).

  Default: 10 minutes (previous value was 6 minutes, which seemed a bit low, so I increased it somewhat. Maybe it should be increased even more now that we no longer have the substantialAmountOfBlocksWereProcessed check.)

Concretely, the node will try to create snapshots for the last immutable blocks before the slots

sfaOffset, sfaOffset + sfaInterval, sfaOffset + 2 * sfaInterval, sfaOffset + 3 * sfaInterval, ...

but can skip creating some of these depending on the sfaRateLimit (which can be disabled by setting it to a non-positive value). Also see the Haddocks.

For example, setting sfaInterval = 10*2160*20 (one mainnet Shelley epoch) and sfaOffset = 172800 will cause the node to create snapshots for the last block in every Shelley epoch (because the first Shelley slot is 4492800, and 4492800 `mod` (10*2160*20) = 172800. By tweaking sfaOffset, one can take snapshots eg right before the midway point in each epoch.

There is some code that could be shared between V1 and V2 (already even before this PR), but given the upcoming removal of V1, this seems acceptable for now.

---

Also includes a test running a ChainDB in IOSim to test that everything is hooked up correctly (in particular regarding the background threads).

See #1513 (comment) for sync regression tests.

[amesgen]

I integrated this into 10.5.1 for testing:

• Consensus branch: amesgen/predictable-snapshots-10.5
• Node branch: amesgen/test-snapshot-stuff

In the Node config, the following config would result in the node creating a snapshot for the last block in every Shelly epoch (having 10*2160/20 = 432000 slots):

LedgerDB:
  Backend: V2InMemory # or V1LMDB
  # Number of slots in a Shelley epoch, so we will create snapshots
  # precisely for the last block in each Shelley epoch
  SnapshotInterval: 432000
  # Due to Byron, the first slots of Shelley epochs are offset by this amount
  SlotOffset: 172800
  # Disable the rate limit, to be sure that we definitely create snapshots
  # for all epochs
  RateLimit: 0

Currently, this treats SnapshotInterval (which previously was a number of seconds) as a number of slots. This is convenient, and should be fine for mainnet AFAICT, but maybe we still want to add some kind of warning (see also the TODO in the code there).

[jasagredo]

Suggested change

	-- number @slot@.
	-- number @slot@ or younger.

?

[amesgen]

Each s in snapshotSlots is the slot of a ledger state in DbChangelog (see the contract of onDiskSnapshotSelector), so it is true as written. (Otherwise, we wouldn't take a snapshot for the requested slot, so snapshots wouldn't be predictable.)

[jasagredo]

Isn't it necessarily positive by using a SlotNo?

[jasagredo]

This nubOrd I imagine is there only because of EBBs, right?

[jasagredo]

I'm confused. I seem to remeber the flow was the opposite, first flush then prune, no?

EDIT: Ah now I think pruning just affects the changelog states and not the diffs.

[jasagredo]

Here we prune the lseq that we provide to the function that takes the snapshot, but we do not modify the ldbSeq. However from what I understood about V1 above, there we do prune the dbChangelog in the environment. Why this discrepancy?

[jasagredo]

Above we do:

atomically $ modifyTVar (ldbChangelog env) (prune pruneStrat)