docs: Document sync modes

[martintomazic]

Add documentation about different syncing modes, syncing stages and corresponding databases.

Include ETA and which stage is blocked on which. Finally, describe how to monitor the progress of each stage using node's status and exported metrics.

Motivation

Having common terminology and overview of the syncing stages would significantly ease the node troubleshooting with e.g. node operators, devops etc.

E.g. troubleshooting and planning of https://github.com/oasisprotocol/internal-ops/issues/1221 would be much easier using this documentation.

Proposed direction

Syncing modes

• Block sync for consensus.
• Diff sync for runtimes.
• State sync / Checkpoint sync (?) for both consensus and runtimes:
  • Document checkpoints as independent concept, so that we can reference them elsewhere.
  • Genesis file is technically a checkpoint as well.

Stages:

• Consensus state sync.
• History (re)-index (in parallel to consensus state sync).
• Runtime state sync (blocked by consensus state sync and or history (re)-index).

Databases used:

• State DB (NodeDB) for both consensus and runtimes.
• Consensus specific DBs: Blockstore, evidence, state...
• Runtime (light) history

Bonus: explain the importance of having no gaps between latest round found inside Runtime State DB and earliest round inside Runtime Light History - relates to #6403.

End goal

Consistent terminology, also reflected in our code, metrics, status.

Challenge

Find good names that are not far off from existing terminology used in code, documentation, metrics and status output.