Idea: data files management

[jeluard]

amaru repository embeds a number fo data files. Some of those files will grow in number (e.g. per epoch) and can be relatively big (e.g. 500MB).

An idea to reduce the burden of those files in the main repo would be to store them compressed via xz and stored separately (either in a separate repo or via a hosting platform like cloudflare).

Those files are generated by a set of scripts. The general logic is:

• run node until it fails
• extract the failing epoch to deduce the last ok epoch
• update data/$NETWORK/config.json to match this epoch
• run npm run fetchto generate data/$NETWORK/*/$EPOCH.json
• run conformance-data/ npm run generate to generate snaps files in crates/amaru/tests/snapshots/$NETWORK
• run ./scripts/generate-snapshot-test-casesto generate crates/amaru/tests/snapshots/$NETWORK/generated_compare_snapshot_test_cases.incl
• run tests via make test-e2e

This process could be automated with nightly jobs given there's already a nightly job syncing a cardano-node.