SyncV2 Depth tracking

[ChrisPenner]

• Create Depth Tables on Prod
• Create and initialize migration tables on Prod.
• Sanity Check Prod after migration

Overview

Iteratively track dependency-depth information for all entities such that we can stream entities to UCM clients in dependency-first order which allows them to save entities as soon as they're received rather than needing to do a client-side sort.

This also reduces required memory usage on the client, and makes the download more robust against Ctrl-c, since you can pick up from where you left off (the stream will skip any pre-saved entities).

Implementation notes

When an entity is saved, set its depth to max(depth of all dependencies) + 1.
When streaming entities to client, sort by depth ASC.
Set sort mode of stream to Dependencies First.

Test coverage

• Test on staging

Loose ends

There's a corresponding improvement in UCM to better parallelize the download stream here: unisonweb/unison#5593