app: if sync fails, ask the server to shard the index as a backoff mechanism

[cfm]

As an alternative to freedomofpress/securedrop#7689 (comment), the app could ask the server to shard the index as a form of backoff:

1. Request the default index with shards=1 (i.e., the whole thing). If the request succeeds, proceed with the rest of sync.
2. If the request times out, request a halved index with shards=shards*2. If the request succeeds, proceed with the rest of sync.
   • Shards are deterministic by UUID prefix as sketched in API v2: have login endpoint return sync data securedrop#7689 (comment).
3. If the request times out, GOTO (2).

Whatever value of shards succeeded for /index could also be used for /data.

[cfm]

The approach I wound up taking in freedomofpress/securedrop#7770 is broadly consistent with the goals I described here, but the mechanism is different (more flexible). I'll write up the client-side implications next week.

[cfm]

Tl;dr: num_records → shards is clever, but num_records → timeout may be all we need. It's at least a simpler place to start.

---

freedomofpress/securedrop#7770 gives a client two things:

1. /api/v1/token returns hints like:

   {
     "version": "abcdef",
     "sources": 100,
     "items": 200
   }

2. /api/v2/index/<shard_spec> can be set based on the total number of records (sources + items) to specify an arbitrarily small shard of the total index, keyed by UUID.

These mechanisms combine with the existing mechanism:

3. /api/v2/data accepts lists of UUIDs for the sources, items, etc. to return.

For all but the initial sync, (3) already minimizes the amount of metadata a client fetches. The point of (2) is to split up the size of the index a client fetches on initial sync. The same shards can then be reused to split up the metadata a client fetches. (1) simply primes (2).

However, making use of (2) in a client requires introducing some new state: which index shards have we already fetched; for which index shards do we need to fetch metadata; etc. That state will complicate what is right now a straight up-and-down sync loop.

To preserve the current execution model (i.e., avoid a new loop or queue!), I now propose to start much more simply:

1. On login, save the hints into the AuthData object.

2. Write helpers around getRealisticTimeout() such that:

   // Existing:
   private getRealisticTimeout(sizeInBytes: number): ms {}
   
   // New:
   private estimateIndexSize(records: number): bytes {}
   private estimateMetadataSize(records: number): bytes {}

3. Use these functions to calculate realistic timeouts for syncMetadata() and submitBatch().

We can see how these calculated timeouts behave in scenarios like #3048 before we decide whether to implement sharding on the client side. If we decide not to do so after all, no harm done: it was the minor part of freedomofpress/securedrop#7770, mostly documentation and tests that help prove the determinism of the index structure, which we need to be able to hint it anyway.