Indexes 5: Adds spk repo index subcommand for index generation and updates

[dcookspi]

This adds a new repo index subcommand to spk for index generation and updates. It adds the --use-indexes and --no-indexes flags for repository index usage. This also updates resolvo solver to get global variables data from an indexed repository. This allows resolvo to solve without needing to restart its solves.

Indexing

The index is design to help the solvers with solves. It doesn't contain enough data to help with other spk operations like building and testing a package. Indexing can be enabled or disabled in the spk config file. If indexing is enabled, you also have to generate an index, with spk repo index, prior to trying to use it.

To generate an index (for the origin):

• spk repo index --disable-repo local

To update an existing index, e.g. after a new python package was published:

• spk repo index --disable-repo local --update python

If index use is enabled in the config file, it can be disabled with the --no-indexes command line flag. If index use is disabled by default, it can be enbled with the --use-indexes flag. If index use is enabled but no index has been generated, spk will fallback to using the underlying repo directly (as it does before this change).

Speed Diferences

Generating the index file on our repo (sizes below) takes about 2 minutes. Updating a package in an existing index, such as after a new build is published, takes a few seconds.

Sample solver time improvements using this indexing with:

• an origin repo that has 2245 packages, 23540 versions, 82517 builds (11 erroring), and 141 global vars
• the index loads in ~0.0003 seconds unverified, or ~0.2 seconds verified, and is about 76 MB on disk
• these times were using an unverified index
• a "toolset" below is a set of requests for the named DCC and our typical in-house plugins and tools

Requests        | Solution size | Num.    | Solve time  |  Indexed solve time, no retries
                | (# packages)  | Retries | (seconds)   |  (seconds)
-----------------------------------------------------------------------------------------
python          |        4      |    1    |     0.17    |   0.03 
boost-python    |        8      |    1    |     0.31    |   0.05 
python-torch    |       37      |    2    |     0.58    |   0.15 
widget toolset  |       60      |    2    |     3.44    |   0.48 
katana toolset  |      181      |   10    |    18.32    |   2.97 
nuke toolset    |      280      |   12    |    24.32    |   4.55 
houdini toolset |      211      |   19    |    37.24    |   6.58 
maya toolset    |      403      |   20    |    59.50    |   9.52 

The indexing doesn't have a noticable impact (to users) on smaller solves. But it allows our larger solves to finish in under 10 seconds.

This is the final 5 of 5 chained PRs for adding indexes to spk solves:

1. Indexes 1: Change Package and related traits to not return references to fields #1336
2. Indexes 2: Add new_unchecked() constructors to spk schema objects #1337
3. Indexes 3: Adds flatbuffers schema and SolverPackageSpec for indexes to spk #1338
4. Indexes 4: Adds Indexes for SPK repositories #1339
5. this PR

[codecov]

Codecov Report

❌ Patch coverage is 40.67797% with 70 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/spk-cli/cmd-repo/src/cmd_repo.rs	0.00%	43 Missing ⚠️
crates/spk-cli/common/src/flags.rs	42.10%	22 Missing ⚠️
crates/spk-storage/src/storage/indexed.rs	0.00%	4 Missing ⚠️
crates/spk-storage/src/storage/flatbuffer_index.rs	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[dcookspi]

Let me know if this should error rather than generate a full index.

[dcookspi]

This is the default we intend to use, but maybe it should be configurable.

[dcookspi]

The plan is not to address this in this PR-chain. Users can specify --no-indexes if they want info when indexes are the default. Although, perhaps we should disable index use for spk info ? That would probably be better.

[dcookspi]

This change and the next one make the times line up in the output.