Support rekor v2

[jku]

I don't have a clear plan yet, just writing down the context first...

background

Rekor is being rewritten to be more scalable and simple in rekor-tiles. Some of the changes are relevant to clients like sigstore-python, see design doc (shared with https://groups.google.com/g/sigstore-dev, join the group to get access)
The main changes:

• auditing the log requires using the new tiles API (this should not affect a basic sigstore client)
• less common entry types are no longer supported (should not affect sigstore-python?)
• search index and attestation storage are not currently supported (should not affect sigstore-python)
• signed entry timestamps (SET) are no longer supported: clients must use a timestamp authority (TSA). sigstore-python already support TSAs. Staging already has a TSA in https://timestamp.sigstage.dev
• the request types in Create Entry API change: this affects sigstore-python
• Create Entry API now returns a TransparencyLogEntry from protobuf-specs instead of a separately defined OpenAPI object
• the entry (that ends up in the signature bundle) now contains a canonicalized_body that is now different (in a way that is still unclear to me)
• entries are now batch processed: this means clients will have to wait until next batch processing to get their TransparencyLogEntry. This lag of potentially multiple seconds has implications on the UX (we may have to warn user) and may require changing how multiple artifacts are signed: at the moment no multi-artifact signing API is planned so clients will have to parallelize requests to get multiple artifacts into the same batch.
• TODO: add trustedroot changes and infra switchover plans: rekor v1 instance and new rekor-tiles instance will both be available for a time.

sigstore-python plan

# TODO

[di]

Probably goes without saying but it would be nice if we could roll this out gradually as a pre-release before making a potentially breaking release.

[jku]

Sounds good to me -- although it's a bit more complicated than just pre-release (since some of the client decisions, like the rekor instances used, are going to be made based on trustedroot and signingconfig changes).

That said there's no intention to make breaking changes: plan is to run both rekor instances for some time: verify should then work with entries from either one and signing should switch over when the signingconfig from the tuf repository says so -- the trustedroot/signingconfig should be available before that happens so we'll be able to test before that (this is sigstore/rekor-tiles#38, sigstore/rekor-tiles#150 )

[haydentherapper]

sigstore/rekor-tiles#255 has more info on required changes.

[jku]

I need to update this description on current state... but the main missing part at this point is the "rekor signing client":

• Ramon has a working POC [DRAFT] Rekor v1 and v2 support via signing-config #1387 🥇
• POC uses the same RekorClient for both APIs
  • I've suggested separating these: I think it would make mistakes with v1 support less likely. It would mean a little more code but IMO easier to reason about and to maintain
• POC contains rekor-tiles protobufs -- these will likely come from protobuf-specs in future: including them in the POC seems logical
  • these will mostly enable handling the canonicalized entry changes
  • I think we should still make sure verify fails in a reasonable way when we see an entry type (kindVersion) we do not recognise: see Cannot verify npm attestations #1384 for example

[ramonpetgrave64]

@jku I've changed the draft PR to use a new RekorV2Client class.

We expect RekorV1 to be supported for at least an additional 18 months during RekorV2's availability. So far, my plan is to have the user select to use V2 only with a trust_config file. I'd also like some additional guidance on the UX for selecting whether to use rekorV1 or rekorV2.

1. Do we want to immediately switch over to signing with v2 by default?
2. Should we have another CLI switch for the user to specify they wish to use v2 for signing, or to use
   V1 for signing if V2 is the default?

[jku]

Do we want to immediately switch over to signing with v2 by default?

I see two potential paths for the signing code:

1. Add rekor v2 client module first, integrate later
   • add the rekor v2 client before we integrate it fully into the CLI client: this way we can run unit tests or use example binaries against a staging/testing rekor
   • Once we're happy with the rekorv2 module and all other work is done, we integrate it in the sigstore client: at this point the client will use rekorv2 if the SigningConfig instructs it to do so
2. Integrate the new the rekor v2 module into the sigstore client in the rekorv2 module PR

• client will now use the rekorv2 if the SigningConfig instructs it to do so

As a practical development path, I think option 1 makes most sense:

• it let's us keep PRs reasonably sized and nothing prevents us from still releasing everything in a single release if it all looks good
• let me know if you think this is means significantly more work

your draft PR is still really useful since it shows all the things we still need to fix.

Should we have another CLI switch for the user to specify they wish to use v2 for signing, or to use
V1 for signing if V2 is the default?

As a documented stable feature of the client, I don't think we should provide a choice. However we could have something like a --experimental flag (to enable rekorv2 only if the flag is given) as a way to release earlier than we otherwise would be confortable... We can decide on that once the all the ground work is done and we can see where we are (compared to e.g. when staging trust root and signing config get updated)

[ramonpetgrave64]

@jku That's a good, plan. What about the types? Should we get them into sigstore/protobuf-specs first?

[jku]

This is done, with the exception of #1471