Default to using the new protobuf format

[steiza]

Summary

This is a first step towards #4221, where we default --new-bundle-format to true instead of false.

We never did add protobuf support to cosign sign (see #3927 and #3139) - maybe we want to wait until that is done? Otherwise we could just have --new-bundle-format default to true when we add it to cosign sign.

Release Note

• Modified attest, attest-blob, sign, sign-blob, dockerfile-verify, manifest-verify, verify, verify-attestation, verify-blob, verify-blob-attestation so that --new-bundle-format defaults to true instead of false.

Documentation

Ran make docgen as part of this PR.

[codecov]

Codecov Report

❌ Patch coverage is 10.81081% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 34.32%. Comparing base (2ef6022) to head (535b5c0).
⚠️ Report is 515 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/cosign/cli/verify/verify.go	0.00%	14 Missing ⚠️
cmd/cosign/cli/verify/verify_attestation.go	0.00%	14 Missing ⚠️
cmd/cosign/cli/options/attest.go	0.00%	1 Missing ⚠️
cmd/cosign/cli/options/attest_blob.go	0.00%	1 Missing ⚠️
cmd/cosign/cli/options/sign.go	0.00%	1 Missing ⚠️
cmd/cosign/cli/options/signblob.go	0.00%	1 Missing ⚠️
cmd/cosign/cli/options/verify.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4318      +/-   ##
==========================================
- Coverage   40.10%   34.32%   -5.78%     
==========================================
  Files         155      217      +62     
  Lines       10044    15549    +5505     
==========================================
+ Hits         4028     5337    +1309     
- Misses       5530     9520    +3990     
- Partials      486      692     +206     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[haydentherapper]

Overall LGTM. Let's hold off on this PR until we land #4316 and #4319, do a pass over the other open PRs, and review the Cosign v3 brainstorm to see if there's anything else we want to get in. Then let's cut a new release of Cosign, hopefully the last v2 release, and then start merging these breaking changes.

[steiza]

So, in some senses this is not a breaking change, in that we've made it so that verify-blob and verify-blob-attestation will "fall back" to trying the old bundle format if they can't parse the provided bundle using the protobuf:

$ cat test.sigstore.json 
{"base64Signature":"MEUCIEjqaXVxJgbngDBIXX9DnZ9yuQBK2Y2dU2C1O74T0RvlAiEA9WI9il4+bHMVtzGl8lxrX3SxCfdg5IMUT90Zk5roOu8="}

Verifies if you explicitly say it's the old bundle format:

$ go run cmd/cosign/main.go verify-blob --bundle=test.sigstore.json --new-bundle-format=false --key=cosign.pub --use-signed-timestamps=false --insecure-ignore-tlog=true ../sigstore-go/examples/sigstore-go-signing/hello_world.txt 
...
Verified OK

... but also verifies if you omit the bundle format (defaults to true):

$ go run cmd/cosign/main.go verify-blob --bundle=test.sigstore.json --key=cosign.pub --use-signed-timestamps=false --insecure-ignore-tlog=true ../sigstore-go/examples/sigstore-go-signing/hello_world.txt 
...
Verified OK

I spent some time looking into providing this fallback behavior for OCI verify commands (verify, verify-attestation) as well. It's technical possible of course, but it would either add quite a few requests to your registry for each verify (essentially doubling the requests) or require a somewhat substantial refactor in order to query once and then pass the results along to pkg/cosign/verify.go functions - what do you think?

[steiza]

There's some issues with how the trusted root file is working in verify - I'll have to take a look later

[steiza]

I couldn't get the scaffolding working with the new bundle format - I think there was some issue with how the trusted root was constructed - but we can update it when we overhaul the scaffolding test.

[steiza]

Okay! With #4346 I think the old scaffolding is working with the new protobuf bundle format. I also added code for verify and verify-attestation to detect if you are using the old signature format and to gracefully fallback. If tests pass, I think this is ready to review.

[haydentherapper]

The one downside with this is that while it's not a breaking change for verification of old metadata, it is a breaking change for verifying new metadata, in that you can't use an old client to verify new signatures. Are we OK with that?

[steiza]

The one downside with this is that while it's not a breaking change for verification of old metadata, it is a breaking change for verifying new metadata, in that you can't use an old client to verify new signatures. Are we OK with that?

I think so. I would hope that an organization would upgrade it's toolchain (for signing and verifying) to a new version at the same time.

If we don't think that's reasonable, the alternative (which I'm okay with) is cutting a last cosign 2.x release before this lands and have this be the start of cosign v3!

[haydentherapper]

Fantastic!

[haydentherapper]

We'll make an extra network call with GetBundles - do you think that's fine? Or should we store the bundles it finds and pass those to the verification function? Not sure if that's straightforward though.

[haydentherapper]

Meant to approve, just one comment!

[haydentherapper]

Just to confirm, this should handle backwards compatibility for verification of either the new or old bundle format?