Merge branch 'zcash-v2.4.2' into zsa1-v2.4.2-conflicts-new

Author: dmidem

.codespellrc

@@ -1,4 +1,4 @@
 [codespell]
-ignore-words-list = crate,Sur,inout,Groth,groth,re-use,
+ignore-words-list = crate,Sur,inout,Groth,groth,re-use,abl,
 exclude-file = book/mermaid.min.js
 skip = ./zebra-rpc/qa/rpc-tests,./supply-chain

.dockerignore

@@ -21,3 +21,4 @@
 !zebra-*
 !zebrad
 !docker/entrypoint.sh
+!docker/default-zebra-config.toml

.github/ISSUE_TEMPLATE/release.md

@@ -2,7 +2,7 @@
 name: "🚀 Zebra Release"
 about: 'Zebra team use only'
 title: 'Publish next Zebra release: (version)'
-labels: 'A-release, C-trivial, P-Medium :zap:'
+labels: 'A-release, C-exclude-from-changelog, P-Medium :zap:'
 assignees: ''
 
 ---
@@ -16,7 +16,7 @@ They can be skipped for urgent releases.
 
 To check consensus correctness, we want to test that the state format is valid after a full sync. (Format upgrades are tested in CI on each PR.)
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or
+- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=event%3Aschedule) since the last state change, or
 - [ ] Start a manual workflow run with a Zebra and `lightwalletd` full sync.
 
 State format changes can be made in `zebra-state` or `zebra-chain`. The state format can be changed by data that is sent to the state, data created within the state using `zebra-chain`, or serialization formats in `zebra-state` or `zebra-chain`. 

.github/ISSUE_TEMPLATE/usability_testing_plan.md

@@ -31,7 +31,7 @@ assignees: ''
 
 ### Method
 
-<!-- Explain whether the tests will be co-located or reomte, moderated or unmoderated, and who will attend the sessions (e.g a moderator and a note-taker) -->
+<!-- Explain whether the tests will be co-located or remote, moderated or unmoderated, and who will attend the sessions (e.g a moderator and a note-taker) -->
 
 ### Test environment, equipment and logistics
 
@@ -56,7 +56,7 @@ assignees: ''
 
 ## Session Outline and timing
 
-<!-- The following sections provide some space to plan out the script and and tasks for your participants -->
+<!-- The following sections provide some space to plan out the script and tasks for your participants -->
 
 ### 1.Introduction to the session (5\')
 

.github/PULL_REQUEST_TEMPLATE/hotfix-release-checklist.md

@@ -0,0 +1,100 @@
+---
+name: 'Hotfix Release Checklist Template'
+about: 'Checklist to create and publish a hotfix Zebra release'
+title: 'Release Zebra (version)'
+labels: 'A-release, C-exclude-from-changelog, P-Critical :ambulance:'
+assignees: ''
+
+---
+
+A hotfix release should only be created when a bug or critical issue is discovered in an existing release, and waiting for the next scheduled release is impractical or unacceptable.
+
+## Create the Release PR
+
+- [ ] Create a branch to fix the issue based on the tag of the release being fixed (not the main branch).
+      for example: `hotfix-v2.3.1` - this needs to be different to the tag name
+- [ ] Make the required changes
+- [ ] Create a hotfix release PR by adding `&template=hotfix-release-checklist.md` to the comparing url ([Example](https://github.com/ZcashFoundation/zebra/compare/bump-v1.0.0?expand=1&template=hotfix-release-checklist.md)).
+- [ ] Add the `C-exclude-from-changelog` label so that the PR is omitted from the next release changelog
+- [ ] Add the `A-release` tag to the release pull request in order for the `check_no_git_refs_in_cargo_lock` to run.
+- [ ] Ensure the `check_no_git_refs_in_cargo_lock` check passes.
+- [ ] Add a changelog entry for the release summarizing user-visible changes.
+
+## Update Versions
+
+The release level for a hotfix should always follow semantic versioning as a `patch` release.
+
+<details>
+<summary>Update crate versions, commit the changes to the release branch, and do a release dry-run:</summary>
+
+```sh
+# Update everything except for alpha crates and zebrad:
+cargo release version --verbose --execute --allow-branch '*' --workspace --exclude zebrad beta
+# Due to a bug in cargo-release, we need to pass exact versions for alpha crates:
+# Update zebrad:
+cargo release version --verbose --execute --allow-branch '*' --package zebrad patch
+# Continue with the release process:
+cargo release replace --verbose --execute --allow-branch '*' --package zebrad
+cargo release commit --verbose --execute --allow-branch '*'
+```
+
+</details>
+
+## Update the Release PR
+
+- [ ] Push the version increments and the release constants to the hotfix release branch.
+
+# Publish the Zebra Release
+
+## Create the GitHub Pre-Release
+
+- [ ] Wait for the hotfix release PR to be reviewed, approved, and merged into main.
+- [ ] Create a new release
+- [ ] Set the tag name to the version tag,
+      for example: `v2.3.1`
+- [ ] Set the release to target the hotfix release branch
+- [ ] Set the release title to `Zebra ` followed by the version tag,
+      for example: `Zebra 2.3.1`
+- [ ] Populate the release description with the final changelog you created;
+      starting just _after_ the title `## [Zebra ...` of the current version being released,
+      and ending just _before_ the title of the previous release.
+- [ ] Mark the release as 'pre-release', until it has been built and tested
+- [ ] Publish the pre-release to GitHub using "Publish Release"
+
+## Test the Pre-Release
+
+- [ ] Wait until the Docker binaries have been built on the hotfix release branch, and the quick tests have passed:
+    - [ ] [ci-tests.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml)
+- [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/cd-deploy-nodes-gcp.yml?query=event%3Arelease)
+
+## Publish Release
+
+- [ ] [Publish the release to GitHub](https://github.com/ZcashFoundation/zebra/releases) by disabling 'pre-release', then clicking "Set as the latest release"
+
+## Publish Crates
+
+- [ ] Checkout the hotfix release branch
+- [ ] [Run `cargo login`](https://zebra.zfnd.org/dev/crate-owners.html#logging-in-to-cratesio)
+- [ ] Run `cargo clean` in the zebra repo
+- [ ] Publish the crates to crates.io: `cargo release publish --verbose --workspace --execute --allow-branch {hotfix-release-branch}`
+- [ ] Check that the published version of Zebra can be installed from `crates.io`:
+      `cargo install --locked --force --version 2.minor.patch zebrad && ~/.cargo/bin/zebrad`
+      and put the output in a comment on the PR.
+
+## Publish Docker Images
+
+- [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml?query=event%3Arelease).
+- [ ] Wait for the new tag in the [dockerhub zebra space](https://hub.docker.com/r/zfnd/zebra/tags)
+
+## Merge hotfix into main
+
+- [ ] Review and merge the hotfix branch into the main branch. The changes and the update to the changelog must be included in the next release from main as well.
+- [ ] If there are conflicts between the hotfix branch and main, the conflicts should be resolved after the hotfix release is tagged and published.
+
+## Release Failures
+
+If building or running fails after tagging:
+
+<details>
+1. Create a new hotfix release, starting from the top of this document.
+</details>

.github/PULL_REQUEST_TEMPLATE/release-checklist.md

@@ -2,17 +2,32 @@
 name: 'Release Checklist Template'
 about: 'Checklist to create and publish a Zebra release'
 title: 'Release Zebra (version)'
-labels: 'A-release, C-trivial, P-Critical :ambulance:'
+labels: 'A-release, C-exclude-from-changelog, P-Critical :ambulance:'
 assignees: ''
 
 ---
 
 # Prepare for the Release
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or start a manual full sync.
-- [ ] Make sure the PRs with the new checkpoint hashes and missed dependencies are already merged.
-      (See the release ticket checklist for details)
+- [ ] Make sure there has been [at least one successful full sync test in the main branch](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=branch%3Amain) since the last state change, or start a manual full sync.
 
+# Checkpoints
+
+For performance and security, we want to update the Zebra checkpoints in every release.
+- [ ] You can copy the latest checkpoints from CI by following [the zebra-checkpoints README](https://github.com/ZcashFoundation/zebra/blob/main/zebra-utils/README.md#zebra-checkpoints).
+
+# Missed Dependency Updates
+
+Sometimes `dependabot` misses some dependency updates, or we accidentally turned them off.
+
+This step can be skipped if there is a large pending dependency upgrade. (For example, shared ECC crates.)
+
+Here's how we make sure we got everything:
+- [ ] Run `cargo update` on the latest `main` branch, and keep the output
+- [ ] If needed, [add duplicate dependency exceptions to deny.toml](https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/continuous-integration.md#fixing-duplicate-dependencies-in-check-denytoml-bans)
+- [ ] If needed, remove resolved duplicate dependencies from `deny.toml`
+- [ ] Open a separate PR with the changes
+- [ ] Add the output of `cargo update` to that PR as a comment
 
 # Summarise Release Changes
 
@@ -26,7 +41,9 @@ Once you are ready to tag a release, copy the draft changelog into `CHANGELOG.md
 We use [the Release Drafter workflow](https://github.com/marketplace/actions/release-drafter) to automatically create a [draft changelog](https://github.com/ZcashFoundation/zebra/releases). We follow the [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
 
 To create the final change log:
-- [ ] Copy the **latest** draft changelog into `CHANGELOG.md` (there can be multiple draft releases)
+- [ ] Copy the [**latest** draft
+  changelog](https://github.com/ZcashFoundation/zebra/releases) into
+  `CHANGELOG.md` (there can be multiple draft releases)
 - [ ] Delete any trivial changes
     - [ ] Put the list of deleted changelog entries in a PR comment to make reviewing easier
 - [ ] Combine duplicate changes
@@ -57,7 +74,13 @@ fastmod --fixed-strings '1.58' '1.65'
 - [ ] Freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Mark all the release PRs as `Critical` priority, so they go in the `urgent` Mergify queue.
 - [ ] Mark all non-release PRs with `do-not-merge`, because Mergify checks approved PRs against every commit, even when a queue is frozen.
+- [ ] Add the `A-release` tag to the release pull request in order for the `check-no-git-dependencies` to run.
+
+## Zebra git sources dependencies
 
+- [ ] Ensure the `check-no-git-dependencies` check passes.
+
+This check runs automatically on pull requests with the `A-release` label. It must pass for crates to be published to crates.io. If the check fails, you should either halt the release process or proceed with the understanding that the crates will not be published on crates.io.
 
 # Update Versions and End of Support
 
@@ -72,38 +95,39 @@ Choose a release level for `zebrad`. Release levels are based on user-visible ch
 - significant new features or behaviour changes; changes to RPCs, command-line, or configs; and deprecations or removals are `minor` releases
 - otherwise, it is a `patch` release
 
-Zebra's Rust API doesn't have any support or stability guarantees, so we keep all the `zebra-*` and `tower-*` crates on a beta `pre-release` version.
-
-### Update Crate Versions
+### Update Crate Versions and Crate Change Logs
 
-If you're publishing crates for the first time, [log in to crates.io](https://github.com/ZcashFoundation/zebra/blob/doc-crate-own/book/src/dev/crate-owners.md#logging-in-to-cratesio),
+If you're publishing crates for the first time, [log in to crates.io](https://zebra.zfnd.org/dev/crate-owners.html#logging-in-to-cratesio),
 and make sure you're a member of owners group.
 
 Check that the release will work:
-- [ ] Update crate versions, commit the changes to the release branch, and do a release dry-run:
+
+- [ ] Determine which crates require release. Run `git diff --stat <previous_tag>`
+      and enumerate the crates that had changes.
+- [ ] Determine which type of release to make. Run `semver-checks` to list API
+      changes: `cargo semver-checks -p <crate> --default-features`. If there are
+      breaking API changes, do a major release, or try to revert the API change
+      if it was accidental. Otherwise do a minor or patch release depending on
+      whether a new API was added. Note that `semver-checks` won't work
+      if the previous realase was yanked; you will have to determine the
+      type of release manually.
+- [ ] Update the crate `CHANGELOG.md` listing the API changes or other
+      relevant information for a crate consumer. It might make sense to copy
+      entries from the `zebrad` changelog.
+- [ ] Update crate versions:
 
 ```sh
-# Update everything except for alpha crates and zebrad:
-cargo release version --verbose --execute --allow-branch '*' --workspace --exclude zebrad --exclude zebra-scan --exclude zebra-grpc beta
-# Due to a bug in cargo-release, we need to pass exact versions for alpha crates:
-cargo release version --verbose --execute --allow-branch '*' --package zebra-scan 0.1.0-alpha.4
-cargo release version --verbose --execute --allow-branch '*' --package zebra-grpc 0.1.0-alpha.2
-# Update zebrad:
-cargo release version --verbose --execute --allow-branch '*' --package zebrad patch # [ major | minor | patch ]
-# Continue with the release process:
-cargo release replace --verbose --execute --allow-branch '*' --package zebrad
-cargo release commit --verbose --execute --allow-branch '*'
+cargo release version --verbose --execute --allow-branch '*' -p <crate> patch # [ major | minor ]
+cargo release replace --verbose --execute --allow-branch '*' -p <crate>
 ```
 
-Crate publishing is [automatically checked in CI](https://github.com/ZcashFoundation/zebra/actions/workflows/release-crates-io.yml) using "dry run" mode, however due to a bug in `cargo-release` we need to pass exact versions to the alpha crates:
-
-- [ ] Update `zebra-scan` and `zebra-grpc` alpha crates in the [release-crates-dry-run workflow script](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/scripts/release-crates-dry-run.sh)
-- [ ] Push the above version changes to the release branch.
+- [ ] Update the crate `CHANGELOG.md`
+- [ ] Commit and push the above version changes to the release branch.
 
 ## Update End of Support
 
 The end of support height is calculated from the current blockchain height:
-- [ ] Find where the Zcash blockchain tip is now by using a [Zcash explorer](https://zcashblockexplorer.com/blocks) or other tool.
+- [ ] Find where the Zcash blockchain tip is now by using a [Zcash Block Explorer](https://mainnet.zcashexplorer.app/) or other tool.
 - [ ] Replace `ESTIMATED_RELEASE_HEIGHT` in [`end_of_support.rs`](https://github.com/ZcashFoundation/zebra/blob/main/zebrad/src/components/sync/end_of_support.rs) with the height you estimate the release will be tagged.
 
 <details>
@@ -141,8 +165,7 @@ The end of support height is calculated from the current blockchain height:
 ## Test the Pre-Release
 
 - [ ] Wait until the Docker binaries have been built on `main`, and the quick tests have passed:
-    - [ ] [ci-unit-tests-docker.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-unit-tests-docker.yml?query=branch%3Amain)
-    - [ ] [ci-integration-tests-gcp.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=branch%3Amain)
+    - [ ] [ci-tests.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-tests.yml?query=branch%3Amain)
 - [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/cd-deploy-nodes-gcp.yml?query=event%3Arelease)
 
 ## Publish Release
@@ -151,15 +174,25 @@ The end of support height is calculated from the current blockchain height:
 
 ## Publish Crates
 
-- [ ] [Run `cargo login`](https://github.com/ZcashFoundation/zebra/blob/doc-crate-own/book/src/dev/crate-owners.md#logging-in-to-cratesio)
-- [ ] Run `cargo clean` in the zebra repo (optional)
-- [ ] Publish the crates to crates.io: `cargo release publish --verbose --workspace --execute`
+- [ ] [Run `cargo login`](https://zebra.zfnd.org/dev/crate-owners.html#logging-in-to-cratesio)
+- [ ] It is recommended that the following step be run from a fresh checkout of
+      the repo, to avoid accidentally publishing files like e.g. logs that might
+      be lingering around
+- [ ] Publish the crates to crates.io; edit the list to only include the crates that
+      have been changed, but keep their overall order:
+
+```
+for c in zebra-test tower-fallback zebra-chain tower-batch-control zebra-node-services zebra-script zebra-state zebra-consensus zebra-network zebra-rpc zebra-utils zebrad; do cargo release publish --verbose --execute -p $c; done
+```
+
 - [ ] Check that Zebra can be installed from `crates.io`:
-      `cargo install --locked --force --version 1.minor.patch zebrad && ~/.cargo/bin/zebrad`
+      `cargo install --locked --force --version <version> zebrad && ~/.cargo/bin/zebrad`
       and put the output in a comment on the PR.
 
 ## Publish Docker Images
+
 - [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml?query=event%3Arelease).
+- [ ] Wait for the new tag in the [dockerhub zebra space](https://hub.docker.com/r/zfnd/zebra/tags)
 - [ ] Un-freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Remove `do-not-merge` from the PRs you added it to
 

.github/dependabot.yml

@@ -3,15 +3,18 @@ updates:
   # Rust section
   - package-ecosystem: cargo
     directory: '/'
+    # Update only the lockfile. We shouldn't update Cargo.toml unless it's for
+    # a security issue, or if we need a new feature of the dependency.
+    versioning-strategy: lockfile-only
     # serde, clap, and other dependencies sometimes have multiple updates in a week
     schedule:
-      interval: weekly
+      interval: monthly
       day: monday
       timezone: America/New_York
     # Limit dependabot to 1 PR per reviewer
     open-pull-requests-limit: 6
     labels:
-      - 'C-trivial'
+      - 'C-exclude-from-changelog'
       - 'A-rust'
       - 'A-dependencies'
       - 'P-Low :snowflake:'
@@ -46,11 +49,11 @@ updates:
       timezone: America/New_York
     open-pull-requests-limit: 4
     labels:
-      - 'C-trivial'
+      - 'C-exclude-from-changelog'
       - 'A-devops'
       - 'A-dependencies'
       - 'P-Low :snowflake:'
     groups:
       devops:
         patterns:
-          - "*"
+          - "*"