|
| 1 | +# Release Tasks |
| 2 | + |
| 3 | +**Notes**: |
| 4 | + |
| 5 | +- The examples in this document are based on the v1.8 release cycle. |
| 6 | + |
| 7 | +<!-- START doctoc generated TOC please keep comment here to allow auto update --> |
| 8 | +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> |
| 9 | + |
| 10 | +- [Prepare main branch for development of the new release](#prepare-main-branch-for-development-of-the-new-release) |
| 11 | +- [Remove previously deprecated code](#remove-previously-deprecated-code) |
| 12 | +- [[Optional] Bump the Kubernetes version](#optional-bump-the-kubernetes-version) |
| 13 | +- [Bump dependencies](#bump-dependencies) |
| 14 | +- [Create a release branch](#create-a-release-branch) |
| 15 | +- [Cut a release](#cut-a-release) |
| 16 | +- [[Continuously] Reduce the amount of flaky tests](#continuously-reduce-the-amount-of-flaky-tests) |
| 17 | + |
| 18 | +<!-- END doctoc generated TOC please keep comment here to allow auto update --> |
| 19 | + |
| 20 | +## Prepare main branch for development of the new release |
| 21 | + |
| 22 | +TODO(sbueringer): Finalize this section once we do this for the first time |
| 23 | + |
| 24 | +The goal of this issue is to bump the versions on the main branch so that the upcoming release version |
| 25 | +is used for e.g. local development and e2e tests. We also modify tests so that they are testing the previous release. |
| 26 | + |
| 27 | +This comes down to changing occurrences of the old version to the new version, e.g. `v1.7` to `v1.8`: |
| 28 | + |
| 29 | +1. Setup E2E tests for the new release: |
| 30 | + 1. Goal is that our clusterctl upgrade tests are testing the right versions. For `v1.8` this means: |
| 31 | + - v1beta1: `v1.7` (will change with each new release) |
| 32 | + - v1alpha4: `v0.8` |
| 33 | + 2. Update providers in `vsphere-ci.yaml`, `vsphere-dev.yaml`, `integration-dev.yaml`: |
| 34 | + 1. Add a new `v1.7.0` entry. |
| 35 | + 2. Remove providers that are not used anymore in clusterctl upgrade tests. |
| 36 | + 3. Change `v1.7.99` to `v1.8.99`. |
| 37 | + 3. Adjust `metadata.yaml`'s: |
| 38 | + 1. Add new release to the top-level `metadata.yaml` |
| 39 | + 2. Create a new `v1.7` `metadata.yaml` (`test/e2e/data/shared/v1.7/metadata.yaml`) by copying |
| 40 | + `test/e2e/data/shared/main/metadata.yaml` |
| 41 | + 3. Add the new v1.8 release to the main `metadata.yaml` (`test/e2e/data/shared/main/metadata.yaml`). |
| 42 | + 4. Remove old `metadata.yaml`'s that are not used anymore in clusterctl upgrade tests. |
| 43 | + 4. Adjust cluster templates in `test/e2e/data/infrastructure-vsphere`: |
| 44 | + 1. Create a new `v1.7` folder. It should be created based on the `main` folder and only contain the templates |
| 45 | + we use in the clusterctl upgrade tests (as of today `remote-management`). |
| 46 | + 2. Remove old folders that are not used anymore in clusterctl upgrade tests. |
| 47 | + 5. Modify the test specs in `test/e2e/capi_clusterctl_upgrade_test.go` (according to the versions we want to test described above). |
| 48 | + Please note that both `InitWithKubernetesVersion` and `WorkloadKubernetesVersion` should be the highest mgmt cluster version supported by the respective Cluster API version. |
| 49 | +2. Update `clusterctl-settings.json`: `v1.7.99` => `v1.8.99`. |
| 50 | +3. Make sure all tests are green (also run `pull-cluster-api-provider-vsphere-e2e-full-main` and `pull-cluster-api-provider-vsphere-conformance-main`). |
| 51 | + |
| 52 | +Prior art: TODO(sbueringer): link example PR |
| 53 | + |
| 54 | +## Remove previously deprecated code |
| 55 | + |
| 56 | +The goal of this task is to remove all previously deprecated code that can be now removed. |
| 57 | + |
| 58 | +1. Check for deprecated code and remove it. |
| 59 | + **Note**: We can't just remove all code flagged with `Deprecated`. In some cases like e.g. in API packages |
| 60 | + we have to keep the old code. |
| 61 | + |
| 62 | +Prior art: TODO(sbueringer): link example PR |
| 63 | + |
| 64 | +## [Optional] Bump the Kubernetes version |
| 65 | + |
| 66 | +TODO(sbueringer): Write this when we do it the next time |
| 67 | + |
| 68 | +## Bump dependencies |
| 69 | + |
| 70 | +The goal of this task is to ensure that we have relatively up-to-date dependencies at the time of the release. |
| 71 | +This reduces the risk that CVEs are found in outdated dependencies after our release. |
| 72 | + |
| 73 | +We should take a look at the following dependencies: |
| 74 | + |
| 75 | +- Go dependencies in the `go.mod` file. (usually dependabot takes care of that) |
| 76 | +- Tools used in our Makefile (e.g. kustomize). |
| 77 | + |
| 78 | +## Create a release branch |
| 79 | + |
| 80 | +The goal of this task is to ensure we have a release branch with test coverage and results in testgrid. While we |
| 81 | +add test coverage for the new release branch we will also drop the tests for old release branches if necessary. |
| 82 | +The milestone applier should also apply milestones accordingly. |
| 83 | +From this point forward changes which should land in the release have to be cherry-picked into the release branch. |
| 84 | + |
| 85 | +1. Create the release branch locally based on the latest commit on main and push it. |
| 86 | + |
| 87 | + ```bash |
| 88 | + # Create the release branch |
| 89 | + git checkout -b release-1.8 |
| 90 | + |
| 91 | + # Push the release branch |
| 92 | + # Note: `upstream` must be the remote pointing to `github.com/kubernetes-sigs/cluster-api`. |
| 93 | + git push -u upstream release-1.8 |
| 94 | + ``` |
| 95 | + |
| 96 | +2. Create the milestone for the new release via [GitHub UI](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/milestones/new). |
| 97 | +3. Update the [milestone applier config](https://github.com/kubernetes/test-infra/blob/151bab62dc023525f592e6d1fdc2a8de5305cd01/config/prow/plugins.yaml#L523) accordingly (e.g. `release-1.8: v1.8` |
| 98 | + and `main: v1.9`) |
| 99 | +4. Create new jobs based on the jobs running against our `main` branch: |
| 100 | + 1. Copy `config/jobs/kubernetes-sigs/cluster-api-provider-vsphere/cluster-api-provider-vsphere-periodics-main.yaml` to `config/jobs/kubernetes-sigs/cluster-api-provider-vsphere/cluster-api-provider-vsphere-periodics-release-1.8.yaml`. |
| 101 | + 2. Copy `config/jobs/kubernetes-sigs/cluster-api-provider-vsphere/cluster-api-provider-vsphere-presubmits-main.yaml` to `config/jobs/kubernetes-sigs/cluster-api-provider-vsphere/cluster-api-provider-vsphere-presubmits-release-1.8.yaml`. |
| 102 | + 3. Modify the following: |
| 103 | + 1. Rename the jobs, e.g.: `periodic-cluster-api-provider-vsphere-test-main` => `periodic-cluster-api-provider-vsphere-test-release-1-8`. |
| 104 | + 2. Change `annotations.testgrid-tab-name`, e.g. `periodic-test-main` => `periodic-test-release-1-8`. |
| 105 | + 3. For periodics additionally: |
| 106 | + - Change `extra_refs[].base_ref` to `release-1.8` (for repo: `cluster-api-provider-vsphere`). |
| 107 | + 4. For presubmits additionally: |
| 108 | + - Adjust branches: `^main$` => `^release-1.8$`. |
| 109 | +5. Remove tests for old release branches if necessary |
| 110 | +6. Verify the jobs and dashboards a day later by taking a look at [testgrid](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-provider-vsphere) |
| 111 | +7. Update `.github/workflows/scan.yaml` - to set up Trivy scanning for the currently supported branches. |
| 112 | + |
| 113 | +## Cut a release |
| 114 | + |
| 115 | +1. Ensure via testgrid that CI is stable before cutting the release |
| 116 | + Note: special attention should be given to image scan results, so we can avoid cutting a release with CVEs or document known CVEs in release notes. |
| 117 | +2. Create and push the release tags to the GitHub repository: |
| 118 | + |
| 119 | + ```bash |
| 120 | + # Export the tag of the release to be cut, e.g.: |
| 121 | + export RELEASE_TAG=v1.8.0-beta.0 |
| 122 | + # Create tags locally |
| 123 | + # Warning: The test tag MUST NOT be an annotated tag. |
| 124 | + git tag -s -a ${RELEASE_TAG} -m ${RELEASE_TAG} |
| 125 | +
|
| 126 | + # Push tags |
| 127 | + # Note: `upstream` must be the remote pointing to `github.com/kubernetes-sigs/cluster-api`. |
| 128 | + git push upstream ${RELEASE_TAG} |
| 129 | + ``` |
| 130 | + |
| 131 | + This will automatically trigger: |
| 132 | + - a [GitHub Action](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/actions/workflows/release.yaml) to create a draft release and |
| 133 | + - a [ProwJob](https://prow.k8s.io/?repo=kubernetes-sigs%2Fcluster-api-provider-vsphere&job=post-cluster-api-provider-vsphere-push-images) to publish images to the staging image repository. |
| 134 | +3. Promote images from the staging repository to the production registry (`registry.k8s.io/cluster-api-vsphere`): |
| 135 | + 1. Wait until images for the tag have been built and pushed to the [staging repository](https://console.cloud.google.com/gcr/images/k8s-staging-capi-vsphere/global/cluster-api-vsphere-controller) by |
| 136 | + the [push images job](https://prow.k8s.io/?repo=kubernetes-sigs%2Fcluster-api-provider-vsphere&job=post-cluster-api-provider-vsphere-push-images). |
| 137 | + 2. If you don't have a GitHub token, create one by going to your GitHub settings in [Personal access tokens](https://github.com/settings/tokens). Make sure you give the token the `repo` scope. |
| 138 | + 3. Create a PR to promote the images to the production registry: |
| 139 | +
|
| 140 | + ```bash |
| 141 | + # Export the tag of the release to be cut, e.g.: |
| 142 | + export RELEASE_TAG=v1.8.0-beta.0 |
| 143 | + export GITHUB_TOKEN=<your GH token> |
| 144 | + make promote-images |
| 145 | + ``` |
| 146 | +
|
| 147 | + **Notes**: |
| 148 | + - `make promote-images` target tries to figure out your Github user handle in order to find the forked [k8s.io](https://github.com/kubernetes/k8s.io) repository. |
| 149 | + If you have not forked the repo, please do it before running the Makefile target. |
| 150 | + - if `make promote-images` fails with an error like `FATAL while checking fork of kubernetes/k8s.io` you may be able to solve it by manually setting the USER_FORK variable |
| 151 | + i.e. `export USER_FORK=<personal GitHub handle>` |
| 152 | + - `kpromo` uses `[email protected]:...` as remote to push the branch for the PR. If you don't have `ssh` set up you can configure |
| 153 | + git to use `https` instead via `git config --global url."https://github.com/".insteadOf [email protected]:`. |
| 154 | + - This will automatically create a PR in [k8s.io](https://github.com/kubernetes/k8s.io) and assign the CAPI maintainers. |
| 155 | +4. Merge the PR (/lgtm + /hold cancel) and verify the images are available in the production registry: |
| 156 | + - Wait for the [promotion prow job](https://prow.k8s.io/?repo=kubernetes%2Fk8s.io&job=post-k8sio-image-promo) to complete successfully. Then verify that the production images are accessible: |
| 157 | + |
| 158 | + ```bash |
| 159 | + docker pull registry.k8s.io/cluster-api-vsphere/cluster-api-vsphere-controller:${RELEASE_TAG} |
| 160 | + ``` |
| 161 | + |
| 162 | +5. Publish the release. |
| 163 | + 1. Finalize release notes |
| 164 | + 1. Pay close attention to the `## :question: Sort these by hand` section, as it contains items that need to be manually sorted. |
| 165 | + 2. Ensure consistent formatting of entries (e.g. prefix). |
| 166 | + 3. Merge dependency bump PR entries for the same dependency into a single entry. |
| 167 | + 4. Move minor changes into a single line at the end of each section. |
| 168 | + 5. Sort entries within a section alphabetically. |
| 169 | + 6. Write highlights section based on the initial release notes doc. (for minor releases and important changes only) |
| 170 | + 7. **For minor releases** Modify `Changes since v1.x.y` to `Changes since v1.x` |
| 171 | + **Note**: The release notes tool includes all merges since the previous release branch was branched of. |
| 172 | + 2. Publish the release and ensure release is flagged as `pre-release` for all `beta` and `rc` releases or `latest` for a new release in the most recent release branch. |
| 173 | +6. **For minor releases** Update supported versions in versions.md. |
| 174 | + |
| 175 | +- Cutting a release as of today requires permissions to: |
| 176 | + - Create a release tag on the GitHub repository. |
| 177 | + - Create/update/publish GitHub releases. |
| 178 | + |
| 179 | +## [Continuously] Reduce the amount of flaky tests |
| 180 | + |
| 181 | +The CAPV tests are pretty stable, but there are still some flaky tests from time to time. To reduce the amount of flakes please periodically: |
| 182 | + |
| 183 | +1. Take a look at recent CI failures via `k8s-triage`: |
| 184 | + - [periodic-cluster-api-provider-vsphere-test-main](https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=periodic-cluster-api-provider-vsphere-test-main) |
| 185 | + - [periodic-cluster-api-provider-vsphere-test-integration-main](https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=periodic-cluster-api-provider-vsphere-test-integration-main) |
| 186 | + - [periodic-cluster-api-provider-vsphere-e2e-full-main](https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=periodic-cluster-api-provider-vsphere-e2e-full-main) |
| 187 | + - [periodic-cluster-api-provider-vsphere-conformance-main](https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=periodic-cluster-api-provider-vsphere-conformance-main) |
| 188 | +2. Open issues for occurring flakes and ideally fix them or find someone who can. |
0 commit comments