Skip to content

Commit f97604b

Browse files
Jiawei0227Jiawei0227
committed
release-tools: update
Commit summary: 7100c12 Only set staging registry when running canary job d129462 Document new method for adding CI jobs are new K8s versions e73c2ce Use staging registry for canary tests 2c09846 Add cleanup instructions to release-notes generation 0979c09 prow.sh: fix E2E suite for Kubernetes >= 1.18 3b4a2f1 prow.sh: fix installing Go for Kubernetes 1.19.0 82d108a switch to Go 1.15 843bddc Add steps on promoting release images 1fdf2d5 cloud build: bump timeout in Prow job 5a54e67 filter-junit: Fix gofmt error 36ea4ff filter-junit: Fix golint error 43e50d6 prow.sh: enable building arm64 image 3df86b7 cloud build: k8s-staging-sig-storage db0c2a7 cloud build: initialize support for running commands in Dockerfile 340e082 build.make: optional inclusion of Windows in multiarch images 5231f05 build.make: properly declare push-multiarch 4569f27 build.make: fix push-multiarch ambiguity bd41690 cloud build: initial set of shared files 6f2322e Update patch release notes generation command d8c76fe Support local snapshot RBAC for pull jobs ea1f94a update release tools instructions 7edc146 Update snapshotter to version 2.0.1 3863a0f build for multiple platforms only in CI, add s390x
2 parents c8b2714 + a0f195c commit f97604b

File tree

7 files changed

+237
-50
lines changed

7 files changed

+237
-50
lines changed

‎release-tools/SIDECAR_RELEASE_PROCESS.md‎

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -39,29 +39,39 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
3939
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
4040
by following the [update
4141
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
42-
1. New pull and CI jobs are configured by
42+
1. New pull and CI jobs are configured by adding new K8s versions to the top of
4343
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
44-
New pull jobs that have been unverified should be initially made optional.
45-
[Example](https://github.com/kubernetes/test-infra/pull/15055)
44+
New pull jobs that have been unverified should be initially made optional by
45+
setting the new K8s version as
46+
[experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
4647
1. Once new pull and CI jobs have been verified, and the new Kubernetes version
4748
is released, we can make the optional jobs required, and also remove the
4849
Kubernetes versions that are no longer supported.
4950

5051
## Release Process
5152
1. Identify all issues and ongoing PRs that should go into the release, and
5253
drive them to resolution.
53-
1. Download [K8s release notes
54+
1. Download v2.8+ [K8s release notes
5455
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
5556
1. Generate release notes for the release. Replace arguments with the relevant
5657
information.
57-
```
58-
GITHUB_TOKEN=<token> ./release-notes --start-sha=0ed6978fd199e3ca10326b82b4b8b8e916211c9b --end-sha=3cb3d2f18ed8cb40371c6d8886edcabd1f27e7b9 \
59-
--github-org=kubernetes-csi --github-repo=external-attacher -branch=master -output out.md
60-
```
61-
* `--start-sha` should point to the last release from the same branch. For
62-
example:
63-
* `1.X-1.0` tag when releasing `1.X.0`
64-
* `1.X.Y-1` tag when releasing `1.X.Y`
58+
* Clean up old cached information (also needed if you are generating release
59+
notes for multiple repos)
60+
```bash
61+
rm -rf /tmp/k8s-repo
62+
```
63+
* For new minor releases on master:
64+
```bash
65+
GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
66+
--github-org=kubernetes-csi --github-repo=external-provisioner
67+
--required-author="" --output out.md
68+
```
69+
* For new patch releases on a release branch:
70+
```bash
71+
GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
72+
--github-org=kubernetes-csi --github-repo=external-provisioner
73+
--required-author="" --output out.md
74+
```
6575
1. Compare the generated output to the new commits for the release to check if
6676
any notable change missed a release note.
6777
1. Reword release notes as needed. Make sure to check notes for breaking
@@ -82,6 +92,12 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
8292
[external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
8393
1. If release was a new major/minor version, create a new `release-<minor>`
8494
branch at that commit.
95+
1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
96+
1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
97+
the [k8s image
98+
repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
99+
run `./generate.sh > images.yaml`, and send a PR with the updated images.
100+
Once merged, the image promoter will copy the images from staging to prod.
85101
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
86102
and feature pages with the new released version.
87103
1. After all the sidecars have been released, update

‎release-tools/build.make‎

Lines changed: 88 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -60,23 +60,30 @@ else
6060
TESTARGS =
6161
endif
6262

63-
ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH))
64-
6563
# Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables
6664
# to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below.
6765

68-
build-%: check-go-version-go
66+
# BUILD_PLATFORMS contains a set of <os> <arch> <suffix> triplets,
67+
# separated by semicolon. An empty variable or empty entry (= just a
68+
# semicolon) builds for the default platform of the current Go
69+
# toolchain.
70+
BUILD_PLATFORMS =
71+
72+
# This builds each command (= the sub-directories of ./cmd) for the target platform(s)
73+
# defined by BUILD_PLATFORMS.
74+
$(CMDS:%=build-%): build-%: check-go-version-go
6975
mkdir -p bin
70-
CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
71-
if [ "$$ARCH" = "amd64" ]; then \
72-
CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \
73-
CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* ; \
74-
fi
76+
echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix; do \
77+
if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o "./bin/$*$$suffix" ./cmd/$*); then \
78+
echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \
79+
exit 1; \
80+
fi; \
81+
done
7582

76-
container-%: build-%
83+
$(CMDS:%=container-%): container-%: build-%
7784
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
7885

79-
push-%: container-%
86+
$(CMDS:%=push-%): push-%: container-%
8087
set -ex; \
8188
push_image () { \
8289
docker tag $*:latest $(IMAGE_NAME):$$tag; \
@@ -98,6 +105,77 @@ build: $(CMDS:%=build-%)
98105
container: $(CMDS:%=container-%)
99106
push: $(CMDS:%=push-%)
100107

108+
# Additional parameters are needed when pushing to a local registry,
109+
# see https://github.com/docker/buildx/issues/94.
110+
# However, that then runs into https://github.com/docker/cli/issues/2396.
111+
#
112+
# What works for local testing is:
113+
# make push-multiarch PULL_BASE_REF=master REGISTRY_NAME=<your account on dockerhub.io> BUILD_PLATFORMS="linux amd64; windows amd64 .exe; linux ppc64le -ppc64le; linux s390x -s390x"
114+
DOCKER_BUILDX_CREATE_ARGS ?=
115+
116+
# This target builds a multiarch image for one command using Moby BuildKit builder toolkit.
117+
# Docker Buildx is included in Docker 19.03.
118+
#
119+
# ./cmd/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
120+
# It is currently optional: if no such file exists, Windows images are not included,
121+
# even when Windows is listed in BUILD_PLATFORMS. That way, projects can test that
122+
# Windows binaries can be built before adding a Dockerfile for it.
123+
#
124+
# BUILD_PLATFORMS determines which individual images are included in the multiarch image.
125+
# PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name, and determines
126+
# the tag for the resulting multiarch image.
127+
$(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
128+
set -ex; \
129+
DOCKER_CLI_EXPERIMENTAL=enabled; \
130+
export DOCKER_CLI_EXPERIMENTAL; \
131+
docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \
132+
trap "docker buildx rm multiarchimage-buildertest" EXIT; \
133+
dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \
134+
dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
135+
if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \
136+
if ! [ -f "$$dockerfile_windows" ]; then \
137+
build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe//g' -e 's/; *;/;/g')"; \
138+
fi; \
139+
pushMultiArch () { \
140+
tag=$$1; \
141+
echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do \
142+
docker buildx build --push \
143+
--tag $(IMAGE_NAME):$$arch-$$os-$$tag \
144+
--platform=$$os/$$arch \
145+
--file $$(eval echo \$${dockerfile_$$os}) \
146+
--build-arg binary=./bin/$*$$suffix \
147+
--label revision=$(REV) \
148+
.; \
149+
done; \
150+
images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do echo $(IMAGE_NAME):$$arch-$$os-$$tag; done); \
151+
docker manifest create --amend $(IMAGE_NAME):$$tag $$images; \
152+
docker manifest push -p $(IMAGE_NAME):$$tag; \
153+
}; \
154+
if [ $(PULL_BASE_REF) = "master" ]; then \
155+
: "creating or overwriting canary image"; \
156+
pushMultiArch canary; \
157+
elif echo $(PULL_BASE_REF) | grep -q -e 'release-*' ; then \
158+
: "creating or overwriting canary image for release branch"; \
159+
release_canary_tag=$$(echo $(PULL_BASE_REF) | cut -f2 -d '-')-canary; \
160+
pushMultiArch $$release_canary_tag; \
161+
elif docker pull $(IMAGE_NAME):$(PULL_BASE_REF) 2>&1 | tee /dev/stderr | grep -q "manifest for $(IMAGE_NAME):$(PULL_BASE_REF) not found"; then \
162+
: "creating release image"; \
163+
pushMultiArch $(PULL_BASE_REF); \
164+
else \
165+
: "ERROR: release image $(IMAGE_NAME):$(PULL_BASE_REF) already exists: a new tag is required!"; \
166+
exit 1; \
167+
fi
168+
169+
.PHONY: check-pull-base-ref
170+
check-pull-base-ref:
171+
if ! [ "$(PULL_BASE_REF)" ]; then \
172+
echo >&2 "ERROR: PULL_BASE_REF must be set to 'master', 'release-x.y', or a tag name."; \
173+
exit 1; \
174+
fi
175+
176+
.PHONY: push-multiarch
177+
push-multiarch: $(CMDS:%=push-multiarch-%)
178+
101179
clean:
102180
-rm -rf bin
103181

‎release-tools/cloudbuild.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#! /bin/bash
2+
3+
# shellcheck disable=SC1091
4+
. release-tools/prow.sh
5+
6+
gcr_cloud_build

‎release-tools/cloudbuild.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# A configuration file for multi-arch image building with the Google cloud build service.
2+
#
3+
# Repos using this file must:
4+
# - import csi-release-tools
5+
# - add a symlink cloudbuild.yaml -> release-tools/cloudbuild.yaml
6+
# - add a .cloudbuild.sh which can be a custom file or a symlink
7+
# to release-tools/cloudbuild.sh
8+
# - accept "binary" as build argument in their Dockerfile(s) (see
9+
# https://github.com/pohly/node-driver-registrar/blob/3018101987b0bb6da2a2657de607174d6e3728f7/Dockerfile#L4-L6)
10+
# because binaries will get built for different architectures and then
11+
# get copied from the built host into the container image
12+
#
13+
# See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md
14+
# for more details on image pushing process in Kubernetes.
15+
#
16+
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage.
17+
18+
# This must be specified in seconds. If omitted, defaults to 600s (10 mins).
19+
timeout: 1800s
20+
# This prevents errors if you don't use both _GIT_TAG and _PULL_BASE_REF,
21+
# or any new substitutions added in the future.
22+
options:
23+
substitution_option: ALLOW_LOOSE
24+
steps:
25+
# The image must contain bash and curl. Ideally it should also contain
26+
# the desired version of Go (currently defined in release-tools/travis.yml),
27+
# but that just speeds up the build and is not required.
28+
- name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8'
29+
entrypoint: ./.cloudbuild.sh
30+
env:
31+
- GIT_TAG=${_GIT_TAG}
32+
- PULL_BASE_REF=${_PULL_BASE_REF}
33+
- REGISTRY_NAME=gcr.io/${_STAGING_PROJECT}
34+
- HOME=/root
35+
substitutions:
36+
# _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and
37+
# can be used as a substitution.
38+
_GIT_TAG: '12345'
39+
# _PULL_BASE_REF will contain the ref that was pushed to trigger this build -
40+
# a branch like 'master' or 'release-0.2', or a tag like 'v0.2'.
41+
_PULL_BASE_REF: 'master'
42+
# The default gcr.io staging project for Kubernetes-CSI
43+
# (=> https://console.cloud.google.com/gcr/images/k8s-staging-sig-storage/GLOBAL).
44+
# Might be overridden in the Prow build job for a repo which wants
45+
# images elsewhere.
46+
_STAGING_PROJECT: 'k8s-staging-sig-storage'

‎release-tools/filter-junit.go‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,10 @@ limitations under the License.
1515
*/
1616

1717
/*
18-
* This command filters a JUnit file such that only tests with a name
19-
* matching a regular expression are passed through. By concatenating
20-
* multiple input files it is possible to merge them into a single file.
21-
*/
18+
This command filters a JUnit file such that only tests with a name
19+
matching a regular expression are passed through. By concatenating
20+
multiple input files it is possible to merge them into a single file.
21+
*/
2222
package main
2323

2424
import (

0 commit comments

Comments
 (0)