diff --git a/.test-build/README.md b/.test-build/README.md new file mode 100644 index 000000000000..9e800ec44612 --- /dev/null +++ b/.test-build/README.md @@ -0,0 +1,94 @@ +## Testing ProwJobs Locally +- [test-infra](https://github.com/kubernetes/test-infra) provides tools that can be used to generate ProwJob specs and run them locally: + - [mkpj](https://docs.prow.k8s.io/docs/components/cli-tools/mkpj/) to generate the `ProwJob` yaml manifest + - [Phaino](https://docs.prow.k8s.io/docs/components/cli-tools/phaino/) to read in the yaml manifest, convert it to a `docker run` command and run the job locally + +## Example using `post-cluster-api-push-images` +- to run this example you will need: + - Docker installed locally: https://docs.docker.com/get-docker/ + - a local clone of [test-infra](https://github.com/kubernetes/test-infra) + - a GCP project with the following services: + - an [object storage](https://cloud.google.com/storage?hl=en) bucket to upload the source tarball for the cloudbuild job + - an [artifact registry](https://cloud.google.com/artifact-registry) as the destination for the built images and manifests + - [Cloudbuild](https://cloud.google.com/build?hl=en) enabled to run the build + - the [Gcloud CLI](https://cloud.google.com/sdk/gcloud) installed locally and authorized to perform the actions above +- in the Makefile, we will make the following modifications: + - update the `STAGING_REGISTRY` var to your artifact registry + - 3 lines exporting the required envvars for customizing the job: + ``` + export TEST_TAG = + export GCP_PROJECT = + export SCRATCH_BUCKET = + ``` +- run the Make target `test-post-cluster-api-push-images` to generate the required ProwJob yaml and Cloudbuild yaml + - this will substitute your exported variables in the template files and write them to the root of `cluster-api` project and + - the ProwJob yaml template originated from the following command in the `test-infra` repo: + ``` + go run ./prow/cmd/mkpj \ + --config-path=./config/prow/config.yaml \ + --job-config-path=./config/jobs/image-pushing/k8s-staging-cluster-api.yaml \ + --job=post-cluster-api-push-images \ + --base-ref=main \ + > test.yaml + ``` + - I've added the placeholder for the variable substitutions as well as a `--allow-dirty=true` option to the container args for faster testing/iteration (without this option the job will fail with a dirty git state) +- to kick off the job run the following command from the `test-infra` repo: + ``` + go run ./prow/cmd/phaino \ + --use-local-gcloud-credentials=true \ + /local-prow.yaml + ``` + - the container will use your local gcloud credentials by mounting your local gcloud config directory into the container + - `phaino` will try to find the `cluster-api` repo locally using this logic: https://github.com/kubernetes/test-infra/blob/7c313e4089802b5849ff973e6137efb9b86d0261/prow/cmd/phaino/local.go#L121-L124 + - you will see the location it finds in the `docker run` output on the command line (e.g. ` "-v" \ "/Users/stephen.cahill/cluster-api:/home/prow/go/src/github.com/kubernetes-sigs/cluster-api"`) + +## Proving out https://github.com/kubernetes-sigs/cluster-api/issues/9752#issuecomment-1873048812 +- run `touch` on the Dockerfiles to line up their timestamps (in our CD the repo is being cloned directly into the container so we need to mimic this) + ``` + touch /cluster-api/test/infrastructure/docker/Dockerfile & touch /cluster-api/test/extension/Dockerfile + ``` +- in this branch I've updated the Makefile's `ALL_DOCKER_BUILD` parameter so that only the two components we saw exhibit this behavior are built, which exposes the underlying bug without adding the `-j` option +- after the job completes, verify the images using the following command: ` ./verify.sh "" ""` + - example output: + ``` + ➜ ./verify.sh "dev4" "us-central1-docker.pkg.dev/datadog-sandbox/cahillsf" + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=amd64 + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=arm + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=arm64 + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=ppc64le + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=s390x + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=amd64 + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=amd64, expected value for path: "sigs.k8s.io/cluster-api/test/extension$" + path command-line-arguments + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm, expected value for path: "sigs.k8s.io/cluster-api/test/extension$" + path command-line-arguments + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm64 + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm64, expected value for path: "sigs.k8s.io/cluster-api/test/extension$" + path command-line-arguments + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=ppc64le + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=ppc64le, expected value for path: "sigs.k8s.io/cluster-api/test/extension$" + path command-line-arguments + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=s390x + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=s390x, expected value for path: "sigs.k8s.io/cluster-api/test/extension$" + path command-line-arguments + ``` +- if you switch the order of the `ALL_DOCKER_BUILD` option with `test-extenstion` first and run the build, you will see the failures flip to the CAPD image in the verify output: + ``` + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=amd64 + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=amd64, expected value for path: "command-line-arguments$" + path sigs.k8s.io/cluster-api/test/extension + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm, expected value for path: "command-line-arguments$" + path sigs.k8s.io/cluster-api/test/extension + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm64 + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm64, expected value for path: "command-line-arguments$" + path sigs.k8s.io/cluster-api/test/extension + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=ppc64le + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=ppc64le, expected value for path: "command-line-arguments$" + path sigs.k8s.io/cluster-api/test/extension + > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=s390x + FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=s390x, expected value for path: "command-line-arguments$" + path sigs.k8s.io/cluster-api/test/extension + ``` +- by parallelizing the build job with `-j 8`, the order in the Makefile was no longer respected as Make can run 8 jobs at once, thereby opening up the possibility for the CAPD manager and test-extension to be built in sequence by one of the jobs and exposing the behavior of the bug in the `BuildKit` version packaged with the Docker server version in use with CloudBuild \ No newline at end of file diff --git a/cloudbuild.yaml b/.test-build/cloudbuild.yaml similarity index 94% rename from cloudbuild.yaml rename to .test-build/cloudbuild.yaml index 66e3418801f4..99d540fc5955 100644 --- a/cloudbuild.yaml +++ b/.test-build/cloudbuild.yaml @@ -10,7 +10,7 @@ steps: env: - DOCKER_CLI_EXPERIMENTAL=enabled - TAG=$_GIT_TAG - - PULL_BASE_REF=$_PULL_BASE_REF + - PULL_BASE_REF=${TEST_TAG} - DOCKER_BUILDKIT=1 args: - release-staging diff --git a/.test-build/test.yaml b/.test-build/test.yaml new file mode 100644 index 000000000000..7a72b04a1316 --- /dev/null +++ b/.test-build/test.yaml @@ -0,0 +1,83 @@ +apiVersion: prow.k8s.io/v1 +kind: ProwJob +metadata: + annotations: + prow.k8s.io/context: post-cluster-api-push-images + prow.k8s.io/job: post-cluster-api-push-images + testgrid-alert-email: sig-cluster-lifecycle-cluster-api-alerts@kubernetes.io + testgrid-dashboards: sig-cluster-lifecycle-image-pushes, sig-k8s-infra-gcb + testgrid-num-failures-to-alert: "2" + testgrid-tab-name: post-cluster-api-push-images + creationTimestamp: null + labels: + created-by-prow: "true" + prow.k8s.io/context: post-cluster-api-push-images + prow.k8s.io/job: post-cluster-api-push-images + prow.k8s.io/refs.base_ref: main + prow.k8s.io/refs.org: kubernetes-sigs + prow.k8s.io/refs.repo: cluster-api + prow.k8s.io/type: postsubmit + name: 98d67ccc-95d0-4f3b-a778-757938065bfa +spec: + agent: kubernetes + cluster: k8s-infra-prow-build-trusted + context: post-cluster-api-push-images + decoration_config: + gcs_configuration: + bucket: kubernetes-jenkins + default_org: kubernetes + default_repo: kubernetes + path_strategy: legacy + gcs_credentials_secret: service-account + grace_period: 15m0s + resources: + clonerefs: + requests: + cpu: 100m + initupload: + requests: + cpu: 100m + place_entrypoint: + requests: + cpu: 100m + sidecar: + requests: + cpu: 100m + timeout: 2h0m0s + utility_images: + clonerefs: gcr.io/k8s-prow/clonerefs:v20231213-380b89a158 + entrypoint: gcr.io/k8s-prow/entrypoint:v20231213-380b89a158 + initupload: gcr.io/k8s-prow/initupload:v20231213-380b89a158 + sidecar: gcr.io/k8s-prow/sidecar:v20231213-380b89a158 + job: post-cluster-api-push-images + namespace: test-pods + pod_spec: + containers: + - args: + - --project=${GCP_PROJECT} + - --scratch-bucket=gs://${SCRATCH_BUCKET} + - --env-passthrough=PULL_BASE_REF + - --with-git-dir + - --allow-dirty=true + - . + command: + - /run.sh + env: + - name: GOPROXY + value: https://proxy.golang.org + image: gcr.io/k8s-staging-test-infra/image-builder:v20230711-e33377c2b4 + name: "" + resources: {} + serviceAccountName: gcb-builder + prowjob_defaults: + tenant_id: GlobalDefaultID + refs: + base_ref: main + base_sha: 6cbf80408b836b5f1caa1887cfb23a298772d546 + org: kubernetes-sigs + repo: cluster-api + report: true + type: postsubmit +status: + startTime: "2023-12-31T02:40:38Z" + state: triggered diff --git a/Makefile b/Makefile index 4ee4973ee287..dda2f6786887 100644 --- a/Makefile +++ b/Makefile @@ -45,6 +45,9 @@ export KUBEBUILDER_CONTROLPLANE_STOP_TIMEOUT ?= 60s # This option is for running docker manifest command export DOCKER_CLI_EXPERIMENTAL := enabled +export TEST_TAG = test +export GCP_PROJECT = datadog-sandbox +export SCRATCH_BUCKET = cahillsf-testing-cli # Enables shell script tracing. Enable by running: TRACE=1 make TRACE ?= 0 @@ -200,7 +203,10 @@ TILT_PREPARE := $(abspath $(TOOLS_BIN_DIR)/$(TILT_PREPARE_BIN)) REGISTRY ?= gcr.io/$(shell gcloud config get-value project) PROD_REGISTRY ?= registry.k8s.io/cluster-api -STAGING_REGISTRY ?= gcr.io/k8s-staging-cluster-api +# STAGING_REGISTRY ?= gcr.io/k8s-staging-cluster-api +STAGING_REGISTRY ?= us-central1-docker.pkg.dev/datadog-sandbox/cahillsf + + STAGING_BUCKET ?= artifacts.k8s-staging-cluster-api.appspot.com # core @@ -767,7 +773,8 @@ docker-build-%: $(MAKE) ARCH=$* docker-build # Choice of images to build/push -ALL_DOCKER_BUILD ?= core kubeadm-bootstrap kubeadm-control-plane docker-infrastructure in-memory-infrastructure test-extension clusterctl +# ALL_DOCKER_BUILD ?= core kubeadm-bootstrap kubeadm-control-plane docker-infrastructure in-memory-infrastructure test-extension clusterctl +ALL_DOCKER_BUILD ?= docker-infrastructure test-extension .PHONY: docker-build docker-build: docker-pull-prerequisites ## Run docker-build-* targets for all the images @@ -928,6 +935,11 @@ tilt-up: kind-cluster ## Start tilt and build kind cluster if needed. serve-book: ## Build and serve the book (with live-reload) $(MAKE) -C docs/book serve +.PHONY: test-post-cluster-api-push-images +test-post-cluster-api-push-images: $(ENVSUBST_BIN) + cat $(ROOT_DIR)/.test-build/test.yaml | $(ENVSUBST_BIN) > $(ROOT_DIR)/local-prow.yaml + cat $(ROOT_DIR)/.test-build/cloudbuild.yaml | $(ENVSUBST_BIN) > $(ROOT_DIR)/cloudbuild.yaml + ## -------------------------------------- ## Release ## -------------------------------------- @@ -1088,14 +1100,19 @@ release-staging-nightly: ## Tag and push container images to the staging bucket. .PHONY: release-alias-tag release-alias-tag: ## Add the release alias tag to the last build tag - gcloud container images add-tag $(CONTROLLER_IMG):$(TAG) $(CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) - gcloud container images add-tag $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(TAG) $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) - gcloud container images add-tag $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(TAG) $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) - gcloud container images add-tag $(CLUSTERCTL_IMG):$(TAG) $(CLUSTERCTL_IMG):$(RELEASE_ALIAS_TAG) gcloud container images add-tag $(CAPD_CONTROLLER_IMG):$(TAG) $(CAPD_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) - gcloud container images add-tag $(CAPIM_CONTROLLER_IMG):$(TAG) $(CAPIM_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) gcloud container images add-tag $(TEST_EXTENSION_IMG):$(TAG) $(TEST_EXTENSION_IMG):$(RELEASE_ALIAS_TAG) +# .PHONY: release-alias-tag +# release-alias-tag: ## Add the release alias tag to the last build tag +# gcloud container images add-tag $(CONTROLLER_IMG):$(TAG) $(CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(TAG) $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(TAG) $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(CLUSTERCTL_IMG):$(TAG) $(CLUSTERCTL_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(CAPD_CONTROLLER_IMG):$(TAG) $(CAPD_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(CAPIM_CONTROLLER_IMG):$(TAG) $(CAPIM_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG) +# gcloud container images add-tag $(TEST_EXTENSION_IMG):$(TAG) $(TEST_EXTENSION_IMG):$(RELEASE_ALIAS_TAG) + .PHONY: release-notes-tool release-notes-tool: go build -o bin/notes -tags tools sigs.k8s.io/cluster-api/hack/tools/release/notes diff --git a/verify.sh b/verify.sh new file mode 100755 index 000000000000..50b89b965744 --- /dev/null +++ b/verify.sh @@ -0,0 +1,24 @@ +function TESTIMAGE() { + IMAGE="${1}" + GREPVALUE="${2}" + BINARYPATH="${3:-manager}" + + MANIFEST="$(crane manifest "${IMAGE}")" + for ARCHITECTURE in $(echo "${MANIFEST}" | jq -r '.manifests[].platform.architecture'); do + echo "> Testing ${IMAGE} ARCH=$ARCHITECTURE" + rm -rf extracted || true + digest=$(echo "${MANIFEST}" | jq -r '.manifests[] | select(.platform.architecture == "'${ARCHITECTURE}'") | .digest') + imgpkg pull -i ${IMAGE}@${digest} -o extracted > /dev/null + go version -m extracted/${BINARYPATH} | grep "\tpath" | grep -q ${GREPVALUE} || (echo "FAILED ${IMAGE} ARCH=${ARCHITECTURE}, expected value for path: \"${GREPVALUE}\""; go version -m extracted/${BINARYPATH} | grep "\tpath") + done +} + + +# function TESTIMAGES() { + TAG="${1}" + REGISTRY="${2}" + TESTIMAGE "${REGISTRY}/capd-manager:${TAG}" "command-line-arguments$" + TESTIMAGE "${REGISTRY}/test-extension:${TAG}" "sigs.k8s.io/cluster-api/test/extension$" +# } + +# TESTIMAGES "dev4" "us-central1-docker.pkg.dev/datadog-sandbox/cahillsf \ No newline at end of file