cahillsf · cahillsf · Jan 9, 2024
diff --git a/.test-build/README.md b/.test-build/README.md
@@ -0,0 +1,94 @@
+## Testing ProwJobs Locally
+- [test-infra](https://github.com/kubernetes/test-infra) provides tools that can be used to generate ProwJob specs and run them locally: 
+  - [mkpj](https://docs.prow.k8s.io/docs/components/cli-tools/mkpj/) to generate the `ProwJob` yaml manifest
+  - [Phaino](https://docs.prow.k8s.io/docs/components/cli-tools/phaino/) to read in the yaml manifest, convert it to a `docker run` command and run the job locally
+
+## Example using `post-cluster-api-push-images`
+- to run this example you will need:
+  - Docker installed locally: https://docs.docker.com/get-docker/
+  - a local clone of [test-infra](https://github.com/kubernetes/test-infra) 
+  - a GCP project with the following services:
+    - an [object storage](https://cloud.google.com/storage?hl=en) bucket to upload the source tarball for the cloudbuild job
+    - an [artifact registry](https://cloud.google.com/artifact-registry) as the destination for the built images and manifests
+    - [Cloudbuild](https://cloud.google.com/build?hl=en) enabled to run the build
+    - the [Gcloud CLI](https://cloud.google.com/sdk/gcloud) installed locally and authorized to perform the actions above
+- in the Makefile, we will make the following modifications:
+  - update the `STAGING_REGISTRY` var to your artifact registry
+  - 3 lines exporting the required envvars for customizing the job: 
+    ```
+    export TEST_TAG = <DESIRED_IMAGE_TAG>
+    export GCP_PROJECT = <SANDBOX_GCP_PROJECT>
+    export SCRATCH_BUCKET = <SANDBOX_OBJECT_STORAGE_BUCKET_NAME>
+    ```
+- run the Make target `test-post-cluster-api-push-images` to generate the required ProwJob yaml and Cloudbuild yaml
+  - this will substitute your exported variables in the template files and write them to the root of `cluster-api` project and 
+  - the ProwJob yaml template originated from the following command in the `test-infra` repo:
+    ```
+    go run ./prow/cmd/mkpj \
+      --config-path=./config/prow/config.yaml \
+      --job-config-path=./config/jobs/image-pushing/k8s-staging-cluster-api.yaml \
+      --job=post-cluster-api-push-images \
+      --base-ref=main \
+      > test.yaml
+    ```
+  - I've added the placeholder for the variable substitutions as well as a `--allow-dirty=true` option to the container args for faster testing/iteration (without this option the job will fail with a dirty git state)
+- to kick off the job run the following command from the `test-infra` repo:
+  ```
+  go run ./prow/cmd/phaino \
+    --use-local-gcloud-credentials=true \
+    <CLUSTER_API_REPO>/local-prow.yaml
+  ```
+  - the container will use your local gcloud credentials by mounting your local gcloud config directory into the container
+  - `phaino` will try to find the `cluster-api` repo locally using this logic: https://github.com/kubernetes/test-infra/blob/7c313e4089802b5849ff973e6137efb9b86d0261/prow/cmd/phaino/local.go#L121-L124
+    - you will see the location it finds in the `docker run` output on the command line (e.g. ` "-v" \ "/Users/stephen.cahill/cluster-api:/home/prow/go/src/github.com/kubernetes-sigs/cluster-api"`)
+
+## Proving out https://github.com/kubernetes-sigs/cluster-api/issues/9752#issuecomment-1873048812
+- run `touch` on the Dockerfiles to line up their timestamps (in our CD the repo is being cloned directly into the container so we need to mimic this)
+  ```
+  touch <RELEVANT_LOCAL_PATH>/cluster-api/test/infrastructure/docker/Dockerfile & touch <RELEVANT_LOCAL_PATH>/cluster-api/test/extension/Dockerfile
+  ```
+- in this branch I've updated the Makefile's `ALL_DOCKER_BUILD` parameter so that only the two components we saw exhibit this behavior are built, which exposes the underlying bug without adding the `-j` option
+- after the job completes, verify the images using the following command: `  ./verify.sh "<TEST_TAG>" "<STAGING_REGISTRY>"`
+  - example output:
+    ```
+    ➜ ./verify.sh "dev4" "us-central1-docker.pkg.dev/datadog-sandbox/cahillsf"
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=amd64
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=arm
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=arm64
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=ppc64le
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:dev4 ARCH=s390x
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=amd64
+    FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=amd64, expected value for path: "sigs.k8s.io/cluster-api/test/extension$"
+            path    command-line-arguments
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm
+    FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm, expected value for path: "sigs.k8s.io/cluster-api/test/extension$"
+            path    command-line-arguments
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm64
+    FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=arm64, expected value for path: "sigs.k8s.io/cluster-api/test/extension$"
+            path    command-line-arguments
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=ppc64le
+    FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=ppc64le, expected value for path: "sigs.k8s.io/cluster-api/test/extension$"
+            path    command-line-arguments
+    > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=s390x
+    FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/test-extension:dev4 ARCH=s390x, expected value for path: "sigs.k8s.io/cluster-api/test/extension$"
+            path    command-line-arguments
+    ```
+- if you switch the order of the `ALL_DOCKER_BUILD` option with `test-extenstion` first and run the build, you will see the failures flip to the CAPD image in the verify output:
+  ```
+  > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=amd64
+  FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=amd64, expected value for path: "command-line-arguments$"
+        path    sigs.k8s.io/cluster-api/test/extension
+  > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm
+  FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm, expected value for path: "command-line-arguments$"
+          path    sigs.k8s.io/cluster-api/test/extension
+  > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm64
+  FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=arm64, expected value for path: "command-line-arguments$"
+          path    sigs.k8s.io/cluster-api/test/extension
+  > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=ppc64le
+  FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=ppc64le, expected value for path: "command-line-arguments$"
+          path    sigs.k8s.io/cluster-api/test/extension
+  > Testing us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=s390x
+  FAILED us-central1-docker.pkg.dev/datadog-sandbox/cahillsf/capd-manager:test5 ARCH=s390x, expected value for path: "command-line-arguments$"
+          path    sigs.k8s.io/cluster-api/test/extension
+  ```
+- by parallelizing the build job with `-j 8`, the order in the Makefile was no longer respected as Make can run 8 jobs at once, thereby opening up the possibility for the CAPD manager and test-extension to be built in sequence by one of the jobs and exposing the behavior of the bug in the `BuildKit` version packaged with the Docker server version in use with CloudBuild
diff --git a/cloudbuild.yaml → .test-build/cloudbuild.yaml b/cloudbuild.yaml → .test-build/cloudbuild.yaml
@@ -10,7 +10,7 @@ steps:
     env:
     - DOCKER_CLI_EXPERIMENTAL=enabled
     - TAG=$_GIT_TAG
-    - PULL_BASE_REF=$_PULL_BASE_REF
+    - PULL_BASE_REF=${TEST_TAG}
     - DOCKER_BUILDKIT=1
     args:
     - release-staging

diff --git a/.test-build/test.yaml b/.test-build/test.yaml
@@ -0,0 +1,83 @@
+apiVersion: prow.k8s.io/v1
+kind: ProwJob
+metadata:
+  annotations:
+    prow.k8s.io/context: post-cluster-api-push-images
+    prow.k8s.io/job: post-cluster-api-push-images
+    testgrid-alert-email: [email protected]
+    testgrid-dashboards: sig-cluster-lifecycle-image-pushes, sig-k8s-infra-gcb
+    testgrid-num-failures-to-alert: "2"
+    testgrid-tab-name: post-cluster-api-push-images
+  creationTimestamp: null
+  labels:
+    created-by-prow: "true"
+    prow.k8s.io/context: post-cluster-api-push-images
+    prow.k8s.io/job: post-cluster-api-push-images
+    prow.k8s.io/refs.base_ref: main
+    prow.k8s.io/refs.org: kubernetes-sigs
+    prow.k8s.io/refs.repo: cluster-api
+    prow.k8s.io/type: postsubmit
+  name: 98d67ccc-95d0-4f3b-a778-757938065bfa
+spec:
+  agent: kubernetes
+  cluster: k8s-infra-prow-build-trusted
+  context: post-cluster-api-push-images
+  decoration_config:
+    gcs_configuration:
+      bucket: kubernetes-jenkins
+      default_org: kubernetes
+      default_repo: kubernetes
+      path_strategy: legacy
+    gcs_credentials_secret: service-account
+    grace_period: 15m0s
+    resources:
+      clonerefs:
+        requests:
+          cpu: 100m
+      initupload:
+        requests:
+          cpu: 100m
+      place_entrypoint:
+        requests:
+          cpu: 100m
+      sidecar:
+        requests:
+          cpu: 100m
+    timeout: 2h0m0s
+    utility_images:
+      clonerefs: gcr.io/k8s-prow/clonerefs:v20231213-380b89a158
+      entrypoint: gcr.io/k8s-prow/entrypoint:v20231213-380b89a158
+      initupload: gcr.io/k8s-prow/initupload:v20231213-380b89a158
+      sidecar: gcr.io/k8s-prow/sidecar:v20231213-380b89a158
+  job: post-cluster-api-push-images
+  namespace: test-pods
+  pod_spec:
+    containers:
+    - args:
+      - --project=${GCP_PROJECT}
+      - --scratch-bucket=gs://${SCRATCH_BUCKET}
+      - --env-passthrough=PULL_BASE_REF
+      - --with-git-dir
+      - --allow-dirty=true
+      - .
+      command:
+      - /run.sh
+      env:
+      - name: GOPROXY
+        value: https://proxy.golang.org
+      image: gcr.io/k8s-staging-test-infra/image-builder:v20230711-e33377c2b4
+      name: ""
+      resources: {}
+    serviceAccountName: gcb-builder
+  prowjob_defaults:
+    tenant_id: GlobalDefaultID
+  refs:
+    base_ref: main
+    base_sha: 6cbf80408b836b5f1caa1887cfb23a298772d546
+    org: kubernetes-sigs
+    repo: cluster-api
+  report: true
+  type: postsubmit
+status:
+  startTime: "2023-12-31T02:40:38Z"
+  state: triggered
diff --git a/Makefile b/Makefile
@@ -45,6 +45,9 @@ export KUBEBUILDER_CONTROLPLANE_STOP_TIMEOUT ?= 60s
 
 # This option is for running docker manifest command
 export DOCKER_CLI_EXPERIMENTAL := enabled
+export TEST_TAG = test
+export GCP_PROJECT = datadog-sandbox
+export SCRATCH_BUCKET = cahillsf-testing-cli
 
 # Enables shell script tracing. Enable by running: TRACE=1 make <target>
 TRACE ?= 0
@@ -200,7 +203,10 @@ TILT_PREPARE := $(abspath $(TOOLS_BIN_DIR)/$(TILT_PREPARE_BIN))
 REGISTRY ?= gcr.io/$(shell gcloud config get-value project)
 PROD_REGISTRY ?= registry.k8s.io/cluster-api
 
-STAGING_REGISTRY ?= gcr.io/k8s-staging-cluster-api
+# STAGING_REGISTRY ?= gcr.io/k8s-staging-cluster-api
+STAGING_REGISTRY ?= us-central1-docker.pkg.dev/datadog-sandbox/cahillsf
+
+
 STAGING_BUCKET ?= artifacts.k8s-staging-cluster-api.appspot.com
 
 # core
@@ -767,7 +773,8 @@ docker-build-%:
 	$(MAKE) ARCH=$* docker-build
 
 # Choice of images to build/push
-ALL_DOCKER_BUILD ?= core kubeadm-bootstrap kubeadm-control-plane docker-infrastructure in-memory-infrastructure test-extension clusterctl
+# ALL_DOCKER_BUILD ?= core kubeadm-bootstrap kubeadm-control-plane docker-infrastructure in-memory-infrastructure test-extension clusterctl
+ALL_DOCKER_BUILD ?= docker-infrastructure test-extension
 
 .PHONY: docker-build
 docker-build: docker-pull-prerequisites ## Run docker-build-* targets for all the images
@@ -928,6 +935,11 @@ tilt-up: kind-cluster ## Start tilt and build kind cluster if needed.
 serve-book: ## Build and serve the book (with live-reload)
 	$(MAKE) -C docs/book serve
 
+.PHONY: test-post-cluster-api-push-images
+test-post-cluster-api-push-images: $(ENVSUBST_BIN)
+	cat $(ROOT_DIR)/.test-build/test.yaml | $(ENVSUBST_BIN) > $(ROOT_DIR)/local-prow.yaml
+	cat $(ROOT_DIR)/.test-build/cloudbuild.yaml | $(ENVSUBST_BIN) > $(ROOT_DIR)/cloudbuild.yaml
+
 ## --------------------------------------
 ## Release
 ## --------------------------------------
@@ -1088,14 +1100,19 @@ release-staging-nightly: ## Tag and push container images to the staging bucket.
 
 .PHONY: release-alias-tag
 release-alias-tag: ## Add the release alias tag to the last build tag
-	gcloud container images add-tag $(CONTROLLER_IMG):$(TAG) $(CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
-	gcloud container images add-tag $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(TAG) $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
-	gcloud container images add-tag $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(TAG) $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
-	gcloud container images add-tag $(CLUSTERCTL_IMG):$(TAG) $(CLUSTERCTL_IMG):$(RELEASE_ALIAS_TAG)
 	gcloud container images add-tag $(CAPD_CONTROLLER_IMG):$(TAG) $(CAPD_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
-	gcloud container images add-tag $(CAPIM_CONTROLLER_IMG):$(TAG) $(CAPIM_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
 	gcloud container images add-tag $(TEST_EXTENSION_IMG):$(TAG) $(TEST_EXTENSION_IMG):$(RELEASE_ALIAS_TAG)
 
+# .PHONY: release-alias-tag
+# release-alias-tag: ## Add the release alias tag to the last build tag
+# 	gcloud container images add-tag $(CONTROLLER_IMG):$(TAG) $(CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(TAG) $(KUBEADM_BOOTSTRAP_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(TAG) $(KUBEADM_CONTROL_PLANE_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(CLUSTERCTL_IMG):$(TAG) $(CLUSTERCTL_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(CAPD_CONTROLLER_IMG):$(TAG) $(CAPD_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(CAPIM_CONTROLLER_IMG):$(TAG) $(CAPIM_CONTROLLER_IMG):$(RELEASE_ALIAS_TAG)
+# 	gcloud container images add-tag $(TEST_EXTENSION_IMG):$(TAG) $(TEST_EXTENSION_IMG):$(RELEASE_ALIAS_TAG)
+
 .PHONY: release-notes-tool
 release-notes-tool:
 	go build -o bin/notes -tags tools sigs.k8s.io/cluster-api/hack/tools/release/notes

diff --git a/verify.sh b/verify.sh
@@ -0,0 +1,24 @@
+function TESTIMAGE() {
+  IMAGE="${1}"
+  GREPVALUE="${2}"
+  BINARYPATH="${3:-manager}"
+
+  MANIFEST="$(crane manifest "${IMAGE}")"
+  for ARCHITECTURE in $(echo "${MANIFEST}" | jq -r '.manifests[].platform.architecture'); do
+    echo "> Testing ${IMAGE} ARCH=$ARCHITECTURE"
+    rm -rf extracted || true
+    digest=$(echo "${MANIFEST}" | jq -r '.manifests[] | select(.platform.architecture == "'${ARCHITECTURE}'") | .digest')
+    imgpkg pull -i ${IMAGE}@${digest} -o extracted > /dev/null
+    go version -m extracted/${BINARYPATH} | grep "\tpath" | grep -q ${GREPVALUE} || (echo "FAILED ${IMAGE} ARCH=${ARCHITECTURE}, expected value for path: \"${GREPVALUE}\""; go version -m extracted/${BINARYPATH} | grep "\tpath")
+  done
+}
+
+
+# function TESTIMAGES() {
+  TAG="${1}"
+  REGISTRY="${2}"
+  TESTIMAGE "${REGISTRY}/capd-manager:${TAG}" "command-line-arguments$"
+  TESTIMAGE "${REGISTRY}/test-extension:${TAG}" "sigs.k8s.io/cluster-api/test/extension$"
+# }
+
+# TESTIMAGES "dev4" "us-central1-docker.pkg.dev/datadog-sandbox/cahillsf