Refactor end-to-end tests

[stankevich]

The end-to-end tests have been pretty flaky lately, which makes it harder to trust them and slows down pull requests. Let's try to get this fixed! Here, I focused on the initial Kind cluster setup, port forwarding, teardown, and some of the most temperamental parts of the test suite. Quick summary of the changes:

• Added a cache mount for Go module and build caches in the Docker image. On my machine, this frees up disk space and cuts image build time from 40+ seconds to 1–5 seconds, depending on the code changes.
• Moved the Kind cluster setup steps into a function, DRYing up the code across all tests.
• Did the same for the teardown logic.
• Replaced hostPath volumes with persistentVolumeClaim volumes. That simplified the code, cluster setup, file permissions, and removed occasional chmod-ing of backup directories.
• Made port forwarding more robust by waiting for expected ports to become ready to serve requests. That helped remove a few sleeps from the tests.
• Reduced VTGate's tablet_refresh_interval from 60 to 10 seconds (same as in vttestserver), which speeds up the initial Vitess cluster setup and upgrades.
• Added a post-upgrade cluster spec verification step. Combined with the previous point, that stabilized the most frequently failing upgrade test.
• Moved commerce keyspace verification commands into a function, DRYing up a decent chunk of the code.
• Removed the local binary build as it isn't used in the tests.
• Decreased the timeouts of Buildkite jobs and added retries on unexpected Buildkite Agent failures.
• Added support for running the tests in environments where multiple Buildkite Agents share a single Docker service.
• Reduced the output when installing or downloading the tools. Fixed the output when building the Docker image. Reduced MySQL client warnings.
• Other minor tweaks that help with test stability and general cleanup.

I ran some "before" and "after" comparisons. On my machine, the average time for 7 test runs on the main branch was 36 minutes 1 second, with a standard deviation of 1 minute 4 seconds. After these changes, the average dropped to 27 minutes 57 seconds, with a standard deviation of 1 minute 1 second. More importantly though, I haven't seen any of the flakiness that was happening before. Will see what CI says and make further improvements if needed.

The diff of this pull request is pretty big, and it's much easier to review commit-by-commit. I decided not to split it into multiple pull requests because I was worried that flaky failures in intermediate ones would stall all of them. 😅

Related to #582.

[stankevich]

Of course all of them failed! 😁 Is there any way for me to see the reason? Pasting the relevant errors here would be helpful. Thanks!

[GuptaManan100]

This is the error message I see in the buildkite output -

[2025-05-22T09:36:40Z]  => => naming to docker.io/library/vitess-operator-pr:latest                                                                                               0.0s
[2025-05-22T09:36:40Z] Setting up the Kind config
[2025-05-22T09:36:40Z] ./test/endtoend/utils.sh: line 512: vtdataroot/config.yaml: No such file or directory
[2025-05-22T09:36:40Z] Creating Kind cluster
[2025-05-22T09:36:40Z] ERROR: failed to create cluster: error reading file: open ./vtdataroot/config.yaml: no such file or directory
[2025-05-22T09:36:40Z] Loading docker image into Kind cluster
[2025-05-22T09:36:40Z] ERROR: no nodes found for cluster "kind-0196f755-c056-4ed2-a1fc-d65fa5775d3d"
[2025-05-22T09:36:40Z] Error response from daemon: network kind not found
[2025-05-22T09:36:40Z] ./test/endtoend/utils.sh: line 449: /root/.kube/config: No such file or directory
[2025-05-22T09:36:40Z] Creating the example namespace
[2025-05-22T09:36:40Z] The connection to the server localhost:8080 was refused - did you specify the right host or port?
[2025-05-22T09:36:40Z] Apply latest operator-latest.yaml
[2025-05-22T09:36:40Z] error: error validating "operator-latest.yaml": error validating data: failed to download openapi: Get "http://localhost:8080/openapi/v2?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
[2025-05-22T09:36:40Z] E0522 09:36:40.566215   14030 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused"
[2025-05-22T09:36:40Z] E0522 09:36:40.567677   14030 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused"
[2025-05-22T09:36:40Z] E0522 09:36:40.569088   14030 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused"
[2025-05-22T09:36:40Z] E0522 09:36:40.570509   14030 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused"
[2025-05-22T09:36:40Z] E0522 09:36:40.571900   14030 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused"
[2025-05-22T09:36:40Z] The connection to the server localhost:8080 was refused - did you specify the right host or port?

[stankevich]

I made a few more tweaks to stabilize the tests, handle the failures better, and clean up the output. The original pull request description has been updated. Two notable changes since opening this pull request:

• The hostPath volumes (backup storage) have been replaced with persistentVolumeClaim volumes. This is a more Kubernetes-native approach and is closer to real-world usage outside of using object storage. It significantly simplifies the configuration, cluster setup, permissions, and cleanup.
• Decreased the timeouts of Buildkite jobs and added retries for unexpected Buildkite Agent failures. This catches sporadic failures caused by external factors, such as spot instance interruptions. It also handles occasional test timeouts, although they are now quite rare. Still, it would be nice to eliminate them completely and not rely on retries.

At this point, the tests are passing fairly reliably. The only one that occasionally gets stuck is the Backup Schedule Test (example). I haven't been able to get to the bottom of it yet. It times out because one out of three tablets fails to become ready after the initial deployment. This only seems to happen in this specific test and only on the public Buildkite Elastic CI Stack in this repository. I haven't been able to reproduce it once in over 30 local test runs or over 50 runs in our custom Buildkite Agent setup, which runs in Kubernetes and uses the Buildkite pipeline from this repository. In any case, I'll keep an eye on the tests and address any further failures that come up.

This is ready to review. I'm happy to split it into multiple pull requests if that would help with the review process. Thanks!

[GuptaManan100]

Looks good to me!

[frouioui]

This is really good. Thank you for doing this

[frouioui]

Why was this change necessary?

[stankevich]

I've added some background to the commit message. Quick summary: before this change, every image build, even after a small and unrelated change (like a README update), would redownload all dependencies and recompile the binary from scratch. This was caused by the COPY . step invalidating the cached Docker layer and running a fresh go install.

This change enables two cache mounts: one for Go modules to avoid redownloading dependencies, and one for build caches to reuse previously built artifacts. That allows go install to reuse outputs from previous builds and cuts down the overall image build time, even if COPY . invalidates the cached layer.

Locally, the speedup is significant, going from 40+ seconds to only a few seconds. In CI, it depends on the setup, but it's never slower than before and can be much faster if the image build runs on the same Buildkite Agent (backed by a Docker service with the cache mounts).

[frouioui]

That sounds great to me!