|
| 1 | +# ADR 0014: Conformance Test Suite Design |
| 2 | + |
| 3 | +* **Status**: proposed |
| 4 | +* **Deciders**: Maintainers |
| 5 | +* **Date**: 2026-02-04 |
| 6 | + |
| 7 | +**Technical Story**: We need a robust conformance test suite to ensure our tooling (CLI and Kubernetes controllers) behaves as expected and adheres to a standard specification. This suite should be inspired by the Kubernetes conformance test suite but adapted to be more lightweight. |
| 8 | + |
| 9 | +## Context and Problem Statement |
| 10 | + |
| 11 | +We are a platform engineering team developing a CLI and a Kubernetes controller. We need to ensure that released versions of our tooling are compatible with each other and external systems. |
| 12 | + |
| 13 | +Currently, our `kubernetes/controller` component has Ginkgo-based end-to-end (e2e) tests. We need to evolve this into a proper conformance suite with the following requirements: |
| 14 | + |
| 15 | +1. **Conformance Definition**: Clearly define which tests constitute our "conformance profile". |
| 16 | +2. **Versioning & Promotion**: Version conformance tests and provide a workflow to promote e2e tests to conformance. |
| 17 | +3. **Tooling Complexity**: Avoid the complexity of Kubernetes' elaborate Ginkgo setup. |
| 18 | +4. **Environment Independence**: Tests must run against configurable environments (various OCI registries, Kubernetes clusters). |
| 19 | + |
| 20 | +## Decision Drivers |
| 21 | + |
| 22 | +* **Simplicity**: Preference for standard Go tooling. |
| 23 | +* **Maintainability**: Easy to write, read, and debug. |
| 24 | +* **Discoverability**: Clear identification of conformance tests. |
| 25 | +* **Automation**: Efficient filtering and execution in CI/CD. |
| 26 | + |
| 27 | +## Considered Options |
| 28 | + |
| 29 | +* **Option 1**: **Ginkgo/Gomega with Labels** (Kubernetes style) |
| 30 | + * *Description*: Use Ginkgo's Label feature to mark tests as `[Conformance]`. |
| 31 | +* **Option 2**: **Standard Go Testing + Testify** |
| 32 | + * *Description*: Use standard Go testing with a custom functional labeling mechanism (e.g., `if !HasLabel("conformance") { t.Skip() }`). |
| 33 | + |
| 34 | +## Decision Outcome |
| 35 | + |
| 36 | +Chosen **Option 2**: "Standard Go Testing + Testify". |
| 37 | + |
| 38 | +**Justification**: |
| 39 | +* **Lightweight**: Avoids Ginkgo's DSL overhead and implicit behaviors. |
| 40 | +* **Native Tooling**: Leveraging `go test` is standard and well-understood. |
| 41 | +* **Flexibility**: Functional labeling provides explicit control without framework magic. |
| 42 | +* **Separation**: A dedicated `e2e` module avoids circular dependencies. |
| 43 | + |
| 44 | +### Implementation Details |
| 45 | + |
| 46 | +We will introduce a new top-level `e2e` Go module containing all end-to-end tests, a subset of which are conformance tests. |
| 47 | + |
| 48 | +#### Key Design Decisions |
| 49 | + |
| 50 | +1. **Framework**: `stretchr/testify` for assertions and standard `testing.T`. |
| 51 | +2. **Identification**: |
| 52 | + * Tests must initialize a `TestMeta` struct containing labels. |
| 53 | + * Conformance tests are identified by the label `test-kind: conformance`. |
| 54 | +3. **Environment Configuration**: |
| 55 | + * Tests receive environment configuration via a command-line argument: `./e2e.test -- --config=<config.yaml>`. |
| 56 | + * This config defines the OCI registry, Kubernetes cluster, etc. |
| 57 | +4. **Promotion**: |
| 58 | + * Promote a test by adding `test-kind: conformance` to its `TestMeta` labels via Pull Request. |
| 59 | +5. **Versioning**: |
| 60 | + * Conformance tests are versioned with the codebase. |
| 61 | +6. **Reference Scenarios**: |
| 62 | + * Core conformance tests will implement the Reference Scenarios (ADR 0013). |
| 63 | + * These will largely be executed as Taskfiles invoked by the Go tests. |
| 64 | + |
| 65 | +#### Technical Details |
| 66 | + |
| 67 | +* **Test Containers**: Use [Testcontainers](https://golang.testcontainers.org/) for isolated dependencies (OCM binary, OCI registries). |
| 68 | +* **Providers**: Implement provider interfaces for external dependencies (OCI registries, Clusters) to ensure tests are agnostic to the specific backing service. |
| 69 | + |
| 70 | +## Pros and Cons of the Options |
| 71 | + |
| 72 | +### Option 1: Ginkgo with Labels |
| 73 | + |
| 74 | +* **Pros**: Rich BDD style, built-in labeling, powerful tooling. |
| 75 | +* **Cons**: Steep learning curve, implicit behavior, complex bootstrapping. |
| 76 | + |
| 77 | +### Option 2: Standard Go + Testify (Chosen) |
| 78 | + |
| 79 | +* **Pros**: Standard patterns, explicit filtering, simple debugging, preferred by the team. |
| 80 | +* **Cons**: Requires manual convention enforcement (mitigated by linters). |
| 81 | + |
| 82 | +## Next Steps |
| 83 | + |
| 84 | +* Create the `e2e` folder and migrate initial tests. |
| 85 | +* Update CI pipelines to execute the suite. |
| 86 | +* Document test creation and promotion workflows in `CONTRIBUTING.md`. |
| 87 | +* Incrementally deprecate and migrate e2e tests to the new suite. |
0 commit comments