Skip to content

Commit 12ab96d

Browse files
feat: introduce ADR for conformance test suite architecture (open-component-model#1726)
<!-- markdownlint-disable MD041 --> #### What this PR does / why we need it This ADR is supposed to introduce a design for our conformance test suite. It outlines important requirements and discusses multiple considered options. #### Which issue(s) this PR fixes <!-- Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> open-component-model/ocm-project#845 --------- Signed-off-by: Fabian Burth <fabian.burth@sap.com> Co-authored-by: Matthias Bruns <github@matthiasbruns.com>
1 parent ca8975f commit 12ab96d

File tree

2 files changed

+92
-0
lines changed

2 files changed

+92
-0
lines changed

.github/config/wordlist.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ aws
3535
backend
3636
backends
3737
basename
38+
BDD
3839
benchmarking
3940
bindata
4041
blobaccess
@@ -128,6 +129,7 @@ devops
128129
digesters
129130
directorytree
130131
dirtree
132+
Discoverability
131133
dns
132134
dockerconfig
133135
downloadconstraint
@@ -136,6 +138,7 @@ downloaderdescriptor
136138
downloaders
137139
downloadhandlers
138140
dry-run
141+
DSL
139142
dup
140143
dups
141144
dynaml
@@ -448,13 +451,15 @@ substsitution
448451
subtree
449452
suppresstitle
450453
tac
454+
Taskfiles
451455
tbd
452456
tcp
453457
tempfile
454458
templated
455459
templater
456460
templaters
457461
templating
462+
Testcontainers
458463
tgz
459464
timeframes
460465
tls
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# ADR 0014: Conformance Test Suite Design
2+
3+
* **Status**: proposed
4+
* **Deciders**: Maintainers
5+
* **Date**: 2026-02-04
6+
7+
**Technical Story**: We need a robust conformance test suite to ensure our tooling (CLI and Kubernetes controllers) behaves as expected and adheres to a standard specification. This suite should be inspired by the Kubernetes conformance test suite but adapted to be more lightweight.
8+
9+
## Context and Problem Statement
10+
11+
We are a platform engineering team developing a CLI and a Kubernetes controller. We need to ensure that released versions of our tooling are compatible with each other and external systems.
12+
13+
Currently, our `kubernetes/controller` component has Ginkgo-based end-to-end (e2e) tests. We need to evolve this into a proper conformance suite with the following requirements:
14+
15+
1. **Conformance Definition**: Clearly define which tests constitute our "conformance profile".
16+
2. **Versioning & Promotion**: Version conformance tests and provide a workflow to promote e2e tests to conformance.
17+
3. **Tooling Complexity**: Avoid the complexity of Kubernetes' elaborate Ginkgo setup.
18+
4. **Environment Independence**: Tests must run against configurable environments (various OCI registries, Kubernetes clusters).
19+
20+
## Decision Drivers
21+
22+
* **Simplicity**: Preference for standard Go tooling.
23+
* **Maintainability**: Easy to write, read, and debug.
24+
* **Discoverability**: Clear identification of conformance tests.
25+
* **Automation**: Efficient filtering and execution in CI/CD.
26+
27+
## Considered Options
28+
29+
* **Option 1**: **Ginkgo/Gomega with Labels** (Kubernetes style)
30+
* *Description*: Use Ginkgo's Label feature to mark tests as `[Conformance]`.
31+
* **Option 2**: **Standard Go Testing + Testify**
32+
* *Description*: Use standard Go testing with a custom functional labeling mechanism (e.g., `if !HasLabel("conformance") { t.Skip() }`).
33+
34+
## Decision Outcome
35+
36+
Chosen **Option 2**: "Standard Go Testing + Testify".
37+
38+
**Justification**:
39+
* **Lightweight**: Avoids Ginkgo's DSL overhead and implicit behaviors.
40+
* **Native Tooling**: Leveraging `go test` is standard and well-understood.
41+
* **Flexibility**: Functional labeling provides explicit control without framework magic.
42+
* **Separation**: A dedicated `e2e` module avoids circular dependencies.
43+
44+
### Implementation Details
45+
46+
We will introduce a new top-level `e2e` Go module containing all end-to-end tests, a subset of which are conformance tests.
47+
48+
#### Key Design Decisions
49+
50+
1. **Framework**: `stretchr/testify` for assertions and standard `testing.T`.
51+
2. **Identification**:
52+
* Tests must initialize a `TestMeta` struct containing labels.
53+
* Conformance tests are identified by the label `test-kind: conformance`.
54+
3. **Environment Configuration**:
55+
* Tests receive environment configuration via a command-line argument: `./e2e.test -- --config=<config.yaml>`.
56+
* This config defines the OCI registry, Kubernetes cluster, etc.
57+
4. **Promotion**:
58+
* Promote a test by adding `test-kind: conformance` to its `TestMeta` labels via Pull Request.
59+
5. **Versioning**:
60+
* Conformance tests are versioned with the codebase.
61+
6. **Reference Scenarios**:
62+
* Core conformance tests will implement the Reference Scenarios (ADR 0013).
63+
* These will largely be executed as Taskfiles invoked by the Go tests.
64+
65+
#### Technical Details
66+
67+
* **Test Containers**: Use [Testcontainers](https://golang.testcontainers.org/) for isolated dependencies (OCM binary, OCI registries).
68+
* **Providers**: Implement provider interfaces for external dependencies (OCI registries, Clusters) to ensure tests are agnostic to the specific backing service.
69+
70+
## Pros and Cons of the Options
71+
72+
### Option 1: Ginkgo with Labels
73+
74+
* **Pros**: Rich BDD style, built-in labeling, powerful tooling.
75+
* **Cons**: Steep learning curve, implicit behavior, complex bootstrapping.
76+
77+
### Option 2: Standard Go + Testify (Chosen)
78+
79+
* **Pros**: Standard patterns, explicit filtering, simple debugging, preferred by the team.
80+
* **Cons**: Requires manual convention enforcement (mitigated by linters).
81+
82+
## Next Steps
83+
84+
* Create the `e2e` folder and migrate initial tests.
85+
* Update CI pipelines to execute the suite.
86+
* Document test creation and promotion workflows in `CONTRIBUTING.md`.
87+
* Incrementally deprecate and migrate e2e tests to the new suite.

0 commit comments

Comments
 (0)