Skip to content

Commit 91d5be0

Browse files
tssuryatssurya
committed
Add conformance profiles NPEP
Signed-off-by: Surya Seetharaman <[email protected]>
1 parent 778a9a1 commit 91d5be0

File tree

1 file changed

+189
-0
lines changed

1 file changed

+189
-0
lines changed

‎npep/npep-137-conformance-profiles.md

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# NPEP-137: Conformance Profiles
2+
3+
* Issue: [#137](https://github.com/kubernetes-sigs/network-policy-api/issues/137)
4+
* Status: Implementable
5+
6+
## TLDR
7+
8+
Add conformance profiles for existing conformance tests which implementations
9+
can leverage when running the conformance test suite in their downstream repos
10+
to report the results of the tests back to the Network Policy API project and
11+
receive recognition (eg. "conformance badge")
12+
13+
* NOTE: Adapted with love from https://gateway-api.sigs.k8s.io/geps/gep-1709/ (Then
14+
a question may arise, why do we need our own NPEP? Answer: Gateway API Project has
15+
lots of resources and features and it gets very complicated to adopt their same API.
16+
We need to tune the profiles to be compatible with what works for Network Policy API
17+
project, thus the need for this NPEP). Our profiles are much simpler than what GatewayAPI
18+
supports.
19+
20+
## Goals
21+
22+
* Add conformance profiles for existing tests that downstream implementations
23+
can subscribe to to run tests associated to the supported API feature sets
24+
25+
* Add a reporting mechanism via a new CRD where conformance results can be
26+
reported back to the Network Policy API project and provide "conformance badges"
27+
via Network Policy official website to recognize these implementations.
28+
29+
* Expand existing conformance testing framework to account for profiles
30+
31+
## Non-Goals
32+
33+
* The first iteration will just have a simple repo under which the report
34+
results will be stored, no automated reporting infrastructure will be built for this now.
35+
Implementations will need to open PRs against network-policy-api repo to upload
36+
the results.
37+
38+
## Introduction
39+
40+
Currently, the conformance tests are grouped into `CoreFeatures` and
41+
`ExtendedFeatures`. The support for features that fall under the `CoreFeatures`
42+
are a requirement for conformant implementations, while the support for features that fall
43+
under `ExtendedFeatures` are optional and do not gate API conformance for implementations.
44+
45+
In this NPEP, we will add a concept called `named profiles` which will indicate a
46+
"level of conformance" for a given resource. We are picking profiles on a resource
47+
level because that is what makes sense for the project as of today. The initial decision
48+
to report conformance profiles on a per-resource level is a choice which may be re-evaluated
49+
by the NetworkPolicy API working group in the future. Using these `named profiles`,
50+
implementations can run the conformance tests, prove they satisfy the requirements for each profile,
51+
and receive conformance badges from the Network Policy API project.
52+
53+
This NPEP aims to provide an API (CRD) for reporting results from these conformance
54+
tests. CLI tooling will also be provided for invoking the conformance tests with
55+
the profile names so that the results can be easily submitted back upstream.
56+
57+
## User-Stories/Use-Cases
58+
59+
Story 1: Receive an API conformance badge
60+
61+
As a CNI maintainer, I want to receive recognition from Network Policy API subgroup
62+
that my plugin implements the `AdminNetworkPolicy` API. This badge will enable us
63+
to advertise upstream feature support in our plugin easily to our users.
64+
65+
Story 2: Track Implementations for each API resource
66+
67+
As a Network Policy project maintainer, I would like to easily track which
68+
implementations are using the defined API resources in our project so that it
69+
paves way for easier collaboration and feedback loops when changes are introduced
70+
to the API in each release.
71+
72+
## API
73+
74+
We will add a new API resource called `ConformanceReport` which will be at
75+
the center of our test result reporting workflow. The implementors running
76+
the tests can then:
77+
78+
1. Choose a `named profile`
79+
2. Integrate the tests required by a given profile in their downstream project
80+
3. Report the results to Network Policy API project using this new API resource
81+
4. Get a conformance badge recognition via our official website
82+
83+
## Profiles
84+
85+
Named Profiles for Network Policy API project will be tied to each API resource.
86+
We will start with two named profiles and expand this as the project evolves.
87+
88+
1. AdminNetworkPolicy
89+
2. BaselineAdminNetworkPolicy
90+
91+
Each of these profiles may have a combination of conformance tests that fall under
92+
`CoreFeatures` and `ExtendedFeatures`. Example; if you pick the profile
93+
`AdminNetworkPolicy`, all tests like `AdminNetworkPolicyEgressSCTP` and
94+
`AdminNetworkPolicyPriorityField` fall under the `SupportAdminNetworkPolicy` feature
95+
which is under the `CoreFeatures` subset. So these tests must pass for the
96+
`AdminNetworkPolicy` profile conformance. Whereas tests (we don't have any today)
97+
that fall under `AdminNetworkPolicySameLabels` feature which is under the
98+
`ExtendedFeatures` subset are not mandatory for `AdminNetworkPolicy`
99+
profile conformance.
100+
101+
## Integration
102+
103+
The conformance profile test suite can be integrated, invoked and run from your implementation
104+
using two methods:
105+
106+
* The `go test` CLI commands specifying the required information need to generate
107+
a conformance test report. Sample:
108+
```
109+
go test -v ./conformance -run TestConformanceProfiles -args --conformance-profiles=AdminNetworkPolicy,BaselineAdminNetworkPolicy --organization=ovn-org -project=ovn-kubernetes -url=<project-url> -version=0.1.1 -contact=<> -additionalinfo=<link-to-implementation>
110+
```
111+
* Using the conformance profile test suite `TestConformanceProfiles` by directly customizing it by providing
112+
the correct arguments. Sample:
113+
```
114+
cpSuite := suite.NewConformanceProfileTestSuite(
115+
suite.ConformanceProfileOptions{
116+
suite.Options: cSuiteDefaultOptions,
117+
Implementation: confv1alpha1.Implementation{
118+
Organization: "ovn-org",
119+
Project: "OVN-Kubernetes CNI",
120+
URL: "https://github.com/ovn-org/ovn-kubernetes",
121+
Version: "0.1.1",
122+
Contact: []string{"@tssurya"},
123+
AdditionalInformation: "https://github.com/ovn-org/ovn-kubernetes/blob/1c9f73dc8a755c07b22858c7404a7884970d1989/test/conformance/network_policy_v2_test.go"
124+
},
125+
ConformanceProfiles: sets.New(
126+
suite.ANPConformanceProfileName,
127+
suite.BANPConformanceProfileName,
128+
),
129+
})
130+
```
131+
132+
## Reporting process and certification
133+
134+
The reporting process is related to a specific API's version and channel (core and experimental).
135+
There are fields in the ConformanceReport CRD that includes such information. Any implementation
136+
can run the existing conformance test suite specifying the profiles they support and that will
137+
generate an output that looks like this:
138+
139+
```
140+
Conformance report:
141+
apiVersion: policy.networking.k8s.io/v1alpha1
142+
date: "2023-10-03T08:15:25+02:00"
143+
implementation:
144+
contact:
145+
- "@tssurya"
146+
organization: ovn-org
147+
project: OVN-Kubernetes CNI
148+
url: "https://github.com/ovn-org/ovn-kubernetes"
149+
version: 0.1.1
150+
additionalInformation: "https://github.com/ovn-org/ovn-kubernetes/blob/1c9f73dc8a755c07b22858c7404a7884970d1989/test/conformance/network_policy_v2_test.go"
151+
kind: ConformanceReport
152+
networkPolicyV2APIVersion: v0.1.1
153+
profiles:
154+
- core:
155+
failedTests:
156+
- AdminNetworkPolicyIngressSCTP
157+
- AdminNetworkPolicyEgressUDP
158+
result: failure
159+
statistics:
160+
Failed: 2
161+
Passed: 5
162+
Skipped: 0
163+
summary: ""
164+
name: AdminNetworkPolicy
165+
- core:
166+
result: success
167+
statistics:
168+
Failed: 0
169+
Passed: 7
170+
Skipped: 0
171+
summary: ""
172+
name: BaselineAdminNetworkPolicy
173+
```
174+
175+
This can then be uploaded to network-policy-api/conformance/reports/v.x.x/cni-name.yaml by
176+
opening a PR. That will then be reviewed and approved by maintainers thus recognizing the
177+
implementations that are conformant. It is recommended for the implementations to use the
178+
`additionalInformation` field to provide links to the implementation or github actions or
179+
jenkins or any other CI/CD job definitions that helped generate this report. This will
180+
help maintainers make an informed decision on merging the report PR.
181+
182+
## Alternatives
183+
184+
N/A - We just went with what Gateway API project already has implemented without
185+
having to reinvent the wheel.
186+
187+
## References
188+
189+
1. https://github.com/kubernetes-sigs/network-policy-api/pull/142

0 commit comments

Comments
 (0)