Skip to content

Commit 487d2ec

Browse files
authored
Merge pull request #2680 from robscott/crd-management
Adding new CRD Management Guide
2 parents 1180445 + a274ef2 commit 487d2ec

File tree

7 files changed

+160
-121
lines changed

7 files changed

+160
-121
lines changed

mkdocs.yml

Lines changed: 19 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,11 @@ plugins:
2727
- redirects:
2828
redirect_maps:
2929
'guides/getting-started.md': 'guides/index.md'
30+
'concepts/guidelines.md': 'guides/api-design.md'
3031
'contributing/community.md': 'contributing/index.md'
3132
'contributing/gamma.md': 'concepts/gamma.md#contributing'
32-
'references/implementers-guide.md': 'reference/implementers-guide.md'
33+
'reference/implementers-guide.md': 'guides/implementers.md'
34+
'references/implementers-guide.md': 'guides/implementers.md'
3335
'references/spec.md': 'reference/spec.md'
3436
'references/policy-attachment.md': 'reference/policy-attachment.md'
3537
- mermaid2
@@ -58,7 +60,6 @@ nav:
5860
API Overview: concepts/api-overview.md
5961
GAMMA: concepts/gamma.md
6062
Conformance: concepts/conformance.md
61-
Implementation Guidelines: concepts/guidelines.md
6263
Roles and Personas: concepts/roles-and-personas.md
6364
Service Facets: concepts/service-facets.md
6465
Security Model: concepts/security-model.md
@@ -68,20 +69,23 @@ nav:
6869
- FAQ: faq.md
6970
- Glossary: concepts/glossary.md
7071
- Guides:
71-
- Getting started: guides/index.md
72-
- Simple Gateway: guides/simple-gateway.md
73-
- HTTP routing: guides/http-routing.md
74-
- HTTP redirects and rewrites: guides/http-redirect-rewrite.md
75-
- HTTP header modifier: guides/http-header-modifier.md
76-
- HTTP traffic splitting: guides/traffic-splitting.md
77-
- Cross-Namespace routing: guides/multiple-ns.md
78-
- TLS: guides/tls.md
79-
- TCP routing: guides/tcp.md
80-
- gRPC Routing: guides/grpc-routing.md
81-
- Migrating from Ingress: guides/migrating-from-ingress.md
82-
- Backend Protocol Selection: guides/backend-protocol.md
72+
- User Guides:
73+
- Getting started: guides/index.md
74+
- Simple Gateway: guides/simple-gateway.md
75+
- HTTP routing: guides/http-routing.md
76+
- HTTP redirects and rewrites: guides/http-redirect-rewrite.md
77+
- HTTP header modifier: guides/http-header-modifier.md
78+
- HTTP traffic splitting: guides/traffic-splitting.md
79+
- Cross-Namespace routing: guides/multiple-ns.md
80+
- TLS: guides/tls.md
81+
- TCP routing: guides/tcp.md
82+
- gRPC Routing: guides/grpc-routing.md
83+
- Migrating from Ingress: guides/migrating-from-ingress.md
84+
- Backend Protocol Selection: guides/backend-protocol.md
85+
- API Design: guides/api-design.md
86+
- CRD Management: guides/crd-management.md
87+
- Implementer's Guide: guides/implementers.md
8388
- Reference:
84-
- Implementer's Guide: reference/implementers-guide.md
8589
- API Types:
8690
- Gateway: api-types/gateway.md
8791
- GatewayClass: api-types/gatewayclass.md

requirements.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ jsmin==3.0.1
55
livereload==2.6.3
66
# mkdocs 2.4.1 requires Markdown < 3.4.0
77
# https://github.com/kubernetes-sigs/gateway-api/pull/1671#issuecomment-1400586465
8-
Markdown==3.3.7
8+
markdown~=3.2
99
MarkupSafe==2.1.3
1010
mkdocs==1.5.3
1111
mkdocs-awesome-pages-plugin==2.9.2

site-src/concepts/versioning.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,8 @@ The resources that already have a beta API version (v1beta1) are:
8787
* GatewayClass
8888
* ReferenceGrant
8989

90-
In the upcoming v1.0 release, HTTPRoute, Gateway, and GatewayClass will all
91-
graduate to a GA API Version (v1).
90+
In the v1.0 release, HTTPRoute, Gateway, and GatewayClass all graduated to
91+
include a GA API Version (v1).
9292

9393
ReferenceGrant is a special case since it is in the process of [transitioning
9494
into an upstream Kubernetes
@@ -166,6 +166,10 @@ consistent upgrade experiences across versions. To accomplish that, we commit to
166166
a conversion webhook needs to be introduced, it will be supported for the
167167
lifetime of the API, or at least until an alternative is available.
168168

169+
## CRD Management
170+
For information on how to manage Gateway API CRDs in your clusters, refer to our
171+
[CRD Management Guide](/guides/crd-management.md).
172+
169173
## Out of Scope
170174
### Unreleased APIs
171175
This project will have frequent updates to the main branch. There are no

site-src/concepts/guidelines.md renamed to site-src/guides/api-design.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Design guidelines
1+
# API Design Guide
22

33
There are some general design guidelines used throughout this API.
44

site-src/guides/crd-management.md

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# CRD Management
2+
3+
Gateway API is built with CRDs. That comes with a number of significant
4+
benefits, notably that each release of Gateway API supports the 5 more recent
5+
minor versions of Kubernetes. That means you likely won't need to upgrade your
6+
Kubernetes cluster to get the latest version of this API.
7+
8+
Unfortunately, this extra flexibility also adds some room for confusion. This
9+
guide aims to answer some of the most common questions related to Gateway API
10+
CRD management.
11+
12+
## Who Should Manage CRDs?
13+
14+
Ultimately CRDs are a highly-privileged cluster-scoped resource. That means that
15+
either a cluster admin or cluster provider should be responsible for managing
16+
the CRDs in a cluster.
17+
18+
Practically that means that any of the following are reasonable approaches:
19+
20+
* Cluster admin installs CRDs
21+
* Cluster provisioning tool or provider installs and manages CRDs
22+
23+
Some implementations may also want to bundle CRDs to simplify installation. This
24+
is acceptable as long as they never:
25+
26+
1. Overwrite Gateway API CRDs that have unrecognized or newer versions.
27+
1. Overwrite Gateway API CRDs that have a different release channel.
28+
1. Remove Gateway API CRDs.
29+
30+
[Issue #2678](https://github.com/kubernetes-sigs/gateway-api/issues/2678)
31+
explores one possible approach implementations could use to accomplish this.
32+
33+
## Upgrading to a new version
34+
35+
Gateway API releases CRDs in two [release
36+
channels](/concepts/versioning/#release-channels-eg-experimental-standard).
37+
Sticking with standard channel CRDs will ensure CRD upgrades are both simpler
38+
and safer.
39+
40+
### Overall Guidelines
41+
42+
1. Avoid moving backwards. New versions of CRDs can add new fields and features.
43+
Rolling back to a previous version of these CRDs could result in a loss of
44+
that configuration.
45+
1. Read the release notes before upgrading. In some cases, they may contain some
46+
guidelines you need to follow before upgrading.
47+
1. Understand the [Gateway API versioning policy](/concepts/versioning) so you
48+
know what can change.
49+
1. Although it is usually safe to upgrade across multiple Gateway API minor
50+
versions at once, the safest and most widely tested path will involve
51+
upgrading one minor version at a time.
52+
53+
### Validating Webhook
54+
55+
A validating webhook was included with earlier versions of Gateway API. Starting
56+
in v1.0, that webhook has formally been deprecated in favor of the CEL
57+
validation included directly within CRDs. In Gateway API v1.1, the webhook will
58+
be fully removed. That means that the validating webhook is no longer a
59+
consideration when upgrading to newer Gateway API versions.
60+
61+
### API Version Removal
62+
63+
!!! note
64+
This is an advanced use case that is currently only applicable to users that
65+
have been using Gateway API since v0.5.0 within the same cluster.
66+
67+
It's possible that a Gateway API release will remove an alpha API version like
68+
v1alpha2 in CRDs that have newer or more stable API versions. Within the
69+
Standard Channel, the removal of an API version is spread into at least four
70+
minor releases:
71+
72+
1. A newer API version is configured as the storage version.
73+
1. Version is deprecated (will be noted in release notes and via deprecation
74+
warning when using deprecated API version).
75+
1. Version is no longer served but is still included in the CRD for the sake
76+
of automatic translation between API versions.
77+
1. Version is no longer included in the CRD.
78+
79+
If you were using a CRD that went through this process (including the storage
80+
version migration), it's possible that some of your resources are stuck on the
81+
older (deprecated) storage version. When a CRD storage version is updated, that
82+
only takes effect when the individual resources using that CRD are saved again.
83+
84+
For example, if you created a "foo" GatewayClass using Gateway API v0.5.0 CRDs,
85+
the storage version of that GatewayClass would be v1alpha2. If that "foo"
86+
GatewayClass had never been modified or updated by the time you would not be
87+
able to upgrade to Gateway API v1.0.0 CRDs because one of our resources was
88+
still using v1alpha2 as a storage version and that was no longer included in the
89+
CRD (step 4 above).
90+
91+
To be able to upgrade, you'd need to take some action that would update any
92+
GatewayClasses that were using the old storage versions. For example, sending
93+
an empty kubectl patch to each GatewayClass would have this effect. Fortunately
94+
there's a tool that can automate this for us -
95+
[kube-storage-version-migrator](https://github.com/kubernetes-sigs/kube-storage-version-migrator)
96+
will automatically update resources to ensure they're using the latest storage
97+
version.
98+
99+
### Experimental Channel
100+
101+
As the name implies, Experimental Channel does not provide the same stability
102+
guarantees that Standard Channel does. When it comes to a minor release, any of
103+
the following are possible for Experimental Channel CRDs:
104+
105+
* Breaking changes for existing API fields or resources
106+
* Removing API fields or resources without prior deprecation
107+
108+
In practice this means that some upgrades to new Experimental versions may
109+
require you to uninstall and reinstall the Experimental CRDs. If that is ever
110+
the case, it will be clearly communicated in the release notes.

site-src/reference/implementers-guide.md renamed to site-src/guides/implementers.md

Lines changed: 18 additions & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Gateway API Implementer's Guide
1+
# Implementer's Guide
22

33
Everything you wanted to know about building a Gateway API implementation
44
but were too afraid to ask.
@@ -57,23 +57,22 @@ the cluster, if any.
5757

5858
[versioning]: /concepts/versioning
5959

60-
### Changes to the Gateway API CRDs are backwards compatible
60+
### Changes to the Standard Channel CRDs are backwards compatible
6161

62-
Part of the contract for Gateway API CRDs is that changes _within an API version_
63-
must be _compatible_.
62+
Part of the contract for Standard Channel CRDs is that changes _within an API
63+
version_ must be _compatible_. Note that CRDs that are part of Experimental
64+
Channel do not provide any backwards compatibility guarantees.
6465

65-
"Within an API Version" means changes to a CRD that occur while the same API version
66-
(`v1alpha2` or `v1` for example) is in use, and "compatible" means that any new
67-
fields, values, or validation will be added to ensure that _previous_
68-
objects _will still be valid objects_ after the change.
66+
Although the [Gateway API versioning policy](/concepts/versioning) largely
67+
aligns with upstream Kubernetes APIs, it does allow for "corrections to
68+
validation". For example, if the API spec stated that a value was invalid but
69+
the corresponding validation did not cover that, it's possible that a future
70+
release may add validation to prevent that invalid input.
6971

70-
This means that once Gateway API objects move to the `v1` API version, then _all_
71-
changes must be compatible.
72-
73-
This contract also means that an implementation will not fail with a higher version
74-
of the API than the version it was written with, because the newer schema being
75-
stored by Kubernetes will definitely be able to be serialized into the older version
76-
used in code by the implementation.
72+
This contract also means that an implementation will not fail with a higher
73+
version of the API than the version it was written with, because the newer
74+
schema being stored by Kubernetes will definitely be able to be serialized into
75+
the older version used in code by the implementation.
7776

7877
Similarly, if an implementation was written with a _higher_ version, the newer
7978
values that it understands will simply _never be used_, as they are not present
@@ -83,92 +82,9 @@ in the older version.
8382

8483
### CRD Management
8584

86-
For a Gateway API implementation to work, the Gateway API CRDs must be installed
87-
in the Kubernetes cluster the implementation is watching.
88-
89-
Implementations have two possible options: installing CRDs themselves (implementation
90-
controlled) or requiring installation by some other mechanism before working
91-
(externally controlled). Both have tradeoffs, but implementation controlled has
92-
significantly more, and so we DO NOT recommend using implementation controlled
93-
methods at this time.
94-
95-
Regardless, either way has certain things that SHOULD be true, however:
96-
97-
Whatever method is used, infra and cluster admins SHOULD attempt to ensure that
98-
the Bundle version of the CRDs is not _downgraded_. Although we ensure that
99-
API changes are backwards compatible, changing CRD definitions can change the
100-
storage version of the resource, which could have unforeseen effects. Most of the
101-
time, things will probably work, but if it doesn't work, it will most likely
102-
break in weird ways.
103-
104-
Additionally, older versions of the API may be missing fields or features, which
105-
could be very disruptive for users.
106-
107-
Try your best to ensure that the bundle version doesn't roll backwards. It's safer.
108-
109-
Implementations SHOULD also handle the Gateway API CRDs _not_ being present in
110-
the cluster without crashing or panicking. Exiting with a clear fatal error is
111-
acceptable in this case, as is disabling Gateway API support even if enabled in
112-
configuration.
113-
114-
Practically, for implementations using tools like `controller-runtime` or
115-
similar tooling, they may need to check for the _presence_ of the CRDs by
116-
getting the list of installed CRDs before attempting to watch those resources.
117-
(Note that this will require the implementation to have `read` access to those
118-
resources though.)
119-
120-
#### Implementation-controlled CRD installation
121-
122-
Implementation-controlled CRD installation also includes automatic installation
123-
mechanisms such as Helm, if the CRDs are included in a Helm chart with the
124-
implementation's installation.
125-
126-
Because of significant caveats we DO NOT recommend doing implementation-controlled
127-
CRD management at this time.
128-
129-
However, if you really must, CRD definitions MAY be installed by implementations,
130-
but if they do, they MUST have a way to ensure:
131-
132-
- there are no other Gateway API CRDs installed in the cluster before starting, or
133-
- that the CRD definitions are only installed if they are a higher bundle version
134-
than any existing Gateway API CRDs. Note that even this may not be safe if there
135-
are breaking changes in the experimental channel resources, so implementations
136-
should be _very_ careful with doing this.
137-
138-
This avoids problems if another implementation is also installed in the cluster
139-
and expects a higher version of the CRDs to be installed.
140-
141-
The worst outcome here would be two implementations trying to do automatic install
142-
of _different_ CRD versions, resulting in the CRD versions flapping between
143-
versions or channels. This would _not_ produce good outcomes.
144-
145-
The safer method for an automatic installation would require the implementation
146-
to:
147-
148-
- Check if there are any Gateway API CRDs installed in the cluster.
149-
- If not, install its most compatible version of the CRDs.
150-
- If so, only install its version of the CRDs if the bundle version is higher
151-
than the existing one, and the mechanism will also need to check if there are
152-
incompatible changes included in any versions as well.
153-
154-
This is going to be _very_ difficult to pull off in practice.
155-
156-
It should also be noted that many infra and cluster admins manage CRDs using
157-
externally controlled methods that will not be visible to a Gateway
158-
implementation, so if you still proceed with automatic installation, it MUST be
159-
able to be disabled by the installation owner (whether that is the infra or cluster
160-
admin).
161-
162-
Because of all these caveats, we DO NOT recommend doing automatic CRD management
163-
at this time.
164-
165-
#### Externally controlled CRD installation
166-
167-
Because of all of the complexities mentioned in the "Implementation controlled"
168-
section of this document, we recommend that implementations supply documentation
169-
on how to check if CRDs are installed and upgrade versions if required.
170-
171-
Additions to this document to add suggested commands here are welcomed.
85+
For information on how to manage Gateway API CRDs, including when it is
86+
acceptable to bundle CRD installation with your implementation, refer to our
87+
[CRD Management Guide](/guides/crd-management.md).
17288

17389
### Conformance and Version compatibility
17490

@@ -270,7 +186,7 @@ Implementations MAY choose only one GatewayClass out of the pool of otherwise
270186
acceptable GatewayClasses if they can only reconcile one, or, if they are capable
271187
of reconciling multiple GatewayClasses, they may also choose as many as they like.
272188

273-
If something in the GatewayClass renders it incompatibie (at the time of writing,
189+
If something in the GatewayClass renders it incompatible (at the time of writing,
274190
the only possible reason for this is that there is a pointer to a `paramsRef`
275191
object that is not supported by the implementation), then the implementation
276192
SHOULD mark the incompatible GatewayClass as not `Accepted`.

site-src/guides/index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,3 +69,8 @@ resources are in-use or if they were installed by a Gateway controller, then do
6969
not uninstall them. This will uninstall the Gateway API resources for the entire
7070
cluster. Do not do this if they might be in-use by someone else as this will
7171
break anything using these resources.
72+
73+
### More on CRD Management
74+
This guide only provides a high level overview of how to get started with
75+
Gateway API. For more on the topic of managing Gateway API CRDs, refer to our
76+
[CRD Management Guide](/guides/crd-management.md).

0 commit comments

Comments
 (0)