Skip to content

Commit 5ce2f11

Browse files
jbw976jbw976
authored
Merge pull request crossplane#6226 from jbw976/usage-beta-migrator
define an API promotion process
2 parents 395512a + 424e081 commit 5ce2f11

File tree

3 files changed

+111
-6
lines changed

3 files changed

+111
-6
lines changed

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ We love pull requests that fix an open issue. If yours does, use the below line
1414
to indicate which issue it fixes, for example "Fixes #500".
1515
-->
1616

17-
Fixes #
17+
Fixes #
1818

1919
I have: <!--You MUST either [x] check or [ ] ~strike through~ every item.-->
2020

@@ -24,10 +24,12 @@ I have: <!--You MUST either [x] check or [ ] ~strike through~ every item.-->
2424
- [ ] Added or updated e2e tests.
2525
- [ ] Linked a PR or a [docs tracking issue] to [document this change].
2626
- [ ] Added `backport release-x.y` labels to auto-backport this PR.
27+
- [ ] Followed the [API promotion workflow] if this PR introduces, removes, or promotes an API.
2728

2829
Need help with this checklist? See the [cheat sheet].
2930

3031
[contribution process]: https://github.com/crossplane/crossplane/tree/main/contributing
3132
[docs tracking issue]: https://github.com/crossplane/docs/issues/new
3233
[document this change]: https://docs.crossplane.io/contribute/contribute
3334
[cheat sheet]: https://github.com/crossplane/crossplane/tree/main/contributing#checklist-cheat-sheet
35+
[API promotion workflow]: https://github.com/crossplane/crossplane/blob/main/contributing/guide-api-promotion.md

‎cmd/crossplane/core/init.go‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,11 +72,6 @@ func (c *initCommand) Run(s *runtime.Scheme, log logging.Logger) error {
7272
}
7373
steps = append(steps,
7474
initializer.NewTLSCertificateGenerator(c.Namespace, c.TLSCASecretName, tlsGeneratorOpts...),
75-
initializer.NewCoreCRDsMigrator("compositionrevisions.apiextensions.crossplane.io", "v1alpha1"),
76-
initializer.NewCoreCRDsMigrator("environmentconfigs.apiextensions.crossplane.io", "v1beta1"),
77-
initializer.NewCoreCRDsMigrator("functions.pkg.crossplane.io", "v1beta1"),
78-
initializer.NewCoreCRDsMigrator("functionrevisions.pkg.crossplane.io", "v1beta1"),
79-
initializer.NewCoreCRDsMigrator("locks.pkg.crossplane.io", "v1alpha1"),
8075
)
8176
if c.WebhookEnabled {
8277
nn := types.NamespacedName{
@@ -97,6 +92,17 @@ func (c *initCommand) Run(s *runtime.Scheme, log logging.Logger) error {
9792
)
9893
}
9994

95+
// CRD migrator steps are done after core CRDs are applied/updated, so we
96+
// are always migrating to the most current storage version
97+
steps = append(steps,
98+
initializer.NewCoreCRDsMigrator("compositionrevisions.apiextensions.crossplane.io", "v1alpha1"),
99+
initializer.NewCoreCRDsMigrator("environmentconfigs.apiextensions.crossplane.io", "v1beta1"),
100+
initializer.NewCoreCRDsMigrator("usages.apiextensions.crossplane.io", "v1beta1"),
101+
initializer.NewCoreCRDsMigrator("functions.pkg.crossplane.io", "v1beta1"),
102+
initializer.NewCoreCRDsMigrator("functionrevisions.pkg.crossplane.io", "v1beta1"),
103+
initializer.NewCoreCRDsMigrator("locks.pkg.crossplane.io", "v1alpha1"),
104+
)
105+
100106
if c.ESSTLSServerSecretName != "" {
101107
steps = append(steps, initializer.NewTLSCertificateGenerator(c.Namespace, c.TLSCASecretName,
102108
initializer.TLSCertificateGeneratorWithServerSecretName(c.ESSTLSServerSecretName, []string{fmt.Sprintf("*.%s", c.Namespace)}),

‎contributing/guide-api-promotion.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Promoting an API
2+
3+
This guide describes the process for promoting an API in Crossplane from one
4+
version to the next. This process should be handled with care over multiple
5+
releases, as it has proven fairly easy for Crossplane's API promotions of the
6+
past to be problematic for both upgrades and downgrades, as demonstrated by
7+
[#6148], [#5932], and [#4400].
8+
9+
An explanation of Crossplane's feature lifecycle can be found in the [Feature
10+
Lifecycle] docs page.
11+
12+
## Core Issue
13+
14+
The root cause we have encountered in past problematic API promotions is that we
15+
try to drop a particular version from a CRD while there may still be resources
16+
of that version stored in etcd. Kubernetes does not allow this action because it
17+
can result in possible data loss.
18+
19+
Note that this can happen on both upgrades as well as downgrades. Let's briefly
20+
summarize one common downgrade path where we've encountered this issue:
21+
22+
1. A Crossplane API is promoted to Beta and the storage version for the CRD is
23+
updated to Beta in the same release
24+
1. User installs this new Crossplane version and creates a Beta resource that is
25+
stored in etcd as Beta
26+
1. User downgrades to the previous Crossplane version where the CRD's Beta
27+
version did not exist
28+
1. The Crossplane init container tries to update the CRD back to the Alpha
29+
version, which drops the Beta version
30+
1. The Kubernetes API server rejects this action because there might still be
31+
Beta resources stored in etcd according to the CRD's
32+
[`status.storedVersions`] field
33+
1. Crossplane init container crashes and the Crossplane pod restarts in a loop
34+
35+
## Guiding Principles
36+
37+
There are a few simple principles we can follow in order to avoid this common
38+
scenario when promoting Crossplane APIs:
39+
40+
1. Never introduce or drop a CRD version while also bumping the storage version
41+
in the same release
42+
1. Always migrate resources to the current storage version during Crossplane
43+
initialization
44+
45+
Adherence to these two principles should result in never dropping a CRD version
46+
that still has resources of that version stored in etcd, for both the upgrade
47+
and downgrade directions.
48+
49+
## Promotion Workflow
50+
51+
The following table outlines the steps to introduce and then promote an API
52+
safely across multiple versions while maintaining safe upgrades and downgrades:
53+
54+
| Version | Description | Alpha | Beta | Migration |
55+
|---------|----------------------------|-----------------|-----------------|------------------|
56+
| v0.1 | introduce new API as alpha | storage, served | not exist | none |
57+
| v0.2 | promote to beta | storage, served | served | migrate to alpha |
58+
| v0.3 | bump storage to beta | served | storage, served | migrate to beta |
59+
| v0.4 | drop alpha | not exist | storage, served | none |
60+
61+
The same process could be applied again when promoting the API from Beta to v1
62+
GA.
63+
64+
## Practical Pointers
65+
66+
This section contains some helpful pointers to areas of the codebase involved
67+
in promoting an API. These are just general direction and not entirely
68+
prescriptive, because you'll need to make specific decisions for your promotion
69+
based on the workflow defined above.
70+
71+
1. Define the API types under the `apis` directory, for example
72+
[`apis/apiextensions/v1beta1`]
73+
1. Set the storage version `+kubebuilder:storageversion` marker on the correct
74+
version, e.g., as shown here for [Usage v1alpha1]
75+
1. Duplicate the API to other versions if needed using [`generate.go`], also
76+
instructing the [duplicate script] to set the storage version if needed
77+
1. Run `earthly +generate` to generate the CRDs and check them for sanity in the
78+
[`cluster/crds`] directory
79+
1. Include a [migrator] if needed to ensure resources are migrated to the
80+
current storage version and the CRD status is updated to declare that it only
81+
has resources stored in etcd of the current storage version
82+
* **Note** that the version passed into the migrator is the **"old"**
83+
version you want to migrate **from**, not the target version you want to
84+
migrate to
85+
86+
<!-- Links -->
87+
[#6148]: https://github.com/crossplane/crossplane/issues/6148
88+
[#5932]: https://github.com/crossplane/crossplane/issues/5932
89+
[#4400]: https://github.com/crossplane/crossplane/issues/4400
90+
[Feature Lifecycle]: https://docs.crossplane.io/latest/learn/feature-lifecycle/
91+
[`status.storedVersions`]: https://github.com/kubernetes/apiextensions-apiserver/blob/v0.32.0/pkg/apis/apiextensions/v1/types.go#L368-L376
92+
[`apis/apiextensions/v1beta1`]: https://github.com/crossplane/crossplane/tree/release-1.18/apis/apiextensions/v1beta1
93+
[Usage v1alpha1]: https://github.com/crossplane/crossplane/blob/release-1.18/apis/apiextensions/v1alpha1/usage_types.go#L89
94+
[`generate.go`]: https://github.com/crossplane/crossplane/blob/release-1.18/apis/generate.go
95+
[duplicate script]: https://github.com/crossplane/crossplane/blob/release-1.18/hack/duplicate_api_type.sh
96+
[`cluster/crds`]: https://github.com/crossplane/crossplane/tree/release-1.18/cluster/crds
97+
[migrator]: https://github.com/crossplane/crossplane/blob/release-1.18/cmd/crossplane/core/init.go#L75-L79

0 commit comments

Comments
 (0)