Skip to content

Commit 455abd6

Browse files
committed
pkg/payload/render: Include release image (Cluster)ImagePolicy
Address [1], where a ClusterImagePolicy manifest from the cluster-update-keys repository was not part of the bootstrap-rendered MachineConfigs, but was part of the production MachineConfigs. That kind of skew causes trouble for the machine-config operator, as in this run [2]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-api-provider-aws/519/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-ovn-techpreview/1818614625273384960/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/machineconfigpools.json | jq -r '.items[0].status.conditions[] | select(.type == "NodeDegraded").message' Node ip-10-0-107-110.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found", Node ip-10-0-20-34.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found", Node ip-10-0-47-251.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found" To address that, this commit renders any ClusterImagePolicy and ImagePolicy manifests from the release image into the output manifests directory. From there: 1. The installer's bootkube service puts the manifest into the central manifests directory [3]: cp cvo-bootstrap/manifests/* manifests/ 2. The installer's bootkube service puts the manifest into into the machine-config controller's manifest directory [4]: cp manifests/* /etc/mcc/bootstrap/ 3. The MCO renders a static MCO bootstrap pod YAML file with --manifest-dir=/etc/mcc/bootstrap/ [5] to configure a manifestDir variable. 4. The installer's bootkube service passes the rendered static MCO bootstrap pod to the kubelet [6]: cp mco-bootstrap/bootstrap/machineconfigoperator-bootstrap-pod.yaml /etc/kubernetes/manifests/ 5. The bootstrap MCO static pod pulls in the ClusterImagePolicy manifest while walking manifestDir [7]. 6. The bootstrap MCO static pod includes feature-gate status and any collected ClusterImagePolicy manifest when rendering bootstrap MachineConfigs [8]. My implementation uses group/kind tuples to figure out what needs rendering, because that is robust against things like manifest renames or other release-image-referenced repositories providing their own manifests (using manifest names in skipFiles is ok, because all the manifests we skip that way are from our own repository, where we control naming). We could extend this to other manifest types that the MCO consumes when rendering MachineConfigs (MachineConfig itself, KubeletConfig, etc.), but I'm leaving those out for now in the optimistic hope that nobody ever needs the CVO to manage those resources. The vendor/ bumps are via: $ go mod vendor with: $ go version go version go1.22.2 linux/amd64 to hopefully address [9]: pkg/payload/render.go:11:2: cannot find module providing package github.com/openshift/api/config: import lookup disabled by -mod=vendor now that we're a direct consumer of that particular package. [1]: https://issues.redhat.com/browse/OCPBUGS-37770 [2]: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-api-provider-aws/519/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-ovn-techpreview/1818614625273384960 [3]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L165 [4]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L431 [5]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/manifests/bootstrap-pod-v2.yaml#L15 [6]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L433 [7]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/pkg/controller/bootstrap/bootstrap.go#L100-L142 [8]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/pkg/controller/bootstrap/bootstrap.go#L183 [9]: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-version-operator/1076/pull-ci-openshift-cluster-version-operator-master-images/1818752166689509376#1:build-log.txt%3A77
1 parent a80fe4e commit 455abd6

File tree

5 files changed

+107
-23
lines changed

5 files changed

+107
-23
lines changed

pkg/payload/payload.go

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -169,9 +169,7 @@ func LoadUpdate(dir, releaseImage, excludeIdentifier string, requiredFeatureSet
169169
continue
170170
}
171171

172-
switch filepath.Ext(file.Name()) {
173-
case ".yaml", ".yml", ".json":
174-
default:
172+
if !hasManifestExtension(file.Name()) {
175173
continue
176174
}
177175

@@ -437,3 +435,12 @@ func loadImageReferences(releaseDir string) (*imagev1.ImageStream, error) {
437435
}
438436
return nil, fmt.Errorf("%s is a %T, not a v1 ImageStream", imageReferencesFile, imageRefObj)
439437
}
438+
439+
func hasManifestExtension(filename string) bool {
440+
switch filepath.Ext(filename) {
441+
case ".yaml", ".yml", ".json":
442+
return true
443+
default:
444+
return false
445+
}
446+
}

pkg/payload/render.go

Lines changed: 67 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,22 @@ import (
88
"strings"
99
"text/template"
1010

11+
"github.com/openshift/api/config"
12+
"github.com/openshift/library-go/pkg/manifest"
1113
"github.com/pkg/errors"
12-
14+
"k8s.io/apimachinery/pkg/runtime/schema"
1315
utilerrors "k8s.io/apimachinery/pkg/util/errors"
1416
"k8s.io/apimachinery/pkg/util/sets"
1517
)
1618

1719
// Render renders critical manifests from /manifests to outputDir.
1820
func Render(outputDir, releaseImage, clusterProfile string) error {
1921
var (
20-
manifestsDir = filepath.Join(DefaultPayloadDir, CVOManifestDir)
21-
oManifestsDir = filepath.Join(outputDir, "manifests")
22-
bootstrapDir = "/bootstrap"
23-
oBootstrapDir = filepath.Join(outputDir, "bootstrap")
22+
manifestsDir = filepath.Join(DefaultPayloadDir, CVOManifestDir)
23+
releaseManifestsDir = filepath.Join(DefaultPayloadDir, ReleaseManifestDir)
24+
oManifestsDir = filepath.Join(outputDir, "manifests")
25+
bootstrapDir = "/bootstrap"
26+
oBootstrapDir = filepath.Join(outputDir, "bootstrap")
2427

2528
renderConfig = manifestRenderConfig{
2629
ReleaseImage: releaseImage,
@@ -29,26 +32,36 @@ func Render(outputDir, releaseImage, clusterProfile string) error {
2932
)
3033

3134
tasks := []struct {
32-
idir string
33-
odir string
34-
skipFiles sets.Set[string]
35+
idir string
36+
odir string
37+
processTemplate bool
38+
skipFiles sets.Set[string]
39+
filterGroupKind sets.Set[schema.GroupKind]
3540
}{{
36-
idir: manifestsDir,
37-
odir: oManifestsDir,
41+
idir: manifestsDir,
42+
odir: oManifestsDir,
43+
processTemplate: true,
3844
skipFiles: sets.New[string](
39-
"image-references",
4045
"0000_90_cluster-version-operator_00_prometheusrole.yaml",
4146
"0000_90_cluster-version-operator_01_prometheusrolebinding.yaml",
4247
"0000_90_cluster-version-operator_02_servicemonitor.yaml",
4348
),
4449
}, {
45-
idir: bootstrapDir,
46-
odir: oBootstrapDir,
47-
skipFiles: sets.Set[string]{},
50+
idir: releaseManifestsDir,
51+
odir: oManifestsDir,
52+
filterGroupKind: sets.New[schema.GroupKind](
53+
schema.GroupKind{Group: config.GroupName, Kind: "ClusterImagePolicy"},
54+
schema.GroupKind{Group: config.GroupName, Kind: "ImagePolicy"},
55+
),
56+
}, {
57+
idir: bootstrapDir,
58+
odir: oBootstrapDir,
59+
processTemplate: true,
60+
skipFiles: sets.Set[string]{},
4861
}}
4962
var errs []error
5063
for _, task := range tasks {
51-
if err := renderDir(renderConfig, task.idir, task.odir, task.skipFiles); err != nil {
64+
if err := renderDir(renderConfig, task.idir, task.odir, task.processTemplate, task.skipFiles, task.filterGroupKind); err != nil {
5265
errs = append(errs, err)
5366
}
5467
}
@@ -60,7 +73,7 @@ func Render(outputDir, releaseImage, clusterProfile string) error {
6073
return nil
6174
}
6275

63-
func renderDir(renderConfig manifestRenderConfig, idir, odir string, skipFiles sets.Set[string]) error {
76+
func renderDir(renderConfig manifestRenderConfig, idir, odir string, processTemplate bool, skipFiles sets.Set[string], filterGroupKind sets.Set[schema.GroupKind]) error {
6477
if err := os.MkdirAll(odir, 0666); err != nil {
6578
return err
6679
}
@@ -73,6 +86,10 @@ func renderDir(renderConfig manifestRenderConfig, idir, odir string, skipFiles s
7386
if file.IsDir() {
7487
continue
7588
}
89+
if !hasManifestExtension(file.Name()) {
90+
continue
91+
}
92+
7693
if skipFiles.Has(file.Name()) {
7794
continue
7895
}
@@ -90,10 +107,40 @@ func renderDir(renderConfig manifestRenderConfig, idir, odir string, skipFiles s
90107
continue
91108
}
92109

93-
rraw, err := renderManifest(renderConfig, iraw)
94-
if err != nil {
95-
errs = append(errs, err)
96-
continue
110+
var rraw []byte
111+
if processTemplate {
112+
rraw, err = renderManifest(renderConfig, iraw)
113+
if err != nil {
114+
errs = append(errs, fmt.Errorf("render template %s from %s: %w", file.Name(), idir, err))
115+
continue
116+
}
117+
} else {
118+
rraw = iraw
119+
}
120+
121+
if len(filterGroupKind) > 0 {
122+
manifests, err := manifest.ParseManifests(bytes.NewReader(rraw))
123+
if err != nil {
124+
errs = append(errs, fmt.Errorf("parse manifest %s from %s: %w", file.Name(), idir, err))
125+
continue
126+
}
127+
128+
filteredManifests := make([]string, 0, len(manifests))
129+
for _, manifest := range manifests {
130+
if filterGroupKind.Has(manifest.GVK.GroupKind()) {
131+
filteredManifests = append(filteredManifests, string(manifest.Raw))
132+
}
133+
}
134+
135+
if len(filteredManifests) == 0 {
136+
continue
137+
}
138+
139+
if len(filteredManifests) == 1 {
140+
rraw = []byte(filteredManifests[0])
141+
} else {
142+
rraw = []byte(strings.Join(filteredManifests, "\n---\n"))
143+
}
97144
}
98145

99146
opath := filepath.Join(odir, file.Name())

vendor/github.com/openshift/api/config/.codegen.yaml

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

vendor/github.com/openshift/api/config/install.go

Lines changed: 27 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

vendor/modules.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ github.com/munnerz/goautoneg
106106
github.com/mwitkow/go-conntrack
107107
# github.com/openshift/api v0.0.0-20240709002213-329ea3755649
108108
## explicit; go 1.22.0
109+
github.com/openshift/api/config
109110
github.com/openshift/api/config/v1
110111
github.com/openshift/api/config/v1/zz_generated.crd-manifests
111112
github.com/openshift/api/config/v1alpha1

0 commit comments

Comments
 (0)