Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ tests:
kind: FeatureGate
spec:
featureSet: bar
expectedError: "spec.featureSet: Unsupported value: \"bar\": supported values: \"CustomNoUpgrade\", \"DevPreviewNoUpgrade\", \"TechPreviewNoUpgrade\", \"\""
expectedError: "spec.featureSet: Unsupported value: \"bar\": supported values: \"CustomNoUpgrade\", \"DevPreviewNoUpgrade\", \"TechPreviewNoUpgrade\", \"OKD\", \"\""
onUpdate:
- name: Default to TechPreview
initial: |
Expand Down Expand Up @@ -121,4 +121,4 @@ tests:
kind: FeatureGate
spec:
featureSet: "bar"
expectedError: "spec.featureSet: Unsupported value: \"bar\": supported values: \"CustomNoUpgrade\", \"DevPreviewNoUpgrade\", \"TechPreviewNoUpgrade\", \"\""
expectedError: "spec.featureSet: Unsupported value: \"bar\": supported values: \"CustomNoUpgrade\", \"DevPreviewNoUpgrade\", \"TechPreviewNoUpgrade\", \"OKD\", \"\""
9 changes: 7 additions & 2 deletions config/v1/types_feature.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,12 @@ var (
// your cluster may fail in an unrecoverable way.
CustomNoUpgrade FeatureSet = "CustomNoUpgrade"

// OKD turns on features for OKD. Turning this feature set ON is supported for OKD clusters, but NOT for OpenShift clusters
// this feature set on CANNOT BE UNDONE for OKD clusters and when enabled on OpenShift clusters it PREVENTS UPGRADES.
OKD FeatureSet = "OKD"
Comment on lines +56 to +58
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we prevent this from being an option on OCP clusters?

What options do we have for including things in OCP but not OKD? Does OKD ship as a different cluster profile perhaps?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes we need to do this. similar to the model that the installer and other repos follow, we can restrict it to be enabled only on OKD clusters.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, you're going to generate an API that will be applied to OCP clusters, which means that in theory we have to either a) block this value from being set at admission time, b) prevent the option from even being included in the API schema

I would probably lean towards b, but that means creating separate payload manifests for OKD I believe 🤔

We also currently have no way to create differences like that in the tooling apart from feature gates, but, that isn't a long term thing, so we would need to provide other options

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

block this value from being set at admission time

can we do this in cluster config operator maybe? if the featureset being enabled is OKD and the image is not built for OKD (through the method above), we reject it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But also - the other angle here: Why do we want to prevent it from being turned on for OCP? It will always be a subset of techpreviewNoUpgrade, the only difference being that it will be upgradeable only on OKD clusters (this we can control through CCO as done here). This will mean from an OCP perspective, it will be just another featureset. There are no OKD specific features (nor do i see any in the future).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do this in cluster config operator maybe?

We used to validate the feature gate transitions and values in the API server directly, but this was removed in favour of CEL, see https://github.com/openshift/kubernetes/pull/1944/files

Config operator doesn't have any admission time validation ability

Why do we want to prevent it from being turned on for OCP?

I don't think we want an extra feature set in OCP. We have TPNU, we have DPNU, we have CNU.

Enabling an OKD feature sat that isn't OKDNU in OCP sends the wrong signal IMO. All of our features sets that have existed in OCP have either been upgradable, or have had NoUpgrade in the name.

The OKD set can be achieved as a version of CNU if a customer so desired.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to validate the feature gate transitions and values in the API server directly, but this was removed in favour of CEL, see https://github.com/openshift/kubernetes/pull/1944/files

Would bringing back validation like this: openshift/kubernetes#2420 make sense?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's necessary to have in tree validations, but we do need to agree on the behaviour of a new featureset.

I'm particularly keen not to expose the feature set to non-okd clusters

I think we should be able to achieve this with the tools we have available to us today, but we may need to think about how the CVO handles certain resources by restricting files from certain cluster profiles

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm particularly keen not to expose the feature set to non-okd clusters

Does this mean we don't even build the featureset related manifests for OCP ? Or do we build the manifests and just not expose that featureset to OCP? (not sure how this is even possible because we do the detection for whether a component is OKD at build time)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want this featureset to be opt-in? Or on by default in all OKD clusters?

For HyperShift and OCP, they both have the Default featureset, but, these featuresets are not necessarily always identical. In fact, HyperShift for some time has been carrying additional gates that are not enabled in the standalone default set.

Personally, I think something like this would make more sense, but it is currently implemented using a cluster profile annotation that CVO uses to choose whether or not to install. I suspect if we were to work with the OTA team they'd have some options so that we could include one yaml file for OKD and one for standalone OCP


// AllFixedFeatureSets are the featuresets that have known featuregates. Custom doesn't for instance. LatencySensitive is dead
AllFixedFeatureSets = []FeatureSet{Default, TechPreviewNoUpgrade, DevPreviewNoUpgrade}
AllFixedFeatureSets = []FeatureSet{Default, TechPreviewNoUpgrade, DevPreviewNoUpgrade, OKD}
)

type FeatureGateSpec struct {
Expand All @@ -67,10 +71,11 @@ type FeatureGateSelection struct {
// Turning on or off features may cause irreversible changes in your cluster which cannot be undone.
// +unionDiscriminator
// +optional
// +kubebuilder:validation:Enum=CustomNoUpgrade;DevPreviewNoUpgrade;TechPreviewNoUpgrade;""
// +kubebuilder:validation:Enum=CustomNoUpgrade;DevPreviewNoUpgrade;TechPreviewNoUpgrade;OKD;""
// +kubebuilder:validation:XValidation:rule="oldSelf == 'CustomNoUpgrade' ? self == 'CustomNoUpgrade' : true",message="CustomNoUpgrade may not be changed"
// +kubebuilder:validation:XValidation:rule="oldSelf == 'TechPreviewNoUpgrade' ? self == 'TechPreviewNoUpgrade' : true",message="TechPreviewNoUpgrade may not be changed"
// +kubebuilder:validation:XValidation:rule="oldSelf == 'DevPreviewNoUpgrade' ? self == 'DevPreviewNoUpgrade' : true",message="DevPreviewNoUpgrade may not be changed"
// +kubebuilder:validation:XValidation:rule="oldSelf == 'OKD' ? self == 'OKD' : true",message="OKD may not be changed"
FeatureSet FeatureSet `json:"featureSet,omitempty"`

// customNoUpgrade allows the enabling or disabling of any feature. Turning this feature set on IS NOT SUPPORTED, CANNOT BE UNDONE, and PREVENTS UPGRADES.
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ spec:
- CustomNoUpgrade
- DevPreviewNoUpgrade
- TechPreviewNoUpgrade
- OKD
- ""
type: string
x-kubernetes-validations:
Expand All @@ -91,6 +92,8 @@ spec:
- message: DevPreviewNoUpgrade may not be changed
rule: 'oldSelf == ''DevPreviewNoUpgrade'' ? self == ''DevPreviewNoUpgrade''
: true'
- message: OKD may not be changed
rule: 'oldSelf == ''OKD'' ? self == ''OKD'' : true'
type: object
x-kubernetes-validations:
- message: .spec.featureSet cannot be removed
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
api-approved.openshift.io: https://github.com/openshift/api/pull/470
api.openshift.io/merged-by-featuregates: "true"
include.release.openshift.io/ibm-cloud-managed: "true"
include.release.openshift.io/self-managed-high-availability: "true"
release.openshift.io/bootstrap-required: "true"
release.openshift.io/feature-set: OKD
name: images.config.openshift.io
spec:
group: config.openshift.io
names:
kind: Image
listKind: ImageList
plural: images
singular: image
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
description: |-
Image governs policies related to imagestream imports and runtime configuration
for external registries. It allows cluster admins to configure which registries
OpenShift is allowed to import images from, extra CA trust bundles for external
registries, and policies to block or allow registry hostnames.
When exposing OpenShift's image registry to the public, this also lets cluster
admins specify the external hostname.

Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer).
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: spec holds user settable values for configuration
properties:
additionalTrustedCA:
description: |-
additionalTrustedCA is a reference to a ConfigMap containing additional CAs that
should be trusted during imagestream import, pod image pull, build image pull, and
imageregistry pullthrough.
The namespace for this config map is openshift-config.
properties:
name:
description: name is the metadata.name of the referenced config
map
type: string
required:
- name
type: object
allowedRegistriesForImport:
description: |-
allowedRegistriesForImport limits the container image registries that normal users may import
images from. Set this list to the registries that you trust to contain valid Docker
images and that you want applications to be able to import from. Users with
permission to create Images or ImageStreamMappings via the API are not affected by
this policy - typically only administrators or system integrations will have those
permissions.
items:
description: |-
RegistryLocation contains a location of the registry specified by the registry domain
name. The domain name might include wildcards, like '*' or '??'.
properties:
domainName:
description: |-
domainName specifies a domain name for the registry
In case the registry use non-standard (80 or 443) port, the port should be included
in the domain name as well.
type: string
insecure:
description: |-
insecure indicates whether the registry is secure (https) or insecure (http)
By default (if not specified) the registry is assumed as secure.
type: boolean
type: object
type: array
x-kubernetes-list-type: atomic
externalRegistryHostnames:
description: |-
externalRegistryHostnames provides the hostnames for the default external image
registry. The external hostname should be set only when the image registry
is exposed externally. The first value is used in 'publicDockerImageRepository'
field in ImageStreams. The value must be in "hostname[:port]" format.
items:
type: string
type: array
x-kubernetes-list-type: atomic
registrySources:
description: |-
registrySources contains configuration that determines how the container runtime
should treat individual registries when accessing images for builds+pods. (e.g.
whether or not to allow insecure access). It does not contain configuration for the
internal cluster registry.
properties:
allowedRegistries:
description: |-
allowedRegistries are the only registries permitted for image pull and push actions. All other registries are denied.

Only one of BlockedRegistries or AllowedRegistries may be set.
items:
type: string
type: array
x-kubernetes-list-type: atomic
blockedRegistries:
description: |-
blockedRegistries cannot be used for image pull and push actions. All other registries are permitted.

Only one of BlockedRegistries or AllowedRegistries may be set.
items:
type: string
type: array
x-kubernetes-list-type: atomic
containerRuntimeSearchRegistries:
description: |-
containerRuntimeSearchRegistries are registries that will be searched when pulling images that do not have fully qualified
domains in their pull specs. Registries will be searched in the order provided in the list.
Note: this search list only works with the container runtime, i.e CRI-O. Will NOT work with builds or imagestream imports.
format: hostname
items:
type: string
minItems: 1
type: array
x-kubernetes-list-type: set
insecureRegistries:
description: insecureRegistries are registries which do not have
a valid TLS certificates or only support HTTP connections.
items:
type: string
type: array
x-kubernetes-list-type: atomic
type: object
x-kubernetes-validations:
- message: Only one of blockedRegistries or allowedRegistries may
be set
rule: 'has(self.blockedRegistries) ? !has(self.allowedRegistries)
: true'
type: object
status:
description: status holds observed values from the cluster. They may not
be overridden.
properties:
externalRegistryHostnames:
description: |-
externalRegistryHostnames provides the hostnames for the default external image
registry. The external hostname should be set only when the image registry
is exposed externally. The first value is used in 'publicDockerImageRepository'
field in ImageStreams. The value must be in "hostname[:port]" format.
items:
type: string
type: array
x-kubernetes-list-type: atomic
internalRegistryHostname:
description: |-
internalRegistryHostname sets the hostname for the default internal image
registry. The value must be in "hostname[:port]" format.
This value is set by the image registry operator which controls the internal registry
hostname.
type: string
type: object
required:
- spec
type: object
served: true
storage: true
subresources:
status: {}
Loading