-
Notifications
You must be signed in to change notification settings - Fork 575
WIP: OKD-259: Support an "OKD" featureset to be enabled by default on OKD clusters #2451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jatinsu
wants to merge
10
commits into
openshift:master
Choose a base branch
from
jatinsu:OKD-featuregates
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+25,131
−151
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
12d7084
Introduce new OKD featureset and enable all default featuregates
jatinsu d00467f
ran make update
jatinsu 031dd5f
adding files to test
jatinsu b3eded2
updated and vendored api
jatinsu a06cae0
updated all generated files to reflect new feature gates
jatinsu 434eefb
Re-add hack/update-protobuf
jatinsu 2c0434b
Add kube-api-server crds
jatinsu d740400
adding listType=atomic to excludeSubnets
jatinsu 3b37112
Fixing integration test failure
jatinsu 7426300
Fix error in integration tests
jatinsu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
761 changes: 761 additions & 0 deletions
761
..._generated.crd-manifests/0000_00_cluster-version-operator_01_clusterversions-OKD.crd.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
437 changes: 437 additions & 0 deletions
437
config/v1/zz_generated.crd-manifests/0000_10_config-operator_01_apiservers-OKD.crd.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
415 changes: 415 additions & 0 deletions
415
...1/zz_generated.crd-manifests/0000_10_config-operator_01_clusterimagepolicies-OKD.crd.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
416 changes: 416 additions & 0 deletions
416
config/v1/zz_generated.crd-manifests/0000_10_config-operator_01_imagepolicies-OKD.crd.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
183 changes: 183 additions & 0 deletions
183
config/v1/zz_generated.crd-manifests/0000_10_config-operator_01_images-OKD.crd.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,183 @@ | ||
apiVersion: apiextensions.k8s.io/v1 | ||
kind: CustomResourceDefinition | ||
metadata: | ||
annotations: | ||
api-approved.openshift.io: https://github.com/openshift/api/pull/470 | ||
api.openshift.io/merged-by-featuregates: "true" | ||
include.release.openshift.io/ibm-cloud-managed: "true" | ||
include.release.openshift.io/self-managed-high-availability: "true" | ||
release.openshift.io/bootstrap-required: "true" | ||
release.openshift.io/feature-set: OKD | ||
name: images.config.openshift.io | ||
spec: | ||
group: config.openshift.io | ||
names: | ||
kind: Image | ||
listKind: ImageList | ||
plural: images | ||
singular: image | ||
scope: Cluster | ||
versions: | ||
- name: v1 | ||
schema: | ||
openAPIV3Schema: | ||
description: |- | ||
Image governs policies related to imagestream imports and runtime configuration | ||
for external registries. It allows cluster admins to configure which registries | ||
OpenShift is allowed to import images from, extra CA trust bundles for external | ||
registries, and policies to block or allow registry hostnames. | ||
When exposing OpenShift's image registry to the public, this also lets cluster | ||
admins specify the external hostname. | ||
|
||
Compatibility level 1: Stable within a major release for a minimum of 12 months or 3 minor releases (whichever is longer). | ||
properties: | ||
apiVersion: | ||
description: |- | ||
APIVersion defines the versioned schema of this representation of an object. | ||
Servers should convert recognized schemas to the latest internal value, and | ||
may reject unrecognized values. | ||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources | ||
type: string | ||
kind: | ||
description: |- | ||
Kind is a string value representing the REST resource this object represents. | ||
Servers may infer this from the endpoint the client submits requests to. | ||
Cannot be updated. | ||
In CamelCase. | ||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds | ||
type: string | ||
metadata: | ||
type: object | ||
spec: | ||
description: spec holds user settable values for configuration | ||
properties: | ||
additionalTrustedCA: | ||
description: |- | ||
additionalTrustedCA is a reference to a ConfigMap containing additional CAs that | ||
should be trusted during imagestream import, pod image pull, build image pull, and | ||
imageregistry pullthrough. | ||
The namespace for this config map is openshift-config. | ||
properties: | ||
name: | ||
description: name is the metadata.name of the referenced config | ||
map | ||
type: string | ||
required: | ||
- name | ||
type: object | ||
allowedRegistriesForImport: | ||
description: |- | ||
allowedRegistriesForImport limits the container image registries that normal users may import | ||
images from. Set this list to the registries that you trust to contain valid Docker | ||
images and that you want applications to be able to import from. Users with | ||
permission to create Images or ImageStreamMappings via the API are not affected by | ||
this policy - typically only administrators or system integrations will have those | ||
permissions. | ||
items: | ||
description: |- | ||
RegistryLocation contains a location of the registry specified by the registry domain | ||
name. The domain name might include wildcards, like '*' or '??'. | ||
properties: | ||
domainName: | ||
description: |- | ||
domainName specifies a domain name for the registry | ||
In case the registry use non-standard (80 or 443) port, the port should be included | ||
in the domain name as well. | ||
type: string | ||
insecure: | ||
description: |- | ||
insecure indicates whether the registry is secure (https) or insecure (http) | ||
By default (if not specified) the registry is assumed as secure. | ||
type: boolean | ||
type: object | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
externalRegistryHostnames: | ||
description: |- | ||
externalRegistryHostnames provides the hostnames for the default external image | ||
registry. The external hostname should be set only when the image registry | ||
is exposed externally. The first value is used in 'publicDockerImageRepository' | ||
field in ImageStreams. The value must be in "hostname[:port]" format. | ||
items: | ||
type: string | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
registrySources: | ||
description: |- | ||
registrySources contains configuration that determines how the container runtime | ||
should treat individual registries when accessing images for builds+pods. (e.g. | ||
whether or not to allow insecure access). It does not contain configuration for the | ||
internal cluster registry. | ||
properties: | ||
allowedRegistries: | ||
description: |- | ||
allowedRegistries are the only registries permitted for image pull and push actions. All other registries are denied. | ||
|
||
Only one of BlockedRegistries or AllowedRegistries may be set. | ||
items: | ||
type: string | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
blockedRegistries: | ||
description: |- | ||
blockedRegistries cannot be used for image pull and push actions. All other registries are permitted. | ||
|
||
Only one of BlockedRegistries or AllowedRegistries may be set. | ||
items: | ||
type: string | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
containerRuntimeSearchRegistries: | ||
description: |- | ||
containerRuntimeSearchRegistries are registries that will be searched when pulling images that do not have fully qualified | ||
domains in their pull specs. Registries will be searched in the order provided in the list. | ||
Note: this search list only works with the container runtime, i.e CRI-O. Will NOT work with builds or imagestream imports. | ||
format: hostname | ||
items: | ||
type: string | ||
minItems: 1 | ||
type: array | ||
x-kubernetes-list-type: set | ||
insecureRegistries: | ||
description: insecureRegistries are registries which do not have | ||
a valid TLS certificates or only support HTTP connections. | ||
items: | ||
type: string | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
type: object | ||
x-kubernetes-validations: | ||
- message: Only one of blockedRegistries or allowedRegistries may | ||
be set | ||
rule: 'has(self.blockedRegistries) ? !has(self.allowedRegistries) | ||
: true' | ||
type: object | ||
status: | ||
description: status holds observed values from the cluster. They may not | ||
be overridden. | ||
properties: | ||
externalRegistryHostnames: | ||
description: |- | ||
externalRegistryHostnames provides the hostnames for the default external image | ||
registry. The external hostname should be set only when the image registry | ||
is exposed externally. The first value is used in 'publicDockerImageRepository' | ||
field in ImageStreams. The value must be in "hostname[:port]" format. | ||
items: | ||
type: string | ||
type: array | ||
x-kubernetes-list-type: atomic | ||
internalRegistryHostname: | ||
description: |- | ||
internalRegistryHostname sets the hostname for the default internal image | ||
registry. The value must be in "hostname[:port]" format. | ||
This value is set by the image registry operator which controls the internal registry | ||
hostname. | ||
type: string | ||
type: object | ||
required: | ||
- spec | ||
type: object | ||
served: true | ||
storage: true | ||
subresources: | ||
status: {} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do we prevent this from being an option on OCP clusters?
What options do we have for including things in OCP but not OKD? Does OKD ship as a different cluster profile perhaps?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes we need to do this. similar to the model that the installer and other repos follow, we can restrict it to be enabled only on OKD clusters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently, you're going to generate an API that will be applied to OCP clusters, which means that in theory we have to either a) block this value from being set at admission time, b) prevent the option from even being included in the API schema
I would probably lean towards b, but that means creating separate payload manifests for OKD I believe 🤔
We also currently have no way to create differences like that in the tooling apart from feature gates, but, that isn't a long term thing, so we would need to provide other options
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we do this in cluster config operator maybe? if the featureset being enabled is OKD and the image is not built for OKD (through the method above), we reject it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But also - the other angle here: Why do we want to prevent it from being turned on for OCP? It will always be a subset of techpreviewNoUpgrade, the only difference being that it will be upgradeable only on OKD clusters (this we can control through CCO as done here). This will mean from an OCP perspective, it will be just another featureset. There are no OKD specific features (nor do i see any in the future).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We used to validate the feature gate transitions and values in the API server directly, but this was removed in favour of CEL, see https://github.com/openshift/kubernetes/pull/1944/files
Config operator doesn't have any admission time validation ability
I don't think we want an extra feature set in OCP. We have TPNU, we have DPNU, we have CNU.
Enabling an OKD feature sat that isn't OKDNU in OCP sends the wrong signal IMO. All of our features sets that have existed in OCP have either been upgradable, or have had
NoUpgrade
in the name.The OKD set can be achieved as a version of CNU if a customer so desired.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would bringing back validation like this: openshift/kubernetes#2420 make sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it's necessary to have in tree validations, but we do need to agree on the behaviour of a new featureset.
I'm particularly keen not to expose the feature set to non-okd clusters
I think we should be able to achieve this with the tools we have available to us today, but we may need to think about how the CVO handles certain resources by restricting files from certain cluster profiles
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean we don't even build the featureset related manifests for OCP ? Or do we build the manifests and just not expose that featureset to OCP? (not sure how this is even possible because we do the detection for whether a component is OKD at build time)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want this featureset to be opt-in? Or on by default in all OKD clusters?
For HyperShift and OCP, they both have the
Default
featureset, but, these featuresets are not necessarily always identical. In fact, HyperShift for some time has been carrying additional gates that are not enabled in the standalone default set.Personally, I think something like this would make more sense, but it is currently implemented using a cluster profile annotation that CVO uses to choose whether or not to install. I suspect if we were to work with the OTA team they'd have some options so that we could include one yaml file for OKD and one for standalone OCP