Skip to content

Commit 568f8d3

Browse files
authored
Merge pull request #56794 from adellape/olm_multitenancy
2 parents cba4b33 + 9e2e289 commit 568f8d3

13 files changed

+314
-11
lines changed

_topic_maps/_topic_map.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1620,6 +1620,9 @@ Topics:
16201620
- Name: Red Hat-provided Operator catalogs
16211621
Distros: openshift-enterprise
16221622
File: olm-rh-catalogs
1623+
- Name: Operators in multitenant clusters
1624+
Distros: openshift-enterprise,openshift-origin
1625+
File: olm-multitenancy
16231626
- Name: CRDs
16241627
Dir: crds
16251628
Topics:
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/understanding/olm/olm-understanding-dependency-resolution.adoc
4+
// * operators/understanding/olm-multitenancy.adoc
5+
6+
:_content-type: CONCEPT
7+
[id="olm-colocation-namespaces_{context}"]
8+
= Colocation of Operators in a namespace
9+
10+
Operator Lifecycle Manager (OLM) handles OLM-managed Operators that are installed in the same namespace, meaning their `Subscription` resources are colocated in the same namespace, as related Operators. Even if they are not actually related, OLM considers their states, such as their version and update policy, when any one of them is updated.
11+
12+
This default behavior manifests in two ways:
13+
14+
* `InstallPlan` resources of pending updates include `ClusterServiceVersion` (CSV) resources of all other Operators that are in the same namespace.
15+
* All Operators in the same namespace share the same update policy. For example, if one Operator is set to manual updates, all other Operators' update policies are also set to manual.
16+
17+
These scenarios can lead to the following issues:
18+
19+
* It becomes hard to reason about install plans for Operator updates, because there are many more resources defined in them than just the updated Operator.
20+
* It becomes impossible to have some Operators in a namespace update automatically while other are updated manually, which is a common desire for cluster administrators.
21+
22+
These issues usually surface because, when installing Operators with the {product-title} web console, the default behavior installs Operators that support the *All namespaces* install mode into the default `openshift-operators` global namespace.
23+
24+
As a cluster administrator, you can bypass this default behavior manually by using the following workflow:
25+
26+
. Create a namespace for the installation of the Operator.
27+
. Create a custom _global Operator group_, which is an Operator group that watches all namespaces. By associating this Operator group with the namespace you just created, it makes the installation namespace a global namespace, which makes Operators installed there available in all namespaces.
28+
. Install the desired Operator in the installation namespace.
29+
30+
If the Operator has dependencies, the dependencies are automatically installed in the pre-created namespace. As a result, it is then valid for the dependency Operators to have the same update policy and shared install plans. For a detailed procedure, see "Installing global Operators in custom namespaces".
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/understanding/olm-multitenancy.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="olm-default-install-modes-behavior_{context}"]
7+
= Default Operator install modes and behavior
8+
9+
When installing Operators with the web console as an administrator, you typically have two choices for the install mode, depending on the Operator's capabilities:
10+
11+
Single namespace:: Installs the Operator in the chosen single namespace, and makes all permissions that the Operator requests available in that namespace.
12+
13+
All namespaces:: Installs the Operator in the default `openshift-operators` namespace to watch and be made available to all namespaces in the cluster. Makes all permissions that the Operator requests available in all namespaces. In some cases, an Operator author can define metadata to give the user a second option for that Operator's suggested namespace.
14+
15+
This choice also means that users in the affected namespaces get access to the Operators APIs, which can leverage the custom resources (CRs) they own, depending on their role in the namespace:
16+
17+
* The `namespace-admin` and `namespace-edit` roles can read/write to the Operator APIs, meaning they can use them.
18+
* The `namespace-view` role can read CR objects of that Operator.
19+
20+
For *Single namespace* mode, because the Operator itself installs in the chosen namespace, its pod and service account are also located there. For *All namespaces* mode, the Operator's privileges are all automatically elevated to cluster roles, meaning the Operator has those permissions in all namespaces.
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/admin/olm-adding-operators-to-cluster.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="olm-installing-global-namespaces_{context}"]
7+
= Installing global Operators in custom namespaces
8+
9+
When installing Operators with the {product-title} web console, the default behavior installs Operators that support the *All namespaces* install mode into the default `openshift-operators` global namespace. This can cause issues related to shared install plans and update policies between all Operators in the namespace. For more details on these limitations, see "Colocation of Operators in a namespace".
10+
11+
As a cluster administrator, you can bypass this default behavior manually by creating a custom global namespace and using that namespace to install your individual or scoped set of Operators and their dependencies.
12+
13+
.Procedure
14+
15+
. Before installing the Operator, create a namespace for the installation of your desired Operator. This installation namespace will become the custom global namespace:
16+
17+
.. Define a `Namespace` resource and save the YAML file, for example, `global-operators.yaml`:
18+
+
19+
[source,yaml]
20+
----
21+
apiVersion: v1
22+
kind: Namespace
23+
metadata:
24+
name: global-operators
25+
----
26+
27+
.. Create the namespace by running the following command:
28+
+
29+
[source,terminal]
30+
----
31+
$ oc create -f global-operators.yaml
32+
----
33+
34+
. Create a custom _global Operator group_, which is an Operator group that watches all namespaces:
35+
36+
.. Define an `OperatorGroup` resource and save the YAML file, for example, `global-operatorgroup.yaml`. Omit both the `spec.selector` and `spec.targetNamespaces` fields to make it a _global Operator group_, which selects all namespaces:
37+
+
38+
[source,yaml]
39+
----
40+
apiVersion: operators.coreos.com/v1
41+
kind: OperatorGroup
42+
metadata:
43+
name: global-operatorgroup
44+
namespace: global-operators
45+
----
46+
+
47+
[NOTE]
48+
====
49+
The `status.namespaces` of a created global Operator group contains the empty string (`""`), which signals to a consuming Operator that it should watch all namespaces.
50+
====
51+
52+
.. Create the Operator group by running the following command:
53+
+
54+
[source,terminal]
55+
----
56+
$ oc create -f global-operatorgroup.yaml
57+
----
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/understanding/olm-multitenancy.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="olm-multitenancy-solution_{context}"]
7+
= Recommended solution for multitenant clusters
8+
9+
While a *Multinamespace* install mode does exist, it is supported by very few Operators. As a middle ground solution between the standard *All namespaces* and *Single namespace* install modes, you can install multiple instances of the same Operator, one for each tenant, by using the following workflow:
10+
11+
. Create a namespace for the tenant Operator that is separate from the tenant's namespace.
12+
. Create an Operator group for the tenant Operator scoped only to the tenant's namespace.
13+
. Install the Operator in the tenant Operator namespace.
14+
15+
As a result, the Operator resides in the tenant Operator namespace and watches the tenant namespace, but neither the Operator's pod nor its service account are visible or usable by the tenant.
16+
17+
This solution provides better tenant separation, least privilege principle at the cost of resource usage, and additional orchestration to ensure the constraints are met. For a detailed procedure, see "Preparing for multiple instances of an Operator for multitenant clusters".
18+
19+
.Limitations and considerations
20+
21+
This solution only works when the following constraints are met:
22+
23+
* All instances of the same Operator must be the same version.
24+
* The Operator cannot have dependencies on other Operators.
25+
* The Operator cannot ship a CRD conversion webhook.
26+
27+
[IMPORTANT]
28+
====
29+
You cannot use different versions of the same Operator on the same cluster. Eventually, the installation of another instance of the Operator would be blocked when it meets the following conditions:
30+
31+
* The instance is not the newest version of the Operator.
32+
* The instance ships an older revision of the CRDs that lack information or versions that newer revisions have that are already in use on the cluster.
33+
====
34+
35+
[WARNING]
36+
====
37+
As an administrator, use caution when allowing non-cluster admintrators to install Operators self-sufficiently, as explained in "Allowing non-cluster administrators to install Operators". These tenants should only have access to a curated catalog of Operators that are known to not have dependencies. These tenants must also be forced to use the same version line of an Operator, to ensure the CRDs do not change. This requires the use of namespace-scoped catalogs and likely disabling the global default catalogs.
38+
====
Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/understanding/olm/olm-understanding-operatorgroups.adoc
4+
5+
:_content-type: CONCEPT
16
[id="olm-operatorgroups-limitations"]
2-
= Limitations for multi-tenant Operator management
7+
= Limitations for multitenant Operator management
8+
9+
{product-title} provides limited support for simultaneously installing different versions of an Operator on the same cluster. Operator Lifecycle Manager (OLM) installs Operators multiple times in different namespaces. One constraint of this is that the Operator's API versions must be the same.
10+
11+
Operators are control plane extensions due to their usage of `CustomResourceDefinition` objects (CRDs), which are global resources in Kubernetes. Different major versions of an Operator often have incompatible CRDs. This makes them incompatible to install simultaneously in different namespaces on a cluster.
312

4-
{product-title} provides limited support for simultaneously installing different variations of an Operator on a cluster. Operators are control plane extensions. All tenants, or namespaces, share the same control plane of a cluster. Therefore, tenants in a multi-tenant environment also have to share Operators.
13+
All tenants, or namespaces, share the same control plane of a cluster. Therefore, tenants in a multitenant cluster also share global CRDs, which limits the scenarios in which different instances of the same Operator can be used in parallel on the same cluster.
514

6-
The Operator Lifecycle Manager (OLM) installs Operators multiple times in different namespaces. One constraint of this is that the Operator’s API versions must be the same.
15+
The supported scenarios include the following:
716

8-
Different major versions of an Operator often have incompatible custom resource definitions (CRDs). This makes it difficult to quickly verify OLMs.
17+
* Operators of different versions that ship the exact same CRD definition (in case of versioned CRDs, the exact same set of versions)
18+
* Operators of different versions that do not ship a CRD, and instead have their CRD available in a separate bundle on the OperatorHub
919
10-
[role="_additional-resources"]
11-
== Additional resources
12-
* xref:../../../operators/admin/olm-creating-policy.adoc#olm-creating-policy[Allowing non-cluster administrators to install Operators]
20+
All other scenarios are not supported, because the integrity of the cluster data cannot be guaranteed if there are multiple competing or overlapping CRDs from different Operator versions to be reconciled on the same cluster.
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * operators/admin/olm-adding-operators-to-cluster.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="olm-preparing-operators-multitenant_{context}"]
7+
= Preparing for multiple instances of an Operator for multitenant clusters
8+
9+
As a cluster administrator, you can add multiple instances of an Operator for use in multitenant clusters. This is an alternative solution to either using the standard *All namespaces* install mode, which can be considered to violate the principle of least privilege, or the *Multinamespace* mode, which is not widely adopted. For more information, see "Operators in multitenant clusters".
10+
11+
In the following procedure, the _tenant_ is a user or group of users that share common access and privileges for a set of deployed workloads. The _tenant Operator_ is the instance of an Operator that is intended for use by only that tenant.
12+
13+
.Prerequisites
14+
15+
* All instances of the Operator you want to install must be the same version across a given cluster.
16+
+
17+
[IMPORTANT]
18+
====
19+
For more information on this and other limitations, see "Operators in multitenant clusters".
20+
====
21+
22+
.Procedure
23+
24+
. Before installing the Operator, create a namespace for the tenant Operator that is separate from the tenant's namespace. For example, if the tenant's namespace is `team1`, you might create a `team1-operator` namespace:
25+
26+
.. Define a `Namespace` resource and save the YAML file, for example, `team1-operator.yaml`:
27+
+
28+
[source,yaml]
29+
----
30+
apiVersion: v1
31+
kind: Namespace
32+
metadata:
33+
name: team1-operator
34+
----
35+
36+
.. Create the namespace by running the following command:
37+
+
38+
[source,terminal]
39+
----
40+
$ oc create -f team1-operator.yaml
41+
----
42+
43+
. Create an Operator group for the tenant Operator scoped to the tenant's namespace, with only that one namespace entry in the `spec.targetNamespaces` list:
44+
45+
.. Define an `OperatorGroup` resource and save the YAML file, for example, `team1-operatorgroup.yaml`:
46+
+
47+
[source,yaml]
48+
----
49+
apiVersion: operators.coreos.com/v1
50+
kind: OperatorGroup
51+
metadata:
52+
name: team1-operatorgroup
53+
namespace: team1-operator
54+
spec:
55+
targetNamespaces:
56+
- team1 <1>
57+
----
58+
<1> Define only the tenant's namespace in the `spec.targetNamespaces` list.
59+
60+
.. Create the Operator group by running the following command:
61+
+
62+
[source,terminal]
63+
----
64+
$ oc create -f team1-operatorgroup.yaml
65+
----
66+

modules/olm-terms.adoc

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
//
33
// * operators/understanding/olm/olm-common-terms.adoc
44

5+
:_content-type: CONCEPT
56
[id="olm-common-terms-glossary_{context}"]
67
= Common Operator Framework terms
78

@@ -15,7 +16,7 @@ In the bundle format, a _bundle image_ is a container image that is built from O
1516

1617
[id="olm-common-terms-catalogsource_{context}"]
1718
== Catalog source
18-
A _catalog source_ is a repository of CSVs, CRDs, and packages that define an application.
19+
A _catalog source_ represents a store of metadata that OLM can query to discover and install Operators and their dependencies.
1920

2021
[id="olm-common-terms-channel_{context}"]
2122
== Channel
@@ -50,6 +51,12 @@ In the bundle format, an _index image_ refers to an image of a database (a datab
5051
== Install plan
5152
An _install plan_ is a calculated list of resources to be created to automatically install or upgrade a CSV.
5253

54+
[id="olm-common-terms-multitenancy_{context}"]
55+
== Multitenancy
56+
A _tenant_ in {product-title} is a user or group of users that share common access and privileges for a set of deployed workloads, typically represented by a namespace or project. You can use tenants to provide a level of isolation between different groups or teams.
57+
58+
When a cluster is shared by multiple users or groups, it is considered a _multitenant_ cluster.
59+
5360
[id="olm-common-terms-operatorgroup_{context}"]
5461
== Operator group
5562

operators/admin/olm-adding-operators-to-cluster.adoc

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,41 @@ include::modules/olm-installing-from-operatorhub-using-cli.adoc[leveloffset=+1]
3333
* xref:../../operators/understanding/olm/olm-understanding-operatorgroups.adoc#olm-operatorgroups-about_olm-understanding-operatorgroups[About Operator groups]
3434
3535
include::modules/olm-installing-specific-version-cli.adoc[leveloffset=+1]
36-
3736
[role="_additional-resources"]
3837
.Additional resources
3938
4039
* xref:../../operators/admin/olm-upgrading-operators.adoc#olm-approving-pending-upgrade_olm-upgrading-operators[Manually approving a pending Operator update]
4140
41+
include::modules/olm-preparing-multitenant-operators.adoc[leveloffset=+1]
42+
.Next steps
43+
44+
* Install the Operator in the tenant Operator namespace. This task is more easily performed by using the OperatorHub in the web console instead of the CLI; for a detailed procedure, see xref:../../operators/admin/olm-adding-operators-to-cluster.adoc#olm-installing-from-operatorhub-using-web-console_olm-adding-operators-to-a-cluster[Installing from OperatorHub using the web console].
45+
+
46+
[NOTE]
47+
====
48+
After completing the Operator installation, the Operator resides in the tenant Operator namespace and watches the tenant namespace, but neither the Operator's pod nor its service account are visible or usable by the tenant.
49+
====
50+
51+
[role="_additional-resources"]
52+
.Additional resources
53+
54+
* xref:../../operators/understanding/olm-multitenancy.adoc#olm-multitenancy[Operators in multitenant clusters]
55+
56+
include::modules/olm-installing-global-namespaces.adoc[leveloffset=+1]
57+
.Next steps
58+
59+
* Install the desired Operator in the installation namespace. This task is more easily performed by using the OperatorHub in the web console instead of the CLI; for a detailed procedure, see xref:../../operators/admin/olm-adding-operators-to-cluster.adoc#olm-installing-from-operatorhub-using-web-console_olm-adding-operators-to-a-cluster[Installing from OperatorHub using the web console].
60+
+
61+
[NOTE]
62+
====
63+
When you initiate the Operator installation, if the Operator has dependencies, the dependencies are also automatically installed in the custom global namespace. As a result, it is then valid for the dependency Operators to have the same update policy and shared install plans.
64+
====
65+
66+
[role="_additional-resources"]
67+
.Additional resources
68+
69+
* xref:../../operators/understanding/olm/olm-understanding-dependency-resolution.adoc#olm-colocation-namespaces_olm-understanding-dependency-resolution[Colocation of Operators in a namespace]
70+
4271
include::modules/olm-pod-placement.adoc[leveloffset=+1]
4372
4473
[role="_additional-resources"]

0 commit comments

Comments
 (0)