The Coherence Operator is available as an image from the GitHub container registry ghcr.io/oracle/coherence-operator:{operator-version} that can
easily be installed into a Kubernetes cluster.
Contents
The prerequisites apply to all installation methods.
-
Access to Oracle Coherence Operator images.
-
Access to a Kubernetes v1.19.0+ cluster. The Operator test pipeline is run using Kubernetes versions v1.19 upto v1.26
-
A Coherence application image using Coherence version 12.2.1.3 or later. Note that some functionality (e.g. metrics) is only available in Coherence 12.2.1.4 and later.
|
Note
|
ARM Support: As of version 3.2.0, the Coherence Operator is build as a multi-architecture image that supports running in Kubernetes on both Linux/amd64 and Linux/arm64. The prerequisite is that the Coherence application image used has been built to support ARM. |
|
Note
|
Istio (or similar service meshes) When installing the Operator and Coherence into Kubernetes cluster that use Istio or similar meshes there are a number of pre-requisites that must be understood. See the Istio example for more details. |
There are a number of ways to install the Coherence Operator documented below:
The Coherence Operator runs in HA mode by default. The Deployment created by the installation will have a replica count of 3.
In reduced capacity Kubernetes clusters, for example, local laptop development and test, the replica count can be reduced. It is recommended to leave the default of 3 for production environments.
Instructions on how to change the replica count for the different install methods are included below.
The Coherence Operator runs a REST server that the Coherence cluster members will query to discover the site and rack names that should be used by Coherence. If the Coherence Operator is not running when a Coherence Pod starts, then the Coherence member in that Pod will be unable to properly configure its site and rack names, possibly leading to data distribution that is not safely distributed over sites. In production, and in Kubernetes clusters that are spread over multiple availability zones and failure domains, it is important to run the Operator in HA mode.
The Operator yaml files and Helm chart include a default Pod scheduling configuration that uses anti-affinity to distribute the three replicas onto nodes that have different topology.kubernetes.io/zone labels. This label is a standard Kubernetes label used to describe the zone the node is running in, and is typically applied by Kubernetes cloud vendors.
|
Note
|
Installing the Coherence Operator using the methods below will create a number of ClusterRole RBAC resources.
Some corporate security policies do not like to give cluster wide roles to third-party products.
To help in this situation the operator can be installed without cluster roles, but with caveats
(see the RBAC documentation) for more details.
|
|
Note
|
OpenShift - the Coherence Operator works without modification on OpenShift, but some versions of the Coherence images will not work out of the box. See the OpensShift section of the documentation that explains how to run Coherence clusters with the Operator on OpenShift. |
|
Note
|
Whilst Coherence works out of the box on many Kubernetes installations, some Kubernetes installations may configure iptables in a way that causes Coherence to fail to create clusters. See the O/S Network Configuration section of the documentation for more details if you have well-known-address issues when Pods attempt to form a cluster. |
The Coherence Operator uses a single image, the Operator also runs as an init-container in the Coherence cluster Pods.
-
{operator-image}- The Operator image.
If no image is specified in the Coherence yaml, then the default Coherence image will also be used,
-
{coherence-image}- The default Coherence image.
If using a private image registry then these images will all need to be pushed to that registry for the Operator to work. The default Coherence image may be omitted if all Coherence applications will use custom Coherence images.
The recommended way to install the Coherence Operator is to install a single instance of the operator into a namespace
and where it will then control Coherence resources in all namespaces across the Kubernetes cluster.
Alternatively it may be configured to watch a sub-set of namespaces by setting the WATCH_NAMESPACE environment variable.
The watch namespace(s) does not have to include the installation namespace.
|
Caution
|
In theory, it is possible to install multiple instances of the Coherence Operator into different namespaces, where each instance monitors a different set of namespaces. There are a number of potential issues with this approach, so it is not recommended.
It is possible to run the Operator without web-hooks, but this has its own caveats see the Web Hooks documentation for how to do this. |
|
Important
|
If multiple instance of the Operator are installed, where they are monitoring the same namespaces, this can cause issues.
For example, when a |
If you want the default Coherence Operator installation then the simplest solution is use kubectl to apply the manifests from the Operator release.
kubectl apply -f https://github.com/oracle/coherence-operator/releases/download/v3.4.3/coherence-operator.yamlThis will create a namespace called coherence and install the Operator into it along with all the required ClusterRole and RoleBinding resources. The coherence namespace can be changed by downloading and editing the yaml file.
|
Note
|
Because the coherence-operator.yaml manifest also creates the namespace, the corresponding kubectl delete command will remove the namespace and everything deployed to it! If you do not want this behaviour you should edit the coherence-operator.yaml to remove the namespace section from the start of the file.
|
Instead of using a hard coded version in the command above you can find the latest Operator version using curl:
export VERSION=$(curl -s \
https://api.github.com/repos/oracle/coherence-operator/releases/latest \
| grep '"name": "v' \
| cut -d '"' -f 4 \
| cut -b 2-10)Then download with:
kubectl apply -f https://github.com/oracle/coherence-operator/releases/download/${VERSION}/coherence-operator.yamlThe default install for the Operator is to have one Operator deployment that manages all Coherence resources across all the namespaces in a Kubernetes cluster. This requires the Operator to have cluster role RBAC permissions to manage and monitor all the resources.
Sometimes, for security reasons or for example in a shared Kubernetes cluster this is not desirable.
The Operator can therefore be installed with plain namespaced scoped roles and role bindings.
The Operator release includes a single yaml file named coherence-operator-restricted.yaml that may be used to install
the Operator into a single namespace without any cluster roles.
The Operator installed with this yaml
-
will not install the CRDs
-
will not use WebHooks
-
wil not look-up Node labels for Coherence site and rack configurations
kubectl apply -f https://github.com/oracle/coherence-operator/releases/download/v3.4.3/coherence-operator-restricted.yaml|
Important
|
When installing the Operator in a restricted mode, the CRDs must have already been manually installed into the Kubernetes cluster. |
Although by default the Operator will install its CRDs, they can be manually installed into Kubernetes. This may be required where the Operator is running with restricted permissions as described above.
The Operator release artifacts include small versions of the two CRDs which can be installed with the following commands:
kubectl apply -f https://github.com/oracle/coherence-operator/releases/download/v3.4.3/coherence.oracle.com_coherence_small.yaml
kubectl apply -f https://github.com/oracle/coherence-operator/releases/download/v3.4.3/coherencejob.oracle.com_coherence_small.yamlThe small versions of the CRDs are identical to the full versions but hav a cut down OpenAPI spec with a lot of comments
removed so that the CRDs are small enough to be installed with kubectl apply
When installing with single manifest yaml file, the replica count can be changed by editing the yaml file itself
to change the occurrence of replicas: 3 in the manifest yaml to replicas: 1
For example, this could be done using sed
sed -i -e 's/replicas: 3/replicas: 1/g' coherence-operator.yamlOr on MacOS, where sed is slightly different:
sed -i '' -e 's/replicas: 3/replicas: 1/g' coherence-operator.yamlFor more flexibility but the simplest way to install the Coherence Operator is to use the Helm chart. This ensures that all the correct resources will be created in Kubernetes.
Add the coherence helm repository using the following commands:
helm repo add coherence https://oracle.github.io/coherence-operator/charts
helm repo update|
Note
|
To avoid confusion, the URL https://oracle.github.io/coherence-operator/charts is a Helm repo, it is not a website you open in a browser. You may think we shouldn’t have to say this, but you’d be surprised.
|
Once the Coherence Helm repo has been configured the Coherence Operator can be installed using a normal Helm 3 install command:
helm install \
--namespace <namespace> \ (1)
coherence \ (2)
coherence/coherence-operator-
where
<namespace>is the namespace that the Coherence Operator will be installed into. -
coherenceis the name of this Helm installation.
The Helm chart uses a default Operator image from ghcr.io/oracle/coherence-operator:{operator-version}.
If the image needs to be pulled from a different location (for example an internal registry) then there are two ways to override the default.
Either set the individual image.registry, image.name and image.tag values, or set the whole image name by setting the image value.
For example, if the Operator image has been deployed into a private registry named foo.com but
with the same image name coherence-operator and tag {operator-version} as the default image,
then just the image.registry needs to be specified.
In the example below, the image used to run the Operator will be foo.com/coherence-operator:{operator-version}.
helm install \
--namespace <namespace> \
--set image.registry=foo.com \
coherence-operator \
coherence/coherence-operatorAll three of the image parts can be specified individually using --set options.
In the example below, the image used to run the Operator will
be foo.com/operator:1.2.3.
helm install \
--namespace <namespace> \
--set image.registry=foo.com \
--set image.name=operator \
--set image.tag=1.2.3
coherence-operator \
coherence/coherence-operatorAlternatively, the image can be set using a single image value.
For example, the command below will set the Operator image to images.com/coherence-operator:0.1.2.
helm install \
--namespace <namespace> \
--set image=images.com/coherence-operator:0.1.2 \
coherence-operator \
coherence/coherence-operatorIf the image is to be pulled from a secure repository that requires credentials then the image pull secrets can be specified. See the Kubernetes documentation on Pulling from a Private Registry.
Create a values file that specifies the secrets, for example the private-repo-values.yaml file below:
imagePullSecrets:
- name: registry-secretsNow use that file in the Helm install command:
helm install \
--namespace <namespace> \
-f private-repo-values.yaml (1)
coherence-operator \
coherence/coherence-operator-
the
private-repo-values.yamlvalues fle will be used by Helm to inject the settings into the Operator deployment
Although the imagePullSecrets field in the values file is an array of name to value pairs it is possible to set
these values with the normal Helm --set parameter.
helm install \
--namespace <namespace> \
--set imagePullSecrets[0].name=registry-secrets (1)
coherence-operator \
coherence/coherence-operator-
this creates the same imagePullSecrets as the values file above.
To change the replica count when installing the Operator using Helm, the replicas value can be set.
For example, to change the replica count from 3 to 1, the --set replicas=1 option can be used.
helm install \
--namespace <namespace> \
--set replicas=1
coherence \
coherence/coherence-operatorTo set the watch namespaces when installing with helm set the watchNamespaces value, for example:
helm install \
--namespace <namespace> \
--set watchNamespaces=payments,catalog,customers \
coherence-operator \
coherence/coherence-operatorThe payments, catalog and customers namespaces will be watched by the Operator.
When installing the Operator using the Helm chart, there is a convenience value that can be set if the
Operator should only monitor the same namespace that it is installed into.
By setting the onlySameNamespace value to true the watch namespace will be set to the installation namespace.
If the onlySameNamespace value is set to true then any value set for the watchNamespaces value will be ignored.
For example, the command below will set onlySameNamespace to true, and the Operator will be installed into,
and only monitor the coh-testing namespace.
helm install \
--namespace coh-testing \
--set onlySameNamespace=true \
coherence-operator \
coherence/coherence-operatorIn the example below, the onlySameNamespace is set to true, so the Operator will be installed into,
and only monitor the coh-testing namespace. Even though the watchNamespaces value is set, it will be ignored.
helm install \
--namespace coh-testing \
--set watchNamespaces=payments,catalog,customers \
--set onlySameNamespace=true \
coherence-operator \
coherence/coherence-operatorThe Operator container can be configured with a Pod securityContext or a container securityContext,
so that it runs as a non-root user.
This can be done using a values file:
Set the Pod securityContext
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000Set the Container securityContext
securityContext:
runAsNonRoot: true
runAsUser: 1000Then the security-values.yaml values file above can be used in the Helm install command.
helm install \
--namespace <namespace> \
--values security-values.yaml \
coherence \
coherence/coherence-operatorAlternatively, the Pod or container securityContext values can be set on the command line as --set parameters:
Set the Pod securityContext
helm install \
--namespace <namespace> \
--set podSecurityContext.runAsNonRoot=true \
--set podSecurityContext.runAsUser=1000 \
coherence \
coherence/coherence-operatorSet the Container securityContext
helm install \
--namespace <namespace> \
--set securityContext.runAsNonRoot=true \
--set securityContext.runAsUser=1000 \
coherence \
coherence/coherence-operatorWhen installing the Operator with Helm, it is possible to set additional labels to be applied to the Operator Pods and to the Operator Deployment.
To add labels to the Operator Pods set the labels value, either on the command line using --set or in the values file.
|
Note
|
Setting |
For example, using the command line:
helm install \
--namespace <namespace> \
--set labels.one=value-one \
--set labels.two=value-two \
coherence \
coherence/coherence-operatorThe command above would add the following additional labels one and two to the Operator Pod as shown below:
apiVersion: v1
kind: Pod
metadata:
name: coherence-operator
labels:
one: value-one
two: value-twoThe same labels could also be specified in a values file:
labels:
one: value-one
two: value-twoTo add labels to the Operator Deployment set the deploymentLabels value, either on the command line using --set or in the values file.
|
Note
|
Setting |
For example, using the command line:
helm install \
--namespace <namespace> \
--set deploymentLabels.one=value-one \
--set deploymentLabels.two=value-two \
coherence \
coherence/coherence-operatorThe command above would add the following additional labels one and two to the Operator Pod as shown below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: coherence-operator
labels:
one: value-one
two: value-twoThe same labels could also be specified in a values file:
deploymentLabels:
one: value-one
two: value-twoWhen installing the Operator with Helm, it is possible to set additional annotations to be applied to the Operator Pods and to the Operator Deployment.
To add annotations to the Operator Pods set the annotations value, either on the command line using --set or in the values file.
|
Note
|
Setting |
For example, using the command line:
helm install \
--namespace <namespace> \
--set annotations.one=value-one \
--set annotations.two=value-two \
coherence \
coherence/coherence-operatorThe command above would add the following additional annotations one and two to the Operator Pod as shown below:
apiVersion: v1
kind: Pod
metadata:
name: coherence-operator
annotations:
one: value-one
two: value-twoThe same annotations could also be specified in a values file:
annotations:
one: value-one
two: value-twoTo add annotations to the Operator Deployment set the deploymentAnnotations value, either on the command line using --set or in the values file.
|
Note
|
Setting |
For example, using the command line:
helm install \
--namespace <namespace> \
--set deploymentAnnotations.one=value-one \
--set deploymentAnnotations.two=value-two \
coherence \
coherence/coherence-operatorThe command above would add the following additional annotations one and two to the Operator Pod as shown below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: coherence-operator
annotations:
one: value-one
two: value-twoThe same annotations could also be specified in a values file:
deploymentAnnotations:
one: value-one
two: value-twoBy default, the Operator will install both CRDs, Coherence and CoherenceJob.
If support for CoherenceJob is not required then it can be excluded from being installed setting the
Operator command line parameter --install-job-crd to false.
When installing with Helm, the allowCoherenceJobs value can be set to false to disable support for CoherenceJob
resources (the default value is true).
helm install \
--namespace <namespace> \
--set allowCoherenceJobs=false \
coherence \
coherence/coherence-operatorIf you want to use yaml directly to install the operator, with something like kubectl, you can use the manifest files
published with the GitHub release at this link:
3.3.5 Manifests
These manifest files are for use with a tool called Kustomize, which is built into kubectl
see the documentation here: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
Download the
3.3.5 Manifests
from the release page and unpack the file, which should produce a directory called manifests with a structure like this:
manifests
default
config.yaml
kustomization.yaml
manager
kustomization.yaml
manager.yaml
service.yaml
rbac
coherence_editor_role.yaml
coherence_viewer_role.yaml
kustomization.yaml
leader_election_role.yaml
leader_election_role_binding.yaml
role.yaml
role_binding.yamlThere are two ways to use these manifest files, either install using kustomize or generate the yaml and manually
install with kubectl.
|
Note
|
All the commands below are run from a console in the manifests/ directory from the extracted file above.
|
If you have Kustomize installed (or can install it from https://github.com/kubernetes-sigs/kustomize) you can use Kustomize to configure the yaml and install.
To change the replica count using Kustomize a patch file needs to be applied.
The Operator manifests include a patch file, named manager/single-replica-patch.yaml, that changes the replica count from 3 to 1. This patch can be applied with the following Kustomize command.
cd ./manager && kustomize edit add patch \
--kind Deployment --name controller-manager \
--path single-replica-patch.yamlIf you need to use different iamge names from the defaults kustomize can be used to specify different names:
Change the name of the Operator image by running the command below, changing the image name to the registry and image name that you are using for the Operator, for example if you have the images in a custom registry
cd ./manager && kustomize edit set image controller=myregistry/coherence-operator:3.3.5Change the name of the Operator image by running the command below, changing the image name to the registry and image name that you are using for the Operator utilities image
cd ./manager && kustomize edit add configmap env-vars --from-literal OPERATOR_IMAGE=myregistry/coherence-operator:3.3.5Change the name of the default Coherence image. If you are always going to be deploying your own application images then this does not need to change.
cd ./manager && $(GOBIN)/kustomize edit add configmap env-vars --from-literal COHERENCE_IMAGE=$(COHERENCE_IMAGE)Set the namespace to install into, the example below sets the namespace to coherence-test:
cd ./default && /kustomize edit set namespace coherence-testThe Operator requires a Secret for its web-hook certificates. This Secret needs to exist but can be empty.
The Secret must be in the same namespace that the Operator will be deployed to.
For example, if the Operator namespace is coherence-test, then the Secret can be created with this command:
kubectl -n coherence-test create secret generic coherence-webhook-server-certThe Operator can now be installed by running the following command from the manifests directory:
kustomize build ./default | kubectl apply -f -Instead of using Kustomize to modify and install the Operator we can use kubectl to generate the yaml from the manifests.
You can then edit this yaml and manually deploy it with kubectl.
Run the following command from the manifests directory:
kubectl create --dry-run -k default/ -o yaml > operator.yamlThis will create a file in the manifests directory called operator.yaml that contains all the yaml required
to install the Operator. You can then edit this yaml to change image names or add other settings.
The Operator can be installed using the generated yaml.
For example if the Operator is to be deployed to the coherence-test namespace:
kubectl -n coherence-test create secret generic coherence-webhook-server-cert
kubectl -n coherence-test create -f operator.yamlIf using VMWare Tanzu the Coherence Operator can be installed as a package. Under the covers, Tanzu uses the Carvel tool set to deploy packages. The Carvel tools can be used outside Tanzu, so the Coherence Operator repo and package images could also be deployed using a standalone Carvel kapp-controller.
The Coherence Operator release published two images required to deploy the Operator as a Tanzu package.
-
ghcr.io/oracle/coherence-operator-package:{operator-version}- the Coherence Operator package -
ghcr.io/oracle/coherence-operator-repo:{operator-version}- the Coherence Operator repository
The first step to deploy the Coherence Operator package in Tanzu is to add the repository. This can be done using the Tanzu CLI.
tanzu package repository add coherence-repo \
--url ghcr.io/oracle/coherence-operator-repo:3.3.5 \
--namespace coherence \
--create-namespaceThe installed repositories can be listed using the CLI:
tanzu package repository list --namespace coherencewhich should display something like the following
NAME REPOSITORY TAG STATUS DETAILS
coherence-repo ghcr.io/oracle/coherence-operator-repo 1h Reconcile succeededThe available packages in the Coherence repository can also be displayed using the CLI
tanzu package available list --namespace coherencewhich should include the Operator package, coherence-operator.oracle.github.com something like the following
NAME DISPLAY-NAME SHORT-DESCRIPTION LATEST-VERSION
coherence-operator.oracle.github.com Oracle Coherence Operator A Kubernetes operator for managing Oracle Coherence clusters 3.3.5Once the Coherence Operator repository has been installed, the coherence-operator.oracle.github.com package can be installed, which will install the Coherence Operator itself.
tanzu package install coherence \
--package-name coherence-operator.oracle.github.com \
--version 3.3.5 \
--namespace coherenceThe Tanzu CLI will display the various steps it is going through to install the package and if all goes well, finally display Added installed package 'coherence'
The packages installed in the coherence namespace can be displayed using the CLI.
tanzu package installed list --namespace coherencewhich should display the Coherence Operator package.
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
coherence coherence-operator.oracle.github.com 3.3.5 Reconcile succeededThe Operator is now installed and ready to mage Coherence clusters.