Skip to content

Commit 302958c

Browse files
lennyphanrjeberhardTom Barnesankediarosemarymarano
authored
OWLS-103562: WKO 4.0: Elasticity - documentation (#3569)
* WKO 4.0: Elasticity - documentation Co-authored-by: Ryan Eberhard <[email protected]> Co-authored-by: Tom Barnes <[email protected]> Co-authored-by: Anil Kedia <[email protected]> Co-authored-by: Rosemary Marano <[email protected]>
1 parent 9f10527 commit 302958c

File tree

2 files changed

+118
-40
lines changed
  • documentation/staging/content/managing-domains/domain-lifecycle
  • kubernetes/samples/scripts/domain-lifecycle

2 files changed

+118
-40
lines changed

‎documentation/staging/content/managing-domains/domain-lifecycle/scaling.md

Lines changed: 113 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -10,44 +10,81 @@ This document describes approaches for scaling WebLogic clusters in a Kubernetes
1010

1111
{{< table_of_contents >}}
1212

13-
#### Overview
13+
### Overview
1414

15-
WebLogic Server supports two types of clustering configurations, configured and dynamic. Configured clusters are created by defining each individual Managed Server instance. In dynamic clusters, the Managed Server configurations are generated from a single, shared template.  With dynamic clusters, when additional server capacity is needed, new server instances can be added to the cluster without having to configure them individually. Also, unlike configured clusters, scaling up of dynamic clusters is not restricted to the set of servers defined in the cluster but can be increased based on runtime demands. For more information on how to create, configure, and use dynamic clusters in WebLogic Server, see [Dynamic Clusters](https://docs.oracle.com/en/middleware/standalone/weblogic-server/14.1.1.0/clust/dynamic_clusters.html#GUID-DA7F7FAD-49AA-4F3D-8A05-0D9921B96971).
15+
WebLogic Server supports two types of clustering configurations, configured and dynamic. Configured clusters are created by defining each individual Managed Server instance. In dynamic clusters, the Managed Server configurations are generated from a single, shared template. With dynamic clusters, when additional server capacity is needed, new server instances can be added to the cluster without having to configure them individually. Also, unlike configured clusters, scaling up of dynamic clusters is not restricted to the set of servers defined in the cluster but can be increased based on runtime demands. For more information on how to create, configure, and use dynamic clusters in WebLogic Server, see [Dynamic Clusters](https://docs.oracle.com/en/middleware/standalone/weblogic-server/14.1.1.0/clust/dynamic_clusters.html#GUID-DA7F7FAD-49AA-4F3D-8A05-0D9921B96971).
1616

17-
For more in-depth information on support for scaling WebLogic clusters in Kubernetes, see [WebLogic Dynamic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/weblogic-dynamic-clusters-on-kubernetes).
17+
WebLogic Kubernetes Operator 4.0 introduces a new custom resource, `Cluster`. A [Cluster resource]({{< relref "/managing-domains/domain-resource.md" >}}) references an individual WebLogic cluster and its configuration. It also controls how many member servers are running, along with potentially any additional Kubernetes resources that are specific to that WebLogic cluster. Because the Cluster resource enables the Kubernetes [Scale subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), the `kubectl scale` command and the [Kubernetes Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) are fully supported for scaling individual WebLogic clusters.
1818

1919
The operator provides several ways to initiate scaling of WebLogic clusters, including:
2020

21-
* [On-demand, updating the Domain directly (using `kubectl`)](#on-demand-updating-the-domain-directly).
22-
* [Calling the operator's REST scale API, for example, from `curl`](#calling-the-operators-rest-scale-api).
23-
* [Using a WLDF policy rule and script action to call the operator's REST scale API](#using-a-wldf-policy-rule-and-script-action-to-call-the-operators-rest-scale-api).
24-
* [Using a Prometheus alert action to call the operator's REST scale API](#using-a-prometheus-alert-action-to-call-the-operators-rest-scale-api).
21+
* [`kubectl` CLI commands](#kubectl-cli-commands)
22+
* [On-demand, updating the Cluster or Domain directly](#on-demand-updating-the-cluster-or-domain-directly) (using `kubectl`)
23+
* [Using Domain lifecycle sample scripts](#using-domain-lifecycle-sample-scripts)
24+
* [Calling the operator's REST scale API, for example, from `curl`](#calling-the-operators-rest-scale-api)
25+
* [Kubernetes Horizontal Pod Autoscalar (HPA)](#kubernetes-horizontal-pod-autoscalar-hpa)
26+
* [Using a WLDF policy rule and script action to call the operator's REST scale API](#using-a-wldf-policy-rule-and-script-action-to-call-the-operators-rest-scale-api)
27+
* [Using a Prometheus alert action to call the operator's REST scale API](#using-a-prometheus-alert-action-to-call-the-operators-rest-scale-api)
2528

26-
#### On-demand, updating the Domain directly
27-
The easiest way to scale a WebLogic cluster in Kubernetes is to simply edit the `replicas` field of a Domain. This can be done by using `kubectl`. More specifically, you can modify the Domain directly by using the `kubectl edit` command. For example:
29+
### `kubectl` CLI commands
30+
Use the Kubernetes command-line tool, `kubectl`, to manually scale WebLogic clusters with the following commands:
31+
32+
* `scale` - Set a new size for a resource.
33+
* `patch` - Update a field or fields of a resource using a strategic merge patch.
34+
35+
#### `kubectl scale` command
36+
You can scale an individual WebLogic cluster directly using the `kubectl scale` command. For example, the following command sets `.spec.replicas` of the Cluster resource `cluster-1` to `5`:
2837
```shell
29-
$ kubectl edit domain domain1 -n [namespace]
38+
$ kubectl scale --replicas=5 clusters/cluster-1
39+
clusters "cluster-1" scaled
40+
41+
$ kubectl get clusters cluster-1 -o jsonpath='{.spec.replicas}'
42+
5
43+
```
44+
45+
#### `kubectl patch` command
46+
Also, you can scale an individual WebLogic cluster directly using the `kubectl patch` command with your patch object inline. For example, the following command sets `.spec.replicas` of the Cluster resource `cluster-1` to `3`:
47+
```shell
48+
$ kubectl patch cluster cluster-1 --type=merge -p '{"spec":{"replicas":3}}'
49+
cluster.weblogic.oracle/cluster-1 patched
50+
51+
$ kubectl get clusters cluster-1 -o jsonpath='{.spec.replicas}'
52+
3
53+
```
54+
55+
### On-demand, updating the Cluster or Domain directly
56+
You can use on-demand scaling (scaling a cluster manually) by directly updating the `.spec.replicas` field of the Cluster or Domain resources.
57+
#### Updating the Cluster directly
58+
To scale an individual WebLogic cluster directly, simply edit the `replicas` field of its associated Cluster resource; you can do this using `kubectl`. More specifically, you can modify the Cluster directly by using the `kubectl edit` command. For example:
59+
```shell
60+
$ kubectl edit cluster cluster-1 -n [namespace]
3061
```
31-
Here we are editing a Domain named `domain1`. The `kubectl edit` command will open the Domain definition in an editor and allow you to modify the `replicas` value directly. Once committed, the operator will be notified of the change and will immediately attempt to scale the corresponding cluster by reconciling the number of running pods/Managed Server instances with the `replicas` value specification.
62+
In this command, you are editing a Cluster named `cluster-1`. The `kubectl edit` command opens the Cluster definition in an editor and lets you modify the `replicas` value directly. Once committed, the operator will be notified of the change and will immediately attempt to scale the corresponding cluster by reconciling the number of running pods/Managed Server instances with the `replicas` value specification.
3263
```yaml
3364
spec:
3465
...
35-
clusters:
36-
- clusterName: cluster-1
37-
replicas: 1
66+
clusterName: cluster-1
67+
replicas: 1
3868
...
3969
```
70+
#### Updating the Domain directly
71+
To scale every WebLogic cluster in a domain that does not have a corresponding Cluster resource, or that has a Cluster resource which does not specify a `cluster.spec.replicas` field, modify the `domain.spec.replicas` field using the `kubectl edit` command:
4072

41-
Alternatively, you can specify a default `replicas` value for all the clusters. If you do this, then you don't need to list the cluster in the Domain (unless you want to customize another property of the cluster).
73+
```shell
74+
$ kubectl edit domain domain1 -n [namespace]
75+
```
76+
In this command, you are editing a Domain named `domain1`. The `kubectl edit` command opens the Domain definition in an editor and lets you modify the `replicas` value directly. Once committed, the operator will be notified of the change and will immediately attempt to scale the corresponding cluster or clusters by reconciling the number of running pods/Managed Server instances with the `replicas` value specification.
4277
```yaml
4378
spec:
4479
...
4580
replicas: 1
4681
...
4782
```
48-
In addition, see the helper scripts in the [Domain lifecycle sample scripts]({{< relref "/managing-domains/domain-lifecycle/startup#domain-lifecycle-sample-scripts" >}}) section.
4983

50-
#### Calling the operator's REST scale API
84+
### Using Domain lifecycle sample scripts
85+
Beginning in version 3.1.0, the operator provides sample lifecycle scripts. See the helper scripts in the [Domain lifecycle sample scripts]({{< relref "/managing-domains/domain-lifecycle/scripts.md" >}}) section, which you can use for scaling WebLogic clusters.
86+
87+
### Calling the operator's REST scale API
5188

5289
Scaling up or scaling down a WebLogic cluster provides increased reliability of customer applications as well as optimization of resource usage. In Kubernetes environments, scaling WebLogic clusters involves scaling the number of corresponding Pods in which Managed Server instances are running. Because the operator manages the life cycle of a WebLogic domain, the operator exposes a REST API that allows an authorized actor to request scaling of a WebLogic cluster.
5390

@@ -56,7 +93,6 @@ The following URL format is used for describing the resources for scaling (up an
5693
```
5794
http(s)://${OPERATOR_ENDPOINT}/operator/<version>/domains/<domainUID>/clusters/<clusterName>/scale
5895
```
59-
6096
For example:
6197

6298
```
@@ -85,7 +121,7 @@ The `replicas` value designates the number of Managed Server instances to scale
85121

86122
When you POST to the `/scale` REST endpoint, you must send the following headers:
87123

88-
* `X-Requested-By` request value. The value is an arbitrary name such as `MyClient`.
124+
* `X-Requested-By` request value. The value is an arbitrary name such as `MyClient`.
89125
* `Authorization: Bearer` request value. The value of the `Bearer` token is the WebLogic domain service account token.
90126

91127
For example, when using `curl`:
@@ -112,10 +148,10 @@ rules:
112148
verbs: ["get", "list", "watch"]
113149
- apiGroups: ["weblogic.oracle"]
114150
resources: ["domains"]
115-
verbs: ["get", "list", "patch", update"]
151+
verbs: ["get", "list", "patch", "update"]
116152
---
117153
```
118-
##### Operator REST endpoints
154+
#### Operator REST endpoints
119155

120156
The WebLogic Kubernetes Operator can expose both an internal and external REST HTTPS endpoint.
121157
The internal REST endpoint is only accessible from within the Kubernetes cluster. The external REST endpoint
@@ -128,19 +164,62 @@ Detailed instructions for configuring the external REST endpoint are available [
128164
Regardless of which endpoint is being invoked, the URL format for scaling is the same.
129165
{{% /notice %}}
130166

131-
##### What does the operator do in response to a scaling request?
167+
#### What does the operator do in response to a scaling request?
132168

133169
When the operator receives a scaling request, it will:
134170

135-
* Perform an authentication and authorization check to verify that the specified user is allowed to perform the specified operation on the specified resource.
136-
* Validate that the specified domain, identified by `domainUID`, exists.
137-
* Validate that the WebLogic cluster, identified by `clusterName`, exists.
138-
* Verify that the specified WebLogic cluster has a sufficient number of configured servers or sufficient dynamic cluster size to satisfy the scaling request.
139-
* Initiate scaling by setting the `replicas` field within the corresponding Domain, which can be done in either:
140-
* A `cluster` entry, if defined within its cluster list.
141-
* At the domain level, if not defined in a `cluster` entry.
171+
* Perform an authentication and authorization check to verify that the specified user is allowed to perform the specified operation on the specified resource.
172+
* Validate that the specified domain, identified by `domainUID`, exists.
173+
* Validate that the WebLogic cluster, identified by `clusterName`, exists.
174+
* Verify that the specified WebLogic cluster has a sufficient number of configured servers or sufficient dynamic cluster size to satisfy the scaling request.
175+
* Verify that the specified cluster has a corresponding Cluster resource defined or else creates one if necessary.
176+
* Initiate scaling by setting the `replicas` field within the corresponding Cluster resource.
177+
178+
In response to a change in the `replicas` field in the Cluster resource, the operator will increase or decrease the number of Managed Server instance Pods to match the desired replica count.
142179

143-
In response to a change to either `replicas` field, in the Domain, the operator will increase or decrease the number of Managed Server instance Pods to match the desired replica count.
180+
### Supported autoscaling controllers
181+
While continuing to support automatic scaling of WebLogic clusters with the WebLogic Diagnostic Framework (WLDF) and Prometheus, Operator 4.0 now supports the [Kubernetes Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
182+
#### Kubernetes Horizontal Pod Autoscalar (HPA)
183+
Automatic scaling of an individual WebLogic cluster, by the Kubernetes Horizontal Pod Autoscalar, is now supported since the Cluster custom resource has enabled the Kubernetes [/scale subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource/). Autoscaling based on resource metrics requires the installation of the [Kubernetes Metrics Server](https://kubernetes-sigs.github.io/metrics-server/) or another implementation of the resource metrics API. If using Prometheus for monitoring WebLogic Server metrics, then you can use the [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) in place of the [Kubernetes Metrics Server](https://kubernetes-sigs.github.io/metrics-server/).
184+
185+
The following step-by-step example illustrates how to configure and run an HPA to scale a WebLogic cluster, `cluster-1`, based on the `cpu utilization` resource metric.
186+
187+
1. To illustrate scaling of a WebLogic cluster based on CPU utilization, deploy the Kubernetes Metrics Server to the Kubernetes cluster.
188+
```shell
189+
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
190+
```
191+
2. Confirm that the Kubernetes Metric Server is running by listing the pods in the `kube-system` namespace.
192+
```shell
193+
$ kubectl get po -n kube-system
194+
```
195+
{{% notice note %}}
196+
If the Kubernetes Metric Server does not reach the READY state (for example, `READY 0/1` ) due to `Readiness probe failed: HTTP probe failed with statuscode: 500`, then you may need to install a valid cluster certificate. For testing purposes, you can resolve this issue by downloading the `components.yaml` file and adding the argument `--kubelet-insecure-tls` to the Metrics Server container.
197+
{{% /notice %}}
198+
199+
3. Assuming a WebLogic domain running in the default namespace, use the following command to create an HPA resource targeted at the Cluster resource (`sample-domain1-cluster-1`) that will autoscale WebLogic Server instances from a minimum of `2` cluster members up to `5` cluster members, and the scale up or down will occur when the average CPU is consistently over 50%.
200+
```shell
201+
$ kubectl autoscale cluster sample-domain1-cluster-1 --cpu-percent=50 --min=2 --max=5
202+
horizontalpodautoscaler.autoscaling/sample-domain1-cluster-1 autoscaled
203+
```
204+
{{% notice note %}}
205+
Beginning with Operator 4.0, the `allowReplicasBelowMinDynClusterSize` field has been removed from the Domain resource schema. When scaling down, the minimum allowed replica count must now be configured on the selected autoscaling controller.
206+
{{% /notice %}}
207+
208+
4. Verify the status of the autoscaler and its behavior by inspecting the HPA resource.
209+
```shell
210+
$ kubectl get hpa
211+
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
212+
sample-domain1-cluster-1 Cluster/sample-domain1-cluster-1 8%/50% 2 5 2 3m27s
213+
```
214+
215+
5. To see the HPA scale up the WebLogic cluster `sample-domain1-cluster-1`, generate a loaded CPU by getting a shell to a running container in one of the cluster member pods and run the following command.
216+
```shell
217+
$ kubectl exec --stdin --tty sample-domain1-managed-server1 -- /bin/bash
218+
[oracle@sample-domain1-managed-server1 oracle]$ dd if=/dev/zero of=/dev/null
219+
```
220+
6. By listing the Managed Server pods, you will see the autoscaler increase the replicas on the Cluster resource and the operator respond by starting additional cluster member servers. Conversely, after stopping the load and when the CPU utilization average is consistently under 50%, the autoscalar will scale down the WebLogic cluster by decreasing the replicas value on the Cluster resource.
221+
222+
For more in-depth information on the Kubernetes Horizontal Pod Autoscalar, see [Horizontal Pod Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
144223

145224
#### Using a WLDF policy rule and script action to call the operator's REST scale API
146225
The WebLogic Diagnostics Framework (WLDF) is a suite of services and APIs that collect and surface metrics that provide visibility into server and application performance.
@@ -216,7 +295,6 @@ A more in-depth description and example on using WLDF's Policies and Actions com
216295
* [Automatic Scaling of WebLogic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/automatic-scaling-of-weblogic-clusters-on-kubernetes-v2)
217296
* [WebLogic Dynamic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/weblogic-dynamic-clusters-on-kubernetes)
218297

219-
220298
##### Create ClusterRoleBindings to allow a namespace user to query WLS Kubernetes cluster information
221299
The script `scalingAction.sh`, specified in the WLDF script action, needs the appropriate RBAC permissions granted for the service account user (in the namespace in which the WebLogic domain is deployed) to query the Kubernetes API server for both configuration and runtime information of the Domain.
222300
The following is an example YAML file for creating the appropriate Kubernetes ClusterRole bindings:
@@ -236,7 +314,7 @@ rules:
236314
verbs: ["get", "list", "watch"]
237315
- apiGroups: ["weblogic.oracle"]
238316
resources: ["domains"]
239-
verbs: ["get", "list", "patch", update"]
317+
verbs: ["get", "list", "patch", "update"]
240318
---
241319
#
242320
# creating role-bindings for cluster role
@@ -280,8 +358,8 @@ In addition to using the WebLogic Diagnostic Framework for automatic scaling of
280358
you can use a third-party monitoring application like Prometheus. Please read the following blog for
281359
details about [Using Prometheus to Automatically Scale WebLogic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/using-prometheus-to-automatically-scale-weblogic-clusters-on-kubernetes-v5).
282360

283-
#### Helpful tips
284-
##### Debugging scalingAction.sh
361+
### Helpful tips
362+
#### Debugging scalingAction.sh
285363
The [`scalingAction.sh`](https://github.com/oracle/weblogic-kubernetes-operator/blob/{{< latestMinorVersion >}}/operator/scripts/scaling/scalingAction.sh) script was designed to be executed within a container of the
286364
Administration Server Pod because the associated diagnostic module is targeted to the Administration Server.
287365

@@ -302,7 +380,7 @@ $ cd /u01/oracle/user-projects/domains/domain1/bin/scripts && \
302380

303381
A log, `scalingAction.log`, will be generated in the same directory in which the script was executed and can be examined for errors.
304382

305-
##### Example on accessing the external REST endpoint
383+
#### Example on accessing the external REST endpoint
306384
The easiest way to test scaling using the external REST endpoint is to use a command-line tool like `curl`. Using `curl` to issue
307385
an HTTPS scale request requires these mandatory header properties:
308386

0 commit comments

Comments
 (0)