Skip to content

Commit 664f95f

Browse files
jshum2479robertpatrick
jshum2479
authored and
robertpatrick
committed
add and clarify disaster recovery steps for dopv
1 parent 89f3fb3 commit 664f95f

File tree

7 files changed

+247
-61
lines changed

7 files changed

+247
-61
lines changed

‎documentation/domains/Domain.json

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@
177177
"type": "object",
178178
"properties": {
179179
"initializeDomainOnPV": {
180-
"description": "Configuration to initialize a WebLogic Domain on persistent volume (`Domain on PV`) and initialize related resources such as a persistent volume and a persistent volume claim. If specified, the operator will perform these one-time initialization steps only if the domain and resources do not already exist. The operator will not recreate or update the domain and resources when they already exist. For more information, see https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/choosing-a-model/ and https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/domain-on-pv-initialization ",
180+
"description": "Configuration to initialize a WebLogic Domain on persistent volume (`Domain on PV`) and initialize related resources such as a persistent volume and a persistent volume claim. If specified, the operator will perform these one-time initialization steps only if the domain and resources do not already exist. The operator will not recreate or update the domain and resources when they already exist. For more information, see https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/choosing-a-model/ and https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/domain-on-pv ",
181181
"$ref": "#/definitions/InitializeDomainOnPV"
182182
},
183183
"overrideDistributionStrategy": {
@@ -190,7 +190,8 @@
190190
]
191191
},
192192
"opss": {
193-
"description": "Settings for OPSS security.",
193+
"deprecated": "true",
194+
"description": "Settings for OPSS security for the Model in Image JRF Domain. This field is deprecated, and will be removed in a future release. For JRF domain on PV initialization, use `configuration.initializeDomainOnPV.domain.opss` section for configuring OPSS security settings.",
194195
"$ref": "#/definitions/Opss"
195196
},
196197
"model": {
@@ -822,11 +823,11 @@
822823
"type": "object",
823824
"properties": {
824825
"walletFileSecret": {
825-
"description": "Name of a Secret containing the OPSS key wallet file, which must be in a key named `walletFile`. Use this to allow a JRF domain to reuse its entries in the RCU database. This allows you to specify a wallet file that was obtained from the domain home after the domain was booted for the first time.",
826+
"description": "Name of a Secret containing the OPSS key wallet file, which must be in a key named `walletFile`. Use this to allow a JRF domain to reuse its schemas in the RCU database. This allows you to specify a wallet file that was obtained from the domain home after the domain was booted for the first time.",
826827
"type": "string"
827828
},
828829
"walletPasswordSecret": {
829-
"description": "Name of a Secret containing the OPSS key passphrase, which must be in a key named `walletPassword`. Used to encrypt and decrypt the wallet that is used for accessing the domain\u0027s entries in its RCU database.The password must have a minimum length of eight characters and contain alphabetic characters combined with numbers or special characters.",
830+
"description": "Name of a Secret containing the OPSS key passphrase, which must be in a key named `walletPassword`. Used to encrypt and decrypt the wallet that is used for accessing the domain\u0027s schemas in its RCU database. The password must have a minimum length of eight characters and contain alphabetic characters combined with numbers or special characters.",
830831
"type": "string"
831832
}
832833
}

‎documentation/domains/Domain.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,10 @@ The current status of the operation of the WebLogic domain. Updated automaticall
8383

8484
| Name | Type | Description |
8585
| --- | --- | --- |
86-
| `initializeDomainOnPV` | [Initialize Domain On PV](#initialize-domain-on-pv) | Configuration to initialize a WebLogic Domain on persistent volume (`Domain on PV`) and initialize related resources such as a persistent volume and a persistent volume claim. If specified, the operator will perform these one-time initialization steps only if the domain and resources do not already exist. The operator will not recreate or update the domain and resources when they already exist. For more information, see https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/choosing-a-model/ and https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/domain-on-pv-initialization |
86+
| `initializeDomainOnPV` | [Initialize Domain On PV](#initialize-domain-on-pv) | Configuration to initialize a WebLogic Domain on persistent volume (`Domain on PV`) and initialize related resources such as a persistent volume and a persistent volume claim. If specified, the operator will perform these one-time initialization steps only if the domain and resources do not already exist. The operator will not recreate or update the domain and resources when they already exist. For more information, see https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/choosing-a-model/ and https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/domain-on-pv |
8787
| `introspectorJobActiveDeadlineSeconds` | integer | The introspector job timeout value in seconds. If this field is specified, then the operator's ConfigMap `data.introspectorJobActiveDeadlineSeconds` value is ignored. Defaults to 120 seconds. |
8888
| `model` | [Model](#model) | Model in image model files and properties. |
89-
| `opss` | [Opss](#opss) | Settings for OPSS security. |
89+
| `opss` | [Opss](#opss) | Settings for OPSS security for the Model in Image JRF Domain. This field is deprecated, and will be removed in a future release. For JRF domain on PV initialization, use `configuration.initializeDomainOnPV.domain.opss` section for configuring OPSS security settings. |
9090
| `overrideDistributionStrategy` | string | Determines how updated configuration overrides are distributed to already running WebLogic Server instances following introspection when the `domainHomeSourceType` is PersistentVolume or Image. Configuration overrides are generated during introspection from Secrets, the `overridesConfigMap` field, and WebLogic domain topology. Legal values are `Dynamic`, which means that the operator will distribute updated configuration overrides dynamically to running servers, and `OnRestart`, which means that servers will use updated configuration overrides only after the server's next restart. The selection of `OnRestart` will not cause servers to restart when there are updated configuration overrides available. See also `domains.spec.introspectVersion`. Defaults to `Dynamic`. |
9191
| `overridesConfigMap` | string | The name of the ConfigMap for WebLogic configuration overrides. |
9292
| `secrets` | Array of string | A list of names of the Secrets for WebLogic configuration overrides or model. |
@@ -245,8 +245,8 @@ The current status of the operation of the WebLogic domain. Updated automaticall
245245

246246
| Name | Type | Description |
247247
| --- | --- | --- |
248-
| `walletFileSecret` | string | Name of a Secret containing the OPSS key wallet file, which must be in a key named `walletFile`. Use this to allow a JRF domain to reuse its entries in the RCU database. This allows you to specify a wallet file that was obtained from the domain home after the domain was booted for the first time. |
249-
| `walletPasswordSecret` | string | Name of a Secret containing the OPSS key passphrase, which must be in a key named `walletPassword`. Used to encrypt and decrypt the wallet that is used for accessing the domain's entries in its RCU database.The password must have a minimum length of eight characters and contain alphabetic characters combined with numbers or special characters. |
248+
| `walletFileSecret` | string | Name of a Secret containing the OPSS key wallet file, which must be in a key named `walletFile`. Use this to allow a JRF domain to reuse its schemas in the RCU database. This allows you to specify a wallet file that was obtained from the domain home after the domain was booted for the first time. |
249+
| `walletPasswordSecret` | string | Name of a Secret containing the OPSS key passphrase, which must be in a key named `walletPassword`. Used to encrypt and decrypt the wallet that is used for accessing the domain's schemas in its RCU database. The password must have a minimum length of eight characters and contain alphabetic characters combined with numbers or special characters. |
250250

251251
### Introspector Job Pod
252252

‎documentation/site/content/managing-domains/domain-on-pv/usage.md

Lines changed: 103 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -238,15 +238,112 @@ spec:
238238
| `createIfNotExists` | Specifies whether the operator should create the RCU schema first, before creating the domain. | `domain` or `domainAndRCU` (drop existing RCU schema and create new RCU schema) | N (default `domain`) |
239239
| `domainCreationImages` | WDT domain images. | An array of images. | Y |
240240
| `domainCreationConfigMap` | Optional ConfigMap containing extra WDT models. | Kubernetes ConfigMap name. | N |
241-
| `osss.walletPasswordSecret` | Password for extracting OPSS wallet encryption key for JRF domain. | Kubernetes secret name with key `walletPassword`. | Y |
242-
| `osss.walletFileSecret` | Extracted OPSS wallet file. | Kubernetes secret name with key `walletFile`. | N (Only needed when recreating the domain during disaster recovery) |
241+
| `osss.walletPasswordSecret` | Password for extracting OPSS wallet encryption key for JRF domain. | Kubernetes Secret name with the key `walletPassword`. | Y |
242+
| `osss.walletFileSecret` | Extracted OPSS wallet file. | Kubernetes Secret name with the key `walletFile`. | N (Only needed when recreating the domain during disaster recovery) |
243243
244-
**After a JRF domain is successfully deployed**: follow the next section [Best Practices](#best-practices) to download and back up the OPSS wallet.
244+
**After a JRF domain is successfully deployed**: follow the next section, [Best practices](#best-practices), to download and back up the OPSS wallet.
245245
246-
### Best Practices
246+
### Best practices
247247
248-
For a JRF domain, it is a best practice to download and save the OPSS wallet in a safely backed-up location _immediately_. It is also highly recommended that you store this wallet in
249-
a Kubernetes secret in the same namespace. In case you need to recover the domain directory, then the secret will be ready. For details, see [Disaster Recovery]({{< relref "/managing-domains/working-with-wdt-models/jrf-domain#disaster-recovery-when-the-domain-home-directory-is-destroyed">}}).
248+
Oracle recommends that you save the OPSS wallet file in a safe, backed-up location __immediately__ after an initial JRF domain is created.
249+
In addition, you should make sure to store the wallet in a Kubernetes Secret in the same namespace. This will allow the secret to be available when the domain needs to be recovered in a disaster scenario or if the domain directory gets corrupted. There is no way to reuse the original RCU schema without this specific wallet key.
250+
Therefore, for disaster recovery, **you should back up this encryption key**.
251+
252+
253+
#### Back up the JRF domain home directory and database
254+
255+
Oracle recommends that you back up the domain home directory and database after the initial JRF domain is created, and then periodically, making sure all the latest changes are backed up.
256+
257+
A JRF domain has a one-to-one relationship with the RCU schema. After a domain is created using a particular RCU schema,
258+
that schema _cannot_ be reused by another domain and the same schema _cannot_ be shared across different domains. Any attempts to
259+
create a new domain using the existing RCU schema will result in an error.
260+
261+
{{% notice warning %}}
262+
If the domain home is not properly backed up, you potentially can lose existing data if the domain home is corrupted or deleted.
263+
That's because recreating the domain requires dropping the existing RCU schema and creating a new RCU schema. Therefore, backing up the existing domain home should be
264+
the highest priority in your Kubernetes environment.
265+
{{% /notice %}}
266+
267+
A Domain on PV domain configuration might get updated after its initial deployment. For example, using WLST or the console, you might have deployed
268+
new applications, added custom OPSS keystores, and added OWSM policies, and such.
269+
In that case, the original WDT model files used to create the initial domain will _not_ match the current state of the domain (the WDT model files are _not_ the source of truth).
270+
Therefore, if you recreate the domain using the original WDT model files, you will lose all the subsequent updates. In order to preserve the domain updates, you should restore the domain from
271+
the backup copy of the domain home directory and connect to the existing RCU schema from the database backup.
272+
273+
#### Store the OPSS wallet in a Kubernetes Secret and update `opss.walletFileSecret` in the domain resource
274+
After the domain is created, the operator automatically exports the OPSS wallet and stores it in an introspector ConfigMap; the name of the ConfigMap follows the pattern `<domain uid>-weblogic-domain-introspect-cm` with the key `ewallet.p12`. Oracle recommends that you save the OPSS wallet file in a safe, backed-up location __immediately__ after an initial JRF domain is created. In addition, you should make sure to store the wallet in a Kubernetes Secret in the same namespace. This will allow the secret to be available when the domain needs to be recovered in a disaster scenario or if the domain directory gets corrupted.
275+
276+
The following are the high-level steps for storing the OPSS wallet in a Kubernetes Secret.
277+
1. The operator provides a utility script, [OPSS wallet utility](https://orahub.oci.oraclecorp.com/weblogic-cloud/weblogic-kubernetes-operator/-/blob/main/kubernetes/samples/scripts/domain-lifecycle/opss-wallet.sh), for extracting the wallet file and storing it in a Kubernetes `walletFileSecret`. In addition, you should also save the wallet file in a safely backed-up location outside of Kubernetes. For example, the following command saves the OPSS wallet for the `sample-domain1` domain in the `sample-ns` namespace to a file named `ewallet.p12` in the `/tmp` directory and also stores it in the wallet secret named `jrf-wallet-file-secret`.
278+
279+
```
280+
$ opss-wallet.sh -n sample-ns -d sample-domain1 -s -r -wf /tmp/ewallet.p12 -ws jrf-wallet-file-secret
281+
```
282+
283+
Replace `/tmp/ewallet.p12` in the previous command with the name of the file to be stored in a safe location and replace `jrf-wallet-file-secret` with the Secret name of your choice. For more information, see the `OPSS wallet utility` section in the [README](https://github.com/oracle/weblogic-kubernetes-operator/tree/{{< latestMinorVersion >}}/kubernetes/samples/scripts/domain-lifecycle/README.md) file.
284+
285+
2. Save the extracted wallet file (`/tmp/ewallet.p12` in the example) to a safe location, outside of Kubernetes.
286+
3. Add the `opss.walletFileSecret` to the domain resource YAML file under `configuration.initializeDomainOnPV.domain`.
287+
288+
```
289+
...
290+
configuration:
291+
initializeDomainOnPV:
292+
...
293+
domain:
294+
...
295+
opss:
296+
walletFileSecret: jrf-wallet-file-secret
297+
...
298+
```
299+
300+
You can use the following `patch` command to add it to the domain resource.
301+
302+
```
303+
$ kubectl -n sample-ns patch domain sample-domain1 --type='JSON' -p='[ { "op" : "add", "path" : "/spec/configuration/initializeDomainOnPV/domain/opss/walletFileSecret", "value" : "jrf-wallet-file-secret" }]'
304+
```
305+
306+
#### Recovering the domain when it's corrupted or in other disaster scenarios
307+
308+
If the domain home directory is corrupted, and you have a recent backup of the domain home directory, then perform the following steps to recover the domain.
309+
310+
1. Restore the domain home directory from the backup copy.
311+
2. Update the `restartVersion` of the domain resource to restart the domain. For example,
312+
313+
```
314+
$ kubectl -n sample-ns patch domain sample-domain1 --type='JSON' -p='[ { "op" : "replace", "path" : "/spec/restartVersion", "value" : "15" }]'
315+
```
316+
3. After the domain is restarted, check the WebLogic domain configuration to ensure that it has the latest changes.
317+
**Note:** If you made any changes that are persisted in the domain home directory after your last backup, you must reapply those changes to the domain home directory.
318+
However, because the operator will reconnect to the same RCU schema, the data stored in the OPSS, MDS, or OWSM tables will be current.
319+
320+
4. Reapply any domain configuration changes persisted to the domain home directory, such as
321+
data source connections, JMS destinations, or new application EAR deployments, after your last backup. To make these changes, use WLST, the WebLogic Server Administration Console, or Enterprise Manager.
322+
323+
In the rare scenario where the domain home directory is corrupted, and you do **not** have a recent backup of the domain home directory, or if the backup copy is also corrupted, then you can recreate the domain from the WDT model files without losing any RCU schema data.
324+
325+
1. Delete the domain home directory from the persistent volume.
326+
2. Add or replace the `domain.spec.introspectVersion` in the domain resource with a new value. The following is a sample `patch` command to update the `introspectVersion` for the sample domain.
327+
328+
```
329+
$ kubectl -n sample-ns patch domain sample-domain1 --type='JSON' -p='[ { "op" : "replace", "path" : "/spec/intropsectVersion", "value" : "15" }]'
330+
```
331+
332+
3. The operator will then create a new domain from the existing WDT models and reuse the original RCU schema.
333+
334+
**NOTE:**
335+
All the updates made to the domain after the initial deployment will **not** be available in the recovered domain.
336+
However, this allows you to access the original RCU schema database without losing all its data.
337+
338+
4. Apply all the domain configuration changes persisted to the domain home file system, such as data source connections, JMS destinations, or new application EAR deployments, that are not in the WDT model files. These are the changes you have made _after_ the initial domain deployment.
339+
340+
5. Update the `restartVersion` of the domain resource to restart the domain.
341+
342+
```
343+
$ kubectl -n sample-ns patch domain sample-domain1 --type='JSON' -p='[ { "op" : "replace", "path" : "/spec/restartVersion", "value" : "15" }]'
344+
```
345+
346+
For more information, see [Disaster Recovery]({{< relref "/managing-domains/working-with-wdt-models/jrf-domain#disaster-recovery-for-domain-on-pv-deployment">}}).
250347
251348
### Troubleshooting
252349

0 commit comments

Comments
 (0)