Deploy Zowe in Sysplex

This documentation only covers differences comparing to normal Zowe installation and configuration. Check https://docs.zowe.org/stable/user-guide/install-zos.html before continue.

Download test Zowe build

Download the most recent staging build from 1.22.0-STAGING and install as usual.

Expected result: install the build and start Zowe as usual, the Zowe instance should pass all sanity test.

Pre-reqs

DVIPA configuration for z/OSMF and Zowe. Can follow instructions here TCP/IP definitions for Sysplex Distributor
NODE_HOME must be defined before you run zowe-install.sh

Changes on `ZWESVSTC` (this change has been temporarily removed from PR-2049)

In PR-2049, there is optional change on ZWESVSTC to allow it to start Zowe with different HA instance ID. If you already have ZWESVSTC in your problib, you will need to copy again from SZWESAMP and overwrite the existing version in proclib.

Expected result, you can use zowe-start.sh [ha-instance-id] to start a specific HA instance. [ha-instance-id] is optional and it will be default to SYSNAME system variable where your LPAR is located in Sysplex.

Customize component-level certificates

When you customize zowe-setup-certificates-for-ha.env before running zowe-setup-certificates.sh, you may already notice new variables COMPONENT_LEVEL_CERTIFICATES, EXTERNAL_COMPONENT_CERTIFICATES and EXTERNAL_COMPONENT_CERTIFICATE_ALIASES. Read the instruction, uncomment and modify value of COMPONENT_LEVEL_CERTIFICATES.

To successfully run zowe-setup-certificates.sh on Sysplex, the script may not be able to find your z/OSMF configuration, or fall back to wrong value. To proper define where is z/OSMF, define ZOWE_ZOSMF_HOST and ZOWE_ZOSMF_PORT environment variables. ZOWE_ZOSMF_HOST should be the domain name of your Dynamic VIP Address (DVIPA). In our case, it's zhaplex.svl.ibm.com. For example.

export ZOWE_ZOSMF_HOST=zhaplex.svl.ibm.com
export ZOWE_ZOSMF_PORT=443

Now you can run zowe-setup-certificates.sh again.

Expected result #1, you should see these extra files in <keystore>/localhost/:

localhost.keystore.app-server.cer
localhost.keystore.app-server.cer-ebcdic
localhost.keystore.app-server.csr
localhost.keystore.app-server.key
localhost.keystore.app-server_signed.cer
localhost.keystore.gateway.cer
localhost.keystore.gateway.cer-ebcdic
localhost.keystore.gateway.csr
localhost.keystore.gateway.key
localhost.keystore.gateway_signed.cer
localhost.keystore.zss.cer
localhost.keystore.zss.cer-ebcdic
localhost.keystore.zss.csr
localhost.keystore.zss.key
localhost.keystore.zss_signed.cer

These are the PEM format certificates.

Expected result #2, run below command to list content of the keystore:

$ keytool -v -list -storepass password -storetype PKCS12 -keystore localhost.keystore.p12

You should see certificates generated for each components with proper alias name like this:

Alias name: gateway
Creation date: Apr 7, 2021
Entry type: keyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=Zowe Service - gateway, OU=API Mediation Layer, O=Zowe Sample, L=Prague, ST=Prague, C=CZ

Expected result #3, check <keystore>/zowe-certificates.env, it should show instructions like this:

# To configure certificate for gateway, you can add these entries to "components.gateway" of your YAML configuration:
# certificate:
#   keystore:
#     alias: gateway
#   pem:
#     key: /global/zowe/keystore/localhost/localhost.keystore.gateway.key
#     certificate: /global/zowe/keystore/localhost/localhost.keystore.gateway.cer-ebcdic
# To configure certificate for app-server, you can add these entries to "components.app-server" of your YAML configuration:
# certificate:
#   keystore:
#     alias: app-server
#   pem:
#     key: /global/zowe/keystore/localhost/localhost.keystore.app-server.key
#     certificate: /global/zowe/keystore/localhost/localhost.keystore.app-server.cer-ebcdic
# To configure certificate for zss, you can add these entries to "components.zss" of your YAML configuration:
# certificate:
#   keystore:
#     alias: zss
#   pem:
#     key: /global/zowe/keystore/localhost/localhost.keystore.zss.key
#     certificate: /global/zowe/keystore/localhost/localhost.keystore.zss.cer-ebcdic

The component level certificate instruction will overwrite part of the values defined in zowe.internalCertificate.

Configure Zowe instance

You can run zowe-configure-instance.sh as usual to create Zowe instance directory. For example, zowe-configure-instance.sh -c /global/zowe/instance. The Zowe instance directory must be put on a shared volume where all LPARs in the Sysplex can access. /global/zowe/instance is the suggested location.

Expected result, you will see these extra information introducing YAML configuration. (This change has been temporarily removed from PR-2049)

As technical preview, Zowe now provides a new way to customize your instance with a YAML file.
You can convert your instance.env file by running this script:
  <instance-dir>/bin/utils/convert-to-zowe-yaml.sh > <instance-dir>/zowe.yaml
The zowe.yaml will take effect once you delete or rename your <instance-dir>/instance.env file.
Please check <instance-dir>/bin/example-zowe.yaml and see how you can customize Zowe instance.
This YAML configuration format is mandatory to deploy Zowe in a Parallel Sysplex environment.

Execute <instance-dir>/bin/utils/convert-to-zowe-yaml.sh <instance-dir>/zowe.yaml to generate zowe.yaml.

Note: please be aware of the tagging of zowe.yaml. To allow Zowe to read the file properly, it must be IBM-1047 encoding.

Check the encoding with ls -laT <instance-dir>. zowe.yaml should be untagged and you should be able to view the content of the file without any issue.

Now you can delete/rename your <instance-dir>/instance.env and start Zowe.

You can follow <instance-dir>/bin/example-zowe.yaml haInstances section to define your instances. Also check the example about how to customize component certificate. The original design of YAML config can be found here: https://drive.google.com/file/d/1Z0QSpQ2qc5tTBUQYAVAva7kyKXj4n9V7/view?usp=sharing.

Expected result, Zowe should start as normal without issues.

Configure launcher `ZWESLSTC`

During installation, you may notice lines like this showing on the output:

launcher/install.sh ZWESLSTC copied to <dataset-prefix>.SZWESAMP
launcher/install.sh ZWELNCH program copied to <dataset-prefix>.SZWEAUTH

Follow these steps to enable launcher.

Copy `ZWESLSTC` to proclib

You can find ZWESLSTC in <dataset-prefix>.SZWESAMP. Copy over to your proclib and make sure these values are correct:

STEPLIB DSNAME should point to your <dataset-prefix>.SZWEAUTH. For example, //STEPLIB DD DSNAME=IBMUSER.ZWE122.SZWEAUTH,DISP=SHR.
INSTANCE_DIR=#instance_dir should point to your instance directory. For example, INSTANCE_DIR=/global/zowe/instance.

You can also use utility script to copy proclib:

cd <runtime>/scripts/utils
./zowe-copy-to-JES.sh -s <dataset-prefix>.SZWESAMP -i ZWESLSTC -r USER.PROCLIB -o ZWESLSTC -f /global/zowe/logs/launcher-stc.log

Define started task

Run below command to to customize runtime user for ZWESLSTC and refresh STARTED profile.

RDEFINE STARTED ZWESLSTC.* STDATA(USER(ZWESVUSR) GROUP(ZWEADMIN) TRUSTED(NO)) DATA('ZOWE LAUNCHER')
SETROPTS RACLIST(STARTED) REFRESH

Start Zowe with `ZWESLSTC`

Run this command to start Zowe: S ZWESLSTC,HAINST=<ha-instance-id>. For example, S ZWESLSTC,HAINST=TIVLP14A.

To terminate Zowe, run P ZWESLSTC command.

To stop a component, run F ZWESLSTC,APPL=STOP(<component>) command. For example, F ZWESLSTC,APPL=STOP(files-api). To start a component, run F ZWESLSTC,APPL=START(<component>) command.

Troubleshooting

ZIS status not OK

We need ZWESISTC be started on every LPAR you plan to run ZSS services.

TCPIP resolver failure

If you see these errors in your Zowe job log, that means your TCP resolver is not configured properly. This is important for Sysplex deployment to make sure 2 LPARs can communicate each other with domain name.

EZZ2376I Could not determine TCPIPjobname, using default of 'INET'                                                
EZZ2377I Could not establish affinity with INET (1011/11B3005A) - can not provide the requested option information

To resolve this issue,

if you are using instance.env, you can add line RESOLVER_CONFIG="//'USER.PARMLIB(TCPDATA)'" to it,
if you are using zowe.yaml, you can add entry RESOLVER_CONFIG: "//'USER.PARMLIB(TCPDATA)'" to zowe.environments.

You need to find correct parmlib where your TCPDATA located.

Zombie address spaces

Reason is still unknown. To remove zombie processes after Zowe is stopped,

in USS, issue su and use ps -elf | grep ZWESVUSR | awk '{print $2}' | xargs kill -9 to kill all processes running as ZWESVUSR,
in SDSF.DA panel, find all ZWE address spaces and cancel them.

Deploy Zowe in Sysplex

Download test Zowe build

Pre-reqs

Changes on ZWESVSTC (this change has been temporarily removed from PR-2049)

Customize component-level certificates

Configure Zowe instance

Configure launcher ZWESLSTC

Copy ZWESLSTC to proclib

Define started task

Start Zowe with ZWESLSTC

Troubleshooting

ZIS status not OK

TCPIP resolver failure

Zombie address spaces

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

System Squad

Zowe Builds

Bill of Materials

SMPE

PSWI

Major Features

v2

High available and Sysplex

Containerization

CI/CD and Automation

Clone this wiki locally

Changes on `ZWESVSTC` (this change has been temporarily removed from PR-2049)

Configure launcher `ZWESLSTC`

Copy `ZWESLSTC` to proclib

Start Zowe with `ZWESLSTC`