Skip to content

Commit fc1b178

Browse files
andrewhughes101andrewhughes101
committed
Add Developer guide and Architecture document
1 parent 6bff921 commit fc1b178

File tree

3 files changed

+837
-1
lines changed

3 files changed

+837
-1
lines changed

‎Architecture.md‎

Lines changed: 364 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,364 @@
1+
# IBM z/OS CICS Collection Architecture
2+
3+
## Overview
4+
5+
The **IBM z/OS CICS collection** (`ibm.ibm_zos_cics`) is an Ansible collection that provides automation capabilities for managing CICS (Customer Information Control System) resources and regions on IBM z/OS. The collection supports two primary use cases:
6+
7+
1. **CICS Resource Management** - Managing CICS resources through the CMCI REST API
8+
2. **CICS Region Provisioning** - Provisioning and managing standalone CICS regions
9+
10+
## Collection Structure
11+
12+
```
13+
ibm_zos_cics/
14+
├── plugins/
15+
│ ├── modules/ # Ansible modules (user-facing)
16+
│ ├── module_utils/ # Shared utility code
17+
│ ├── action/ # Action plugins for module execution
18+
│ ├── doc_fragments/ # Reusable documentation
19+
│ └── plugin_utils/ # Plugin utilities
20+
├── docs/ # Documentation
21+
├── tests/ # Integration and unit tests
22+
└── meta/ # Collection metadata
23+
```
24+
25+
## Architecture Components
26+
27+
### 1. Module Categories
28+
29+
The collection's modules are organized into two main categories:
30+
31+
#### A. CMCI Modules (Resource Management)
32+
33+
These modules interact with the CICS Management Client Interface (CMCI) REST API to manage CICS resources:
34+
35+
- [`cmci_get`](plugins/modules/cmci_get.py) - Query CICS resources and definitions
36+
- [`cmci_create`](plugins/modules/cmci_create.py) - Create new CICS resource definitions
37+
- [`cmci_update`](plugins/modules/cmci_update.py) - Update existing CICS resources
38+
- [`cmci_delete`](plugins/modules/cmci_delete.py) - Delete CICS resources
39+
- [`cmci_action`](plugins/modules/cmci_action.py) - Perform actions on CICS resources (e.g., NEWCOPY, INSTALL)
40+
41+
**Key Characteristics:**
42+
- Use HTTP/HTTPS to communicate with CMCI
43+
- Support both basic authentication and certificate-based authentication
44+
- Work with CICSPlex SM or standalone SMSS regions
45+
- Return structured data about CICS resources
46+
47+
#### B. Provisioning Modules (Region Management)
48+
49+
These modules manage CICS region data sets and lifecycle:
50+
51+
**Data Set Management:**
52+
- [`csd`](plugins/modules/csd.py) - CICS System Definition (CSD) data set
53+
- [`local_catalog`](plugins/modules/local_catalog.py) - Local catalog data set
54+
- [`global_catalog`](plugins/modules/global_catalog.py) - Global catalog data set
55+
- [`aux_temp_storage`](plugins/modules/aux_temp_storage.py) - Auxiliary temporary storage
56+
- [`aux_trace`](plugins/modules/aux_trace.py) - Auxiliary trace data sets
57+
- [`local_request_queue`](plugins/modules/local_request_queue.py) - Local request queue
58+
- [`td_intrapartition`](plugins/modules/td_intrapartition.py) - Transient data intrapartition
59+
- [`transaction_dump`](plugins/modules/transaction_dump.py) - Transaction dump data sets
60+
61+
**Region Lifecycle:**
62+
- [`region_jcl`](plugins/modules/region_jcl.py) - Generate CICS startup JCL
63+
- [`stop_region`](plugins/modules/stop_region.py) - Stop a running CICS region
64+
65+
**Key Characteristics:**
66+
- Create, initialize, and manage VSAM and sequential data sets
67+
- Support templated data set naming conventions
68+
- Provide state management (absent, initial, warm)
69+
- Execute z/OS utilities (IDCAMS, DFHCSDUP, DFHCCUTL, etc.)
70+
71+
### 2. Module Utilities Layer
72+
73+
The [`module_utils`](plugins/module_utils/) directory contains shared code used by multiple modules:
74+
75+
#### Core Utilities
76+
77+
- [`cmci.py`](plugins/module_utils/cmci.py) - Base class for all CMCI modules
78+
- Handles HTTP communication with CMCI REST API
79+
- Manages authentication (basic and certificate)
80+
- Parses XML responses using xmltodict
81+
- Provides filter and parameter handling
82+
- Validates CMCI responses and error handling
83+
84+
- [`_data_set.py`](plugins/module_utils/_data_set.py) - Base class for data set modules
85+
- Common data set operations (create, delete, initialize)
86+
- State management (absent, initial, warm)
87+
- Parameter validation using BetterArgParser
88+
- VSAM and sequential data set handling
89+
90+
#### Specialized Utilities
91+
92+
- [`_data_set_utils.py`](plugins/module_utils/_data_set_utils.py) - Data set operations
93+
- IDCAMS command building and execution
94+
- LISTDS operations for data set inspection
95+
- IEFBR14 for sequential data set creation
96+
97+
- [`_csd.py`](plugins/module_utils/_csd.py) - CSD-specific operations
98+
- DFHCSDUP command execution
99+
- CSD initialization scripts
100+
101+
- [`_local_catalog.py`](plugins/module_utils/_local_catalog.py) - Local catalog operations
102+
- DFHCCUTL utility execution
103+
104+
- [`_global_catalog.py`](plugins/module_utils/_global_catalog.py) - Global catalog operations
105+
- DFHRMUTL utility execution
106+
107+
- [`_jcl_helper.py`](plugins/module_utils/_jcl_helper.py) - JCL generation
108+
- Builds CICS startup JCL
109+
- Handles DD statements and parameters
110+
- Manages SIT (System Initialization Table) parameters
111+
112+
- [`_icetool.py`](plugins/module_utils/_icetool.py) - ICETOOL operations
113+
- Record counting for VSAM data sets
114+
115+
- [`_response.py`](plugins/module_utils/_response.py) - Response handling
116+
- Execution result structures
117+
- Exception handling
118+
119+
- [`_zoau_version_checker.py`](plugins/module_utils/_zoau_version_checker.py) - ZOAU validation
120+
- Ensures compatible ZOAU version is installed
121+
122+
### 3. Action Plugins
123+
124+
Action plugins in [`plugins/action/`](plugins/action/) provide custom execution logic for specific modules:
125+
The collection includes 10 action plugins for provisioning modules. **CMCI modules do not have custom action plugins** but support `module_defaults` via action_groups defined in `meta/runtime.yml`.
126+
127+
**Data Set Action Plugins** (8 plugins extending `_DataSetActionPlugin`):
128+
- [`aux_temp_storage.py`](plugins/action/aux_temp_storage.py), [`aux_trace.py`](plugins/action/aux_trace.py), [`csd.py`](plugins/action/csd.py), [`global_catalog.py`](plugins/action/global_catalog.py), [`local_catalog.py`](plugins/action/local_catalog.py), [`local_request_queue.py`](plugins/action/local_request_queue.py), [`td_intrapartition.py`](plugins/action/td_intrapartition.py), [`transaction_dump.py`](plugins/action/transaction_dump.py)
129+
- Run on controller to expand templated data set names (e.g., `<< data_set_name >>` → `DFHCSD`)
130+
- Validate parameters and resolve library references before sending to modules
131+
- `csd.py` reads local DFHCSDUP scripts; `aux_trace.py` and `transaction_dump.py` select A/B destinations
132+
133+
**Region JCL Action Plugin**:
134+
- [`region_jcl.py`](plugins/action/region_jcl.py) - Processes all region data sets and library templates for JCL generation
135+
136+
**Stop Region Action Plugin**:
137+
- [`stop_region.py`](plugins/action/stop_region.py) - Orchestrates multi-step CICS shutdown with job status polling
138+
139+
### 4. Documentation Fragments
140+
141+
Reusable documentation in [`plugins/doc_fragments/`](plugins/doc_fragments/) provides consistent parameter documentation across related modules:
142+
143+
- `cmci.py` - Common CMCI parameters (host, port, authentication)
144+
- `csd.py` - CSD module parameters
145+
- `local_catalog.py` - Local catalog parameters
146+
- `region_jcl.py` - Region JCL parameters
147+
- And others for each data set type
148+
149+
## Data Flow Architecture
150+
151+
### CMCI Module Flow
152+
153+
```mermaid
154+
flowchart TD
155+
A[User Playbook] --> B[Ansible Module<br/>e.g., cmci_get]
156+
B --> C[AnsibleCMCIModule<br/>cmci.py]
157+
C --> D[HTTP Request]
158+
D --> E[CMCI REST API]
159+
E --> F[CICS Region /<br/>CICSPlex SM]
160+
F --> G[XML Response]
161+
G --> H[Parse with xmltodict]
162+
H --> I[Return structured data to user]
163+
```
164+
165+
### Provisioning Module Flow
166+
167+
```mermaid
168+
flowchart TD
169+
A[User Playbook] --> B[Action Plugin<br/>e.g., csd.py<br/>Controller]
170+
B --> |Expand templates<br/>Validate parameters<br/>Read local files| C[Ansible Module<br/>e.g., csd<br/>Managed Node]
171+
C --> D[DataSet Base Class<br/>_data_set.py]
172+
D --> E[Specialized Module Utils<br/>e.g., _csd.py]
173+
E --> F[z/OS Utilities<br/>IDCAMS, DFHCSDUP, etc.]
174+
F --> G[ZOAU<br/>Z Open Automation Utilities]
175+
G --> H[z/OS System]
176+
H --> I[Return execution results to user]
177+
```
178+
179+
## Key Design Patterns
180+
181+
### 1. Inheritance Hierarchy
182+
183+
**CMCI Modules:**
184+
```
185+
AnsibleCMCIModule (base)
186+
├── AnsibleCMCIGetModule
187+
├── AnsibleCMCICreateModule
188+
├── AnsibleCMCIUpdateModule
189+
├── AnsibleCMCIDeleteModule
190+
└── AnsibleCMCIInstallModule (cmci_action)
191+
```
192+
193+
**Data Set Modules:**
194+
```
195+
DataSet (base)
196+
├── AnsibleCSDModule
197+
├── AnsibleLocalCatalogModule
198+
├── AnsibleGlobalCatalogModule
199+
├── AnsibleAuxiliaryTempModule
200+
├── AnsibleAuxiliaryTraceModule
201+
├── AnsibleLocalRequestQueueModule
202+
├── AnsibleTDIntraModule
203+
└── AnsibleTransactionDumpModule
204+
```
205+
206+
### 2. State Management
207+
208+
Data set modules implement a state-based approach:
209+
210+
- **`absent`** - Ensure data set does not exist (delete if present)
211+
- **`initial`** - Create new or reinitialize existing data set (empty)
212+
- **`warm`** - Ensure data set exists with current content preserved
213+
- **`cold`** - Cold start the global catalog (global_catalog only)
214+
- **`changed`** - Apply changes to existing data set (csd only)
215+
216+
### 3. Template-Based Configuration
217+
218+
Modules support templated data set names for consistency:
219+
220+
```yaml
221+
region_data_sets:
222+
template: "REGIONS.ABCD0001.<< data_set_name >>"
223+
```
224+
225+
This expands to:
226+
- `REGIONS.ABCD0001.DFHCSD`
227+
- `REGIONS.ABCD0001.DFHLCD`
228+
- `REGIONS.ABCD0001.DFHTEMP`
229+
- etc.
230+
231+
### 4. Execution Tracking
232+
233+
All modules track execution details:
234+
235+
```python
236+
executions = [
237+
{
238+
"name": "Create data set",
239+
"rc": 0,
240+
"stdout": "...",
241+
"stderr": ""
242+
}
243+
]
244+
```
245+
246+
This provides transparency and debugging capability.
247+
248+
## Dependencies
249+
250+
### External Dependencies
251+
252+
1. **ZOAU (Z Open Automation Utilities)** - Required for provisioning modules
253+
- Provides Python APIs for z/OS operations
254+
- Minimum version checked at runtime
255+
256+
2. **xmltodict** - Required for CMCI modules
257+
- Parses XML responses from CMCI REST API
258+
259+
3. **ibm.ibm_zos_core** - Ansible collection dependency
260+
- Provides core z/OS functionality
261+
- Used for data set operations, job management, TSO commands
262+
263+
### z/OS Requirements
264+
265+
**For CMCI Modules:**
266+
- CMCI REST API enabled in CICSPlex SM or SMSS
267+
- Network connectivity to CMCI host
268+
- Valid credentials or certificates
269+
270+
**For Provisioning Modules:**
271+
- ZOAU installed on managed node
272+
- CICS libraries (SDFHLOAD, etc.)
273+
- Language Environment libraries
274+
- Appropriate z/OS authorizations
275+
276+
## Error Handling
277+
278+
### CMCI Modules
279+
280+
- HTTP errors are caught and reported with status codes
281+
- CMCI response codes are validated against expected values
282+
- Feedback records provide detailed error information
283+
- XML parsing errors are handled gracefully
284+
285+
### Provisioning Modules
286+
287+
- MVSExecutionException wraps z/OS utility failures
288+
- Return codes from utilities are checked
289+
- Execution history is preserved for debugging
290+
- Data set state is validated before operations
291+
292+
## Security Considerations
293+
294+
### Authentication
295+
296+
**CMCI Modules:**
297+
- Support basic authentication (username/password)
298+
- Support certificate-based authentication (cert/key)
299+
- Credentials can be provided via environment variables
300+
- Passwords and certificates are marked as `no_log`
301+
302+
**Provisioning Modules:**
303+
- Rely on SSH authentication to managed node
304+
- Execute with user's z/OS credentials
305+
- Require appropriate data set and utility authorizations
306+
307+
## Performance Considerations
308+
309+
### CMCI Modules
310+
311+
- Single HTTP request per module invocation
312+
- Configurable timeout (default 30 seconds)
313+
- Support for filtering to reduce data transfer
314+
- Connection reuse within session
315+
316+
### Provisioning Modules
317+
318+
- Data set operations can be time-consuming
319+
- Large data sets may require significant space
320+
- VSAM operations are synchronous
321+
- Multiple modules can run in parallel (different data sets)
322+
323+
## Extensibility
324+
325+
### Adding New CMCI Modules
326+
327+
1. Extend `AnsibleCMCIModule` base class
328+
2. Implement required methods:
329+
- `init_argument_spec()` - Define parameters
330+
- `init_body()` - Build request body (if needed)
331+
- `init_request_params()` - Build URL parameters
332+
333+
### Adding New Data Set Modules
334+
335+
1. Extend `DataSet` base class
336+
2. Implement required methods:
337+
- `_get_arg_spec()` - Define parameters
338+
- `create_data_set()` - Data set creation logic
339+
- `execute_target_state()` - State-specific logic (optional)
340+
3. Create corresponding module_utils helper (e.g., `_new_dataset.py`)
341+
342+
## Testing Strategy
343+
344+
### Integration Tests
345+
346+
Located in [`tests/integration/targets/`](tests/integration/):
347+
- Test against real CICS regions
348+
- Validate end-to-end workflows
349+
- Cover error scenarios
350+
351+
### Unit Tests
352+
353+
Located in [`tests/unit/`](tests/unit/):
354+
- Test individual functions and classes
355+
- Mock external dependencies
356+
- Validate parameter parsing and validation
357+
358+
## Version Compatibility
359+
360+
- **Ansible**: Requires >= 2.15.0
361+
- **Python**: Compatible with Python 3.6+
362+
- **CICS**: Supports CICS TS 5.6 and later
363+
- **z/OS**: Tested on z/OS 2.4 and later
364+
- **ZOAU**: Version checked at runtime

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# (c) Copyright IBM Corp. 2020,2023
22
# Developer guide
33

4+
Read the [development guide](DEVELOPER_GUIDE.md) to learn how to contribute to this project.
5+
46
### Licensing
57

68
All code must have an Apache-2.0 header.
@@ -60,4 +62,4 @@ You must use your real name and email address.
6062

6163
To save you having to type the above for every commit, Git can add the `Signed-off-by` line. When committing, add the `-s` option to your `git commit` command.
6264

63-
If you haven't signed each commit, then the pull request will fail to pass all checks.
65+
If you haven't signed each commit, then the pull request will fail to pass all checks.

0 commit comments

Comments
 (0)