[zos_find] Support GDG and special characters in data set names for zos_find (#1518)

* initial fragment and added test case

* added changelog file
modified new test, added limit to test value

* needed to add limit parameter for gdg in the batch suboption of data_set

* added limit as a dependancy to batch/datasettpe

* added detail to the error raised on missing limit

* changing test to eliminate batch call

* fixed parenthesis

* update test constructor to create gds (no paren) and use force=true

* detailing the decho response

* trying to escape the gdg number

* updated zos_data_set definition for batch mode of gdg in dataset
altered test to re-test the new definition

* fixed parenthesis issue

* data set creator is now working.
removed escapes around gdg version indicator in test.

* expanded test to write content generation entry

* updating test to remove prior output and truncate the final version info

* removed cars print statement.

* corrected join statement in string split

* updated join statement

* added cleanup to gdg test routine

* adding detail to pds writer

* updated test updated for specific file type, not just top 2 names

* added special character tests for zos_find

* changed an assertion in special/symbol test

* added force=True to gdg deletion in that test, so the subs would all get removed

* added escape to decho command

* updated changelog fragment to include PR number
removed print statements from test

* added dataset import back into test function

* working on re-establishing zos_data_set in the test form

* correcting import for zos_data_set

* Modified test cases to use random generated names and improve performance

* Removed hard coded names

* Adding support for GDG filter in zos_find

* Modified tests for filtering GDGs

* Added documentation samples and fixed sanity issues

* fixed test suite

---------

Co-authored-by: Fernando Flores <[email protected]>

Author: richp405

changelogs/fragments/1374-enhancement-zos-find-gdg-gds-special-chars.yml

@@ -0,0 +1,3 @@
+minor_changes:
+  - zos_find - added support for GDG/GDS and special characters
+    (https://github.com/ansible-collections/ibm_zos_core/pull/1518).

plugins/modules/zos_find.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
 
-# Copyright (c) IBM Corporation 2020, 2023
+# Copyright (c) IBM Corporation 2020, 2024
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -109,11 +109,13 @@
       - C(nonvsam) refers to one of SEQ, LIBRARY (PDSE), PDS, LARGE, BASIC, EXTREQ, or EXTPREF.
       - C(cluster) refers to a VSAM cluster. The C(data) and C(index) are the data and index
         components of a VSAM cluster.
+      - C(gdg) refers to Generation Data Groups. The module searches based on the GDG base name.
     choices:
       - nonvsam
       - cluster
       - data
       - index
+      - gdg
     type: str
     required: false
     default: "nonvsam"
@@ -126,6 +128,43 @@
     required: false
     aliases:
       - volumes
+  empty:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(empty) attribute set as provided.
+    type: bool
+    required: false
+  extended:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(extended) attribute set as provided.
+    type: bool
+    required: false
+  fifo:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(fifo) attribute set as provided.
+    type: bool
+    required: false
+  limit:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(limit) attribute set as provided.
+    type: int
+    required: false
+  purge:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(purge) attribute set as provided.
+    type: bool
+    required: false
+  scratch:
+    description:
+      - A GDG attribute, only valid when C(resource_type=gdg).
+      - If provided, will search for data sets with I(scratch) attribute set as provided.
+    type: bool
+    required: false
+
 notes:
   - Only cataloged data sets will be searched. If an uncataloged data set needs to
     be searched, it should be cataloged first. The L(zos_data_set,./zos_data_set.html) module can be
@@ -186,6 +225,16 @@
     patterns:
       - USER.*
     resource_type: cluster
+
+- name: Find all Generation Data Groups starting with the word 'USER' and specific GDG attributes.
+  zos_find:
+    patterns:
+      - USER.*
+    resource_type: gdg
+    limit: 30
+    scratch: true
+    purge: true
+
 """
 
 
@@ -249,6 +298,7 @@
 import time
 import datetime
 import math
+import json
 
 from copy import deepcopy
 from re import match as fullmatch
@@ -535,6 +585,66 @@ def data_set_attribute_filter(
     return filtered_data_sets
 
 
+def gdg_filter(module, data_sets, limit, empty, fifo, purge, scratch, extended):
+    """ Filter Generation Data Groups based on their attributes.
+
+    Parameters
+    ----------
+    module : AnsibleModule
+        The Ansible module object being used.
+    data_sets : set[str]
+        A set of data set names.
+    limit : int
+        The limit GDG attribute that should be used to filter GDGs.
+    empty : bool
+        The empty GDG attribute, that should be used to filter GDGs.
+    fifo : bool
+        The fifo GDG attribute, that should be used to filter GDGs.
+    purge : bool
+        The purge GDG attribute, that should be used to filter GDGs.
+    scratch : bool
+        The scratch GDG attribute, that should be used to filter GDGs.
+    extended : bool
+        The extended GDG attribute, that should be used to filter GDGs.
+
+    Returns
+    -------
+    set[str]
+        Matched GDG base names.
+
+    Raises
+    ------
+    fail_json
+        Non-zero return code received while executing ZOAU shell command 'dls'.
+    """
+    filtered_data_sets = set()
+    for ds in data_sets:
+        rc, out, err = _dls_wrapper(ds, data_set_type='gdg', list_details=True, json=True)
+
+        if rc != 0:
+            module.fail_json(
+                msg="Non-zero return code received while executing ZOAU shell command 'dls'",
+                rc=rc, stdout=out, stderr=err
+            )
+        try:
+            response = json.loads(out)
+            gdgs = response['data']['gdgs']
+            for gdg in gdgs:
+                if (
+                    gdg['limit'] == (gdg['limit'] if limit is None else limit) and
+                    gdg['empty'] == (gdg['empty'] if empty is None else empty) and
+                    gdg['purge'] == (gdg['purge'] if purge is None else purge) and
+                    gdg['fifo'] == (gdg['fifo'] if fifo is None else fifo) and
+                    gdg['scratch'] == (gdg['scratch'] if scratch is None else scratch) and
+                    gdg['extended'] == (gdg['extended'] if extended is None else extended)
+                ):
+                    filtered_data_sets.add(gdg['base'])
+        except Exception as e:
+            module.fail_json(repr(e))
+
+        return filtered_data_sets
+
+
 # TODO:
 # Implement volume_filter() using "vtocls" shell command from ZOAU
 # when it becomes available.
@@ -779,7 +889,9 @@ def _dls_wrapper(
     u_time=False,
     size=False,
     verbose=False,
-    migrated=False
+    migrated=False,
+    data_set_type="",
+    json=False,
 ):
     """A wrapper for ZOAU 'dls' shell command.
 
@@ -797,6 +909,10 @@ def _dls_wrapper(
         Display verbose information.
     migrated : bool
         Display migrated data sets.
+    data_set_type : str
+        Data set type to look for.
+    json : bool
+        Return content in json format.
 
     Returns
     -------
@@ -815,6 +931,10 @@ def _dls_wrapper(
             dls_cmd += " -s"
     if verbose:
         dls_cmd += " -v"
+    if data_set_type:
+        dls_cmd += f" -t{data_set_type}"
+    if json:
+        dls_cmd += " -j"
 
     dls_cmd += " {0}".format(quote(data_set_pattern))
     return AnsibleModuleHelper(argument_spec={}).run_command(dls_cmd)
@@ -926,6 +1046,12 @@ def run_module(module):
     )
     resource_type = module.params.get('resource_type').upper()
     volume = module.params.get('volume') or module.params.get('volumes')
+    limit = module.params.get('limit')
+    empty = module.params.get('empty')
+    scratch = module.params.get('scratch')
+    purge = module.params.get('purge')
+    extended = module.params.get('extended')
+    fifo = module.params.get('fifo')
 
     res_args = dict(data_sets=[])
     filtered_data_sets = set()
@@ -983,9 +1109,11 @@ def run_module(module):
 
         res_args['examined'] = init_filtered_data_sets.get("searched")
 
-    else:
+    elif resource_type == "CLUSTER":
         filtered_data_sets = vsam_filter(module, patterns, resource_type, age=age)
         res_args['examined'] = len(filtered_data_sets)
+    elif resource_type == "GDG":
+        filtered_data_sets = gdg_filter(module, patterns, limit, empty, fifo, purge, scratch, extended)
 
     # Filter out data sets that match one of the patterns in 'excludes'
     if excludes and not pds_paths:
@@ -1045,14 +1173,20 @@ def main():
             ),
             resource_type=dict(
                 type="str", required=False, default="nonvsam",
-                choices=["cluster", "data", "index", "nonvsam"]
+                choices=["cluster", "data", "index", "nonvsam", "gdg"]
             ),
             volume=dict(
                 type="list",
                 elements="str",
                 required=False,
                 aliases=["volumes"]
-            )
+            ),
+            limit=dict(type="int", required=False),
+            empty=dict(type="bool", required=False),
+            purge=dict(type="bool", required=False),
+            scratch=dict(type="bool", required=False),
+            extended=dict(type="bool", required=False),
+            fifo=dict(type="bool", required=False),
         )
     )
 
@@ -1077,9 +1211,15 @@ def main():
             arg_type="str",
             required=False,
             default="nonvsam",
-            choices=["cluster", "data", "index", "nonvsam"]
+            choices=["cluster", "data", "index", "nonvsam", "gdg"]
         ),
-        volume=dict(arg_type="list", required=False, aliases=["volumes"])
+        volume=dict(arg_type="list", required=False, aliases=["volumes"]),
+        limit=dict(type="int", required=False),
+        empty=dict(type="bool", required=False),
+        purge=dict(type="bool", required=False),
+        scratch=dict(type="bool", required=False),
+        extended=dict(type="bool", required=False),
+        fifo=dict(type="bool", required=False),
     )
     try:
         BetterArgParser(arg_def).parse_args(module.params)