[Enabler] [zos_job_submit] Add support for GDG/GDS (#1497)

* modified DatasetCreatedError message

* Added gdg functions

* Created unit test for validating gds relative name

* Updated to fail when future gen

* Update arg parser

* Adding gdg support for zos_data_set

* Add escaping function for data set names

* Add unit tests for name escaping

* Remove calls to escape_data_set_name

* renamed tests

* Added MVSDataset class

* Updated escaped symbols

* Updated tests

* Added utils

* Add changelog

* Uncommented test

* Updated exception

* Updated mvsdataset class

* Updated class

* Added type

* Added gds tests

* Testing for special symbols

* Made data set name escaping optional

* Use new class for GDS handling

* Update special chars data set name

* Escape dollar sign in test

* Add positive test for GDG/GDS

* Add negative GDG tests

* Update docs

* Fix data set existence check

* Update docs again

* Add changelog fragment

* Fix merge with dev

* Fix source validation

* Fix validate-modules issue

---------

Co-authored-by: Fernando Flores <[email protected]>

Author: rexemin

changelogs/fragments/1497-gdg-support-zos-job-submit.yml

@@ -0,0 +1,4 @@
+minor_changes:
+  - zos_job_submit - add support for generation data groups and generation
+    data sets as sources for jobs.
+    (https://github.com/ansible-collections/ibm_zos_core/pull/1497)

docs/source/modules/zos_job_submit.rst

@@ -31,11 +31,13 @@ Parameters
 src
   The source file or data set containing the JCL to submit.
 
-  It could be a physical sequential data set, a partitioned data set qualified by a member or a path. (e.g "USER.TEST","USER.JCL(TEST)")
+  It could be a physical sequential data set, a partitioned data set qualified by a member or a path (e.g. \ :literal:`USER.TEST`\ , \ :literal:`USER.JCL(TEST)`\ ), or a generation data set from a generation data group (for example, \ :literal:`USER.TEST.GDG(-2)`\ ).
 
-  Or a USS file. (e.g "/u/tester/demo/sample.jcl")
+  Or a USS file. (e.g \ :literal:`/u/tester/demo/sample.jcl`\ )
 
-  Or a LOCAL file in ansible control node. (e.g "/User/tester/ansible-playbook/sample.jcl")
+  Or a LOCAL file in ansible control node. (e.g \ :literal:`/User/tester/ansible-playbook/sample.jcl`\ )
+
+  When using a generation data set, only already created generations are valid. If either the relative name is positive, or negative but not found, the module will fail.
 
   | **required**: True
   | **type**: str
@@ -44,11 +46,11 @@ src
 location
   The JCL location. Supported choices are \ :literal:`data\_set`\ , \ :literal:`uss`\  or \ :literal:`local`\ .
 
-  \ :literal:`data\_set`\  can be a PDS, PDSE, or sequential data set.
+  \ :literal:`data\_set`\  can be a PDS, PDSE, sequential data set, or a generation data set.
 
   \ :literal:`uss`\  means the JCL location is located in UNIX System Services (USS).
 
-  \ :literal:`local`\  means locally to the ansible control node.
+  \ :literal:`local`\  means locally to the Ansible control node.
 
   | **required**: False
   | **type**: str
@@ -311,6 +313,16 @@ Examples
        location: data_set
        max_rc: 16
 
+   - name: Submit JCL from the latest generation data set in a generation data group.
+     zos_job_submit:
+       src: HLQ.DATA.GDG(0)
+       location: data_set
+
+   - name: Submit JCL from a previous generation data set in a generation data group.
+     zos_job_submit:
+       src: HLQ.DATA.GDG(-2)
+       location: data_set
+
 
 
 

plugins/module_utils/data_set.py

@@ -1809,6 +1809,7 @@ class MVSDataSet():
     def __init__(
         self,
         name,
+        escape_name=False,
         data_set_type=None,
         state=None,
         organization=None,
@@ -1858,15 +1859,16 @@ def __init__(
         self.is_cataloged = False
 
         # If name has escaped chars or is GDS relative name we clean it.
-        # self.name = DataSet.escape_data_set_name(self.name)
+        if escape_name:
+            self.name = DataSet.escape_data_set_name(self.name)
         if DataSet.is_gds_relative_name(self.name):
             try:
                 self.name = DataSet.resolve_gds_absolute_name(self.name)
                 self.is_gds_active = True
             except Exception:
                 # This means the generation is a positive version so is only used for creation.
                 self.is_gds_active = False
-        if self.data_set_type.upper() in DataSet.MVS_VSAM or self.data_set_type == "zfs":
+        if self.data_set_type and (self.data_set_type.upper() in DataSet.MVS_VSAM or self.data_set_type == "zfs"):
             # When trying to create a new VSAM with a specified record format will fail
             # with ZOAU
             self.record_format = None

plugins/modules/zos_job_submit.py

@@ -36,10 +36,15 @@
     description:
       - The source file or data set containing the JCL to submit.
       - It could be a physical sequential data set, a partitioned data set
-        qualified by a member or a path. (e.g "USER.TEST","USER.JCL(TEST)")
-      - Or a USS file. (e.g "/u/tester/demo/sample.jcl")
+        qualified by a member or a path (e.g. C(USER.TEST), V(USER.JCL(TEST\))),
+        or a generation data set from a generation data group
+        (for example, V(USER.TEST.GDG(-2\))).
+      - Or a USS file. (e.g C(/u/tester/demo/sample.jcl))
       - Or a LOCAL file in ansible control node.
-        (e.g "/User/tester/ansible-playbook/sample.jcl")
+        (e.g C(/User/tester/ansible-playbook/sample.jcl))
+      - When using a generation data set, only already created generations
+        are valid. If either the relative name is positive, or negative but
+        not found, the module will fail.
   location:
     required: false
     default: data_set
@@ -50,9 +55,9 @@
       - local
     description:
       - The JCL location. Supported choices are C(data_set), C(uss) or C(local).
-      - C(data_set) can be a PDS, PDSE, or sequential data set.
+      - C(data_set) can be a PDS, PDSE, sequential data set, or a generation data set.
       - C(uss) means the JCL location is located in UNIX System Services (USS).
-      - C(local) means locally to the ansible control node.
+      - C(local) means locally to the Ansible control node.
   wait_time_s:
     required: false
     default: 10
@@ -601,6 +606,16 @@
     src: HLQ.DATA.LLQ
     location: data_set
     max_rc: 16
+
+- name: Submit JCL from the latest generation data set in a generation data group.
+  zos_job_submit:
+    src: HLQ.DATA.GDG(0)
+    location: data_set
+
+- name: Submit JCL from a previous generation data set in a generation data group.
+  zos_job_submit:
+    src: HLQ.DATA.GDG(-2)
+    location: data_set
 """
 
 from ansible_collections.ibm.ibm_zos_core.plugins.module_utils.encode import (
@@ -647,7 +662,7 @@
 MAX_WAIT_TIME_S = 86400
 
 
-def submit_src_jcl(module, src, src_name=None, timeout=0, is_unix=True, volume=None, start_time=timer()):
+def submit_src_jcl(module, src, src_name=None, timeout=0, is_unix=True, start_time=timer()):
     """Submit src JCL whether JCL is local (Ansible Controller), USS or in a data set.
 
         Parameters
@@ -666,9 +681,6 @@ def submit_src_jcl(module, src, src_name=None, timeout=0, is_unix=True, volume=N
             True if JCL is a file in USS, otherwise False; Note that all
             JCL local to a controller is transfered to USS thus would be
             True.
-        volume : str
-            volume the data set JCL is located on that will be cataloged before
-            being submitted.
         start_time : int
             time the JCL started its submission.
 
@@ -704,20 +716,6 @@ def submit_src_jcl(module, src, src_name=None, timeout=0, is_unix=True, volume=N
     result = {}
 
     try:
-        if volume is not None:
-            volumes = [volume]
-            # Get the PDS name to catalog it
-            src_ds_name = data_set.extract_dsname(src)
-            present, changed = DataSet.attempt_catalog_if_necessary(
-                src_ds_name, volumes)
-
-            if not present:
-                result["changed"] = False
-                result["failed"] = True
-                result["msg"] = ("Unable to submit job {0} because the data set could "
-                                 "not be cataloged on the volume {1}.".format(src, volume))
-                module.fail_json(**result)
-
         job_submitted = jobs.submit(src, is_unix=is_unix, **kwargs)
 
         # Introducing a sleep to ensure we have the result of job submit carrying the job id.
@@ -952,9 +950,32 @@ def run_module():
     job_submitted_id = None
     duration = 0
     start_time = timer()
+
     if location == "data_set":
+        # Resolving a relative GDS name and escaping special symbols if needed.
+        src_data = data_set.MVSDataSet(src)
+
+        # Checking that the source is actually present on the system.
+        if volume is not None:
+            volumes = [volume]
+            # Get the data set name to catalog it.
+            src_ds_name = data_set.extract_dsname(src_data.name)
+            present, changed = DataSet.attempt_catalog_if_necessary(src_ds_name, volumes)
+
+            if not present:
+                module.fail_json(
+                    msg=(f"Unable to submit job {src_data.name} because the data set could "
+                         f"not be cataloged on the volume {volume}.")
+                )
+        elif data_set.is_member(src_data.name):
+            if not DataSet.data_set_member_exists(src_data.name):
+                module.fail_json(msg=f"Cannot submit job, the data set member {src_data.raw_name} was not found.")
+        else:
+            if not DataSet.data_set_exists(src_data.name):
+                module.fail_json(msg=f"Cannot submit job, the data set {src_data.raw_name} was not found.")
+
         job_submitted_id, duration = submit_src_jcl(
-            module, src, src_name=src, timeout=wait_time_s, is_unix=False, volume=volume, start_time=start_time)
+            module, src_data.name, src_name=src_data.raw_name, timeout=wait_time_s, is_unix=False, start_time=start_time)
     elif location == "uss":
         job_submitted_id, duration = submit_src_jcl(
             module, src, src_name=src, timeout=wait_time_s, is_unix=True)

tests/functional/modules/test_zos_job_submit_func.py

@@ -399,7 +399,7 @@
 """
 
 TEMP_PATH = "/tmp/jcl"
-DATA_SET_NAME_SPECIAL_CHARS = "imstestl.im@1.xxx05"
+DATA_SET_NAME_SPECIAL_CHARS = "imstestl.im@1.x#$xx05"
 
 @pytest.mark.parametrize(
     "location", [
@@ -460,7 +460,7 @@ def test_job_submit_PDS_special_characters(ansible_zos_module):
         )
         hosts.all.shell(
             cmd="cp {0}/SAMPLE \"//'{1}(SAMPLE)'\"".format(
-                TEMP_PATH, DATA_SET_NAME_SPECIAL_CHARS
+                TEMP_PATH, DATA_SET_NAME_SPECIAL_CHARS.replace('$', '\$')
             )
         )
         results = hosts.all.zos_job_submit(
@@ -922,6 +922,79 @@ def test_job_submit_local_jcl_typrun_jclhold(ansible_zos_module):
         assert result.get("jobs")[0].get("ret_code").get("msg_code") is None
 
 
+@pytest.mark.parametrize("generation", ["0", "-1"])
+def test_job_from_gdg_source(ansible_zos_module, generation):
+    hosts = ansible_zos_module
+
+    try:
+        # Creating a GDG for the test.
+        source = get_tmp_ds_name()
+        gds_name = f"{source}({generation})"
+        hosts.all.zos_data_set(name=source, state="present", type="gdg", limit=3)
+        hosts.all.zos_data_set(name=f"{source}(+1)", state="present", type="seq")
+        hosts.all.zos_data_set(name=f"{source}(+1)", state="present", type="seq")
+
+        # Copying the JCL to the GDS.
+        hosts.all.file(path=TEMP_PATH, state="directory")
+        hosts.all.shell(
+            cmd="echo {0} > {1}/SAMPLE".format(quote(JCL_FILE_CONTENTS), TEMP_PATH)
+        )
+        hosts.all.shell(
+            cmd="dcp '{0}/SAMPLE' '{1}'".format(TEMP_PATH, gds_name)
+        )
+
+        results = hosts.all.zos_job_submit(src=gds_name, location="data_set")
+        for result in results.contacted.values():
+            assert result.get("jobs")[0].get("ret_code").get("msg_code") == "0000"
+            assert result.get("jobs")[0].get("ret_code").get("code") == 0
+            assert result.get("changed") is True
+    finally:
+        hosts.all.file(path=TEMP_PATH, state="absent")
+        hosts.all.zos_data_set(name=f"{source}(0)", state="absent")
+        hosts.all.zos_data_set(name=f"{source}(-1)", state="absent")
+        hosts.all.zos_data_set(name=source, state="absent")
+
+
+def test_inexistent_negative_gds(ansible_zos_module):
+    hosts = ansible_zos_module
+
+    try:
+        # Creating a GDG for the test.
+        source = get_tmp_ds_name()
+        gds_name = f"{source}(-1)"
+        hosts.all.zos_data_set(name=source, state="present", type="gdg", limit=3)
+        # Only creating generation 0.
+        hosts.all.zos_data_set(name=f"{source}(+1)", state="present", type="seq")
+
+        results = hosts.all.zos_job_submit(src=gds_name, location="data_set")
+        for result in results.contacted.values():
+            assert result.get("changed") is False
+            assert "was not found" in result.get("msg")
+    finally:
+        hosts.all.zos_data_set(name=f"{source}(0)", state="absent")
+        hosts.all.zos_data_set(name=source, state="absent")
+
+
+def test_inexistent_positive_gds(ansible_zos_module):
+    hosts = ansible_zos_module
+
+    try:
+        # Creating a GDG for the test.
+        source = get_tmp_ds_name()
+        gds_name = f"{source}(+1)"
+        hosts.all.zos_data_set(name=source, state="present", type="gdg", limit=3)
+        # Only creating generation 0.
+        hosts.all.zos_data_set(name=gds_name, state="present", type="seq")
+
+        results = hosts.all.zos_job_submit(src=gds_name, location="data_set")
+        for result in results.contacted.values():
+            assert result.get("changed") is False
+            assert "was not found" in result.get("msg")
+    finally:
+        hosts.all.zos_data_set(name=f"{source}(0)", state="absent")
+        hosts.all.zos_data_set(name=source, state="absent")
+
+
 # This test case is related to the following GitHub issues:
 # - https://github.com/ansible-collections/ibm_zos_core/issues/677
 # - https://github.com/ansible-collections/ibm_zos_core/issues/972