[Enabler] [zos_copy] Add GDG/GDS support to zos_copy (#1564)

* Add use of new data set class

* Add tests for GDS source

* Add support for a GDS destination

* Add more GDS tests

* Fix name resolution when allocating a new GDS

* Fix GDS validations

* Add more type validations

* Remove unnecessary name resolution

* Add support to copy a GDG to USS

* Add copy of complete GDGs

* Add support for a GDS as backup

* Add special symbols test and fix backup ones

* Add changelog fragment

* Update docs

* Add GDG attributes for dest_data_set

* Update module RST

* Fix pep8 issue

* Fix backups without backup names

* Fix non-GDG dest dataset allocation

* Fix GDS allocation

Author: rexemin

changelogs/fragments/1564-zos_copy_gdg_support.yml

@@ -0,0 +1,4 @@
+minor_changes:
+  - zos_copy - add support for copying generation data sets (GDS) and
+    generation data groups (GDG), as well as using a GDS for backup.
+    (https://github.com/ansible-collections/ibm_zos_core/pull/1564).

docs/source/modules/zos_copy.rst

@@ -67,6 +67,8 @@ backup_name
 
   If \ :literal:`dest`\  is a data set member and \ :literal:`backup\_name`\  is not provided, the data set member will be backed up to the same partitioned data set with a randomly generated member name.
 
+  If \ :emphasis:`backup\_name`\  is a generation data set (GDS), it must be a relative positive name (for example, \ :literal:`HLQ.USER.GDG(+1)`\ ).
+
   | **required**: False
   | **type**: str
 
@@ -105,6 +107,10 @@ dest
 
   When \ :literal:`dest`\  is and existing VSAM (LDS), then source must be an LDS. The VSAM (LDS) will be deleted and recreated following the process outlined in the \ :literal:`volume`\  option.
 
+  \ :literal:`dest`\  can be a previously allocated generation data set (GDS) or a new GDS.
+
+  When \ :literal:`dest`\  is a generation data group (GDG), \ :literal:`src`\  must be a GDG too. The copy will allocate successive new generations in \ :literal:`dest`\ , the module will verify it has enough available generations before starting the copy operations.
+
   When \ :literal:`dest`\  is a data set, you can override storage management rules by specifying \ :literal:`volume`\  if the storage class being used has GUARANTEED\_SPACE=YES specified, otherwise, the allocation will fail. See \ :literal:`volume`\  for more volume related processes.
 
   | **required**: True
@@ -298,6 +304,10 @@ src
 
   If \ :literal:`src`\  is a VSAM data set, \ :literal:`dest`\  must also be a VSAM.
 
+  If \ :literal:`src`\  is a generation data set (GDS), it must be a previously allocated one.
+
+  If \ :literal:`src`\  is a generation data group (GDG), \ :literal:`dest`\  can be another GDG or a USS directory.
+
   Wildcards can be used to copy multiple PDS/PDSE members to another PDS/PDSE.
 
   Required unless using \ :literal:`content`\ .
@@ -334,6 +344,8 @@ volume
 dest_data_set
   Data set attributes to customize a \ :literal:`dest`\  data set to be copied into.
 
+  Some attributes only apply when \ :literal:`dest`\  is a generation data group (GDG).
+
   | **required**: False
   | **type**: dict
 
@@ -343,7 +355,7 @@ dest_data_set
 
     | **required**: True
     | **type**: str
-    | **choices**: ksds, esds, rrds, lds, seq, pds, pdse, member, basic, library
+    | **choices**: ksds, esds, rrds, lds, seq, pds, pdse, member, basic, library, gdg
 
 
   space_primary
@@ -470,6 +482,68 @@ dest_data_set
     | **type**: str
 
 
+  limit
+    Sets the \ :emphasis:`limit`\  attribute for a GDG.
+
+    Specifies the maximum number, from 1 to 255(up to 999 if extended), of generations that can be associated with the GDG being defined.
+
+    \ :emphasis:`limit`\  is required when \ :emphasis:`type=gdg`\ .
+
+    | **required**: False
+    | **type**: int
+
+
+  empty
+    Sets the \ :emphasis:`empty`\  attribute for a GDG.
+
+    If false, removes only the oldest GDS entry when a new GDS is created that causes GDG limit to be exceeded.
+
+    If true, removes all GDS entries from a GDG base when a new GDS is created that causes the GDG limit to be exceeded.
+
+    | **required**: False
+    | **type**: bool
+
+
+  scratch
+    Sets the \ :emphasis:`scratch`\  attribute for a GDG.
+
+    Specifies what action is to be taken for a generation data set located on disk volumes when the data set is uncataloged from the GDG base as a result of EMPTY/NOEMPTY processing.
+
+    | **required**: False
+    | **type**: bool
+
+
+  purge
+    Sets the \ :emphasis:`purge`\  attribute for a GDG.
+
+    Specifies whether to override expiration dates when a generation data set (GDS) is rolled off and the \ :literal:`scratch`\  option is set.
+
+    | **required**: False
+    | **type**: bool
+
+
+  extended
+    Sets the \ :emphasis:`extended`\  attribute for a GDG.
+
+    If false, allow up to 255 generation data sets (GDSs) to be associated with the GDG.
+
+    If true, allow up to 999 generation data sets (GDS) to be associated with the GDG.
+
+    | **required**: False
+    | **type**: bool
+
+
+  fifo
+    Sets the \ :emphasis:`fifo`\  attribute for a GDG.
+
+    If false, the order is the newest GDS defined to the oldest GDS. This is the default value.
+
+    If true, the order is the oldest GDS defined to the newest GDS.
+
+    | **required**: False
+    | **type**: bool
+
+
 
 use_template
   Whether the module should treat \ :literal:`src`\  as a Jinja2 template and render it before continuing with the rest of the module.
@@ -794,6 +868,19 @@ Examples
        dest: HLQ.PRINT.NEW
        asa_text: true
 
+   - name: Copy a file to a new generation data set.
+     zos_copy:
+       src: /path/to/uss/src
+       dest: HLQ.TEST.GDG(+1)
+       remote_src: true
+
+   - name: Copy a local file and take a backup of the existing file with a GDS.
+     zos_copy:
+       src: /path/to/local/file
+       dest: /path/to/dest
+       backup: true
+       backup_name: HLQ.BACKUP.GDG(+1)
+
 
 
 

plugins/module_utils/copy.py

@@ -15,6 +15,8 @@
 __metaclass__ = type
 
 
+import traceback
+from os import path
 from ansible.module_utils.six import PY3
 from ansible_collections.ibm.ibm_zos_core.plugins.module_utils.ansible_module import (
     AnsibleModuleHelper,
@@ -25,12 +27,19 @@
 from ansible_collections.ibm.ibm_zos_core.plugins.module_utils.mvs_cmd import (
     ikjeft01
 )
+from ansible_collections.ibm.ibm_zos_core.plugins.module_utils.import_handler import \
+    ZOAUImportError
 
 if PY3:
     from shlex import quote
 else:
     from pipes import quote
 
+try:
+    from zoautil_py import datasets, gdgs
+except Exception:
+    datasets = ZOAUImportError(traceback.format_exc())
+    gdgs = ZOAUImportError(traceback.format_exc())
 
 REPRO = """  REPRO INDATASET({}) -
     OUTDATASET({}) REPLACE """
@@ -216,6 +225,49 @@ def copy_pds2uss(src, dest, is_binary=False, asa_text=False):
     return rc, out, err
 
 
+def copy_gdg2uss(src, dest, is_binary=False, asa_text=False):
+    """Copy a whole GDG to a USS path.
+
+    Parameters
+    ----------
+    src : str
+        The MVS data set to be copied, it must be a generation data group.
+    dest : str
+        The destination USS path.
+
+    Keyword Parameters
+    ------------------
+    is_binary : bool
+        Whether the file to be copied contains binary data.
+    asa_text : bool
+        Whether the file to be copied contains ASA control
+        characters.
+
+    Returns
+    -------
+    bool
+        True if all copies were successful, False otherwise.
+    """
+    src_view = gdgs.GenerationDataGroupView(src)
+    generations = src_view.generations()
+
+    copy_args = {
+        "options": ""
+    }
+
+    if is_binary or asa_text:
+        copy_args["options"] = "-B"
+
+    for gds in generations:
+        dest_file = path.join(dest, gds.name)
+        rc = datasets.copy(gds.name, dest_file, **copy_args)
+
+        if rc != 0:
+            return False
+
+    return True
+
+
 def copy_uss2uss_binary(src, dest):
     """Copy a USS file to a USS location in binary mode.
 

plugins/module_utils/data_set.py

@@ -343,6 +343,73 @@ def allocate_model_data_set(ds_name, model, executable=False, asa_text=False, vo
         if rc != 0:
             raise MVSCmdExecError(rc, out, err)
 
+    @staticmethod
+    def allocate_gds_model_data_set(ds_name, model, executable=False, asa_text=False, vol=None):
+        """
+        Allocates a new current generation of a generation data group using a model
+        data set to set its attributes.
+
+        Parameters
+        ----------
+        ds_name : str
+            Name of the data set that will be allocated. It must be a GDS
+            relative name.
+        model : str
+            The name of the data set whose allocation parameters
+            should be used to allocate the new data set.
+        executable : bool, optional
+            Whether the new data set should support executables.
+        asa_text : bool, optional
+            Whether the new data set should support ASA control
+            characters (have record format FBA).
+        vol : str, optional
+            The volume where the new data set should be allocated.
+
+        Returns
+        -------
+        str
+            Absolute name of the newly allocated generation data set.
+
+        Raises
+        ------
+        DatasetCreateError
+            When the allocation fails.
+        """
+        model_attributes = datasets.list_datasets(model)[0]
+        dataset_type = model_attributes.organization
+        record_format = model_attributes.record_format
+
+        if executable:
+            dataset_type = "library"
+        elif dataset_type in DataSet.MVS_SEQ:
+            dataset_type = "seq"
+        elif dataset_type in DataSet.MVS_PARTITIONED:
+            dataset_type = "pdse"
+
+        if asa_text:
+            record_format = "fba"
+        elif executable:
+            record_format = "u"
+
+        data_set_object = MVSDataSet(
+            name=ds_name,
+            data_set_type=dataset_type,
+            state="absent",
+            record_format=record_format,
+            volumes=vol,
+            block_size=model_attributes.block_size,
+            record_length=model_attributes.record_length,
+            space_primary=model_attributes.total_space,
+            space_type=''
+        )
+
+        success = data_set_object.ensure_present()
+        if not success:
+            raise DatasetCreateError(
+                data_set=ds_name,
+                msg=f"Error while trying to allocate {ds_name}."
+            )
+
     @staticmethod
     def data_set_cataloged(name, volumes=None):
         """Determine if a data set is in catalog.