Merge pull request #626 from IBM/develop

Release zAppBuild 3.11.0

Author: dennis-behm

README.md

@@ -1,24 +1,30 @@
 # zAppBuild
+
 zAppBuild is a generic build solution for building z/OS applications using Apache Groovy build scripts and IBM Dependency Based Build (DBB) APIs.
 
 ## Resources
+
 * [IBM Dependency Based Build Product Page](https://www.ibm.com/products/dependency-based-build)
-* [IBM DBB Knowledge Center](https://www.ibm.com/docs/en/dbb/1.1.0)
+* [IBM DBB Knowledge Center](https://www.ibm.com/docs/en/adffz/dbb/3.0.0)
 * [IBM/dbb Repository](https://github.com/IBM/dbb/)
 * [IBM IDZ Community](https://community.ibm.com/community/user/ibmz-and-linuxone/groups/topic-home?CommunityKey=f461c55d-159c-4a94-b708-9f7fe11d972b)
 * [IBM DBB Community](https://community.ibm.com/community/user/ibmz-and-linuxone/groups/topic-home?CommunityKey=20c9b889-9450-4ab6-8f11-8a5eb2b3342d)
 
 
 ## Contributing
+
 For instructions on how to contribute enhancements and bug fixes to zAppBuild, please read the [Contributions Guidelines](CONTRIBUTIONS.md).
 
 ## How zAppBuild works
+
 The zAppBuild repository is intended to be cloned to a single location on Unix Systems Services (USS) and used to build all of your z/OS applications. Global configuration properties are configured in the properties files in the [build-conf](build-conf/) directory. Specifying application-level properties is done by simply copying the supplied `application-conf` folder (located in the [samples folder](samples)) to the application source repository you want to build and then verify/update the contained default configuration property values to ensure they meet the build requirements of your application. See the included [MortgageApplication](samples/MortgageApplication) sample for an example of an application that has been modified to be built by zAppBuild.  
 
 **IMPORTANT** : The [datasets.properties](build-conf/datasets.properties) must be configured for your build machine before executing a build!  See [build-conf/README.md](build-conf/README.md) for more information.
 
 ## Supported Languages
+
 The zAppBuild sample provides the following *language* build scripts by default:
+
 * Assembler.groovy
 * BMS.groovy
 * Cobol.groovy
@@ -37,11 +43,13 @@ The zAppBuild sample provides the following *language* build scripts by default:
 All language scripts both compile and optionally link-edit programs. The language build scripts are intended to be useful out of the box but depending on the complexity of your applications' build requirements, may require modifications to meet your development team's needs.  By following the examples used in the existing language build scripts of keeping all application specific references out of the build scripts and instead using configuration properties with strong default values, the zAppBuild sample can continue to be a generic build solution for all of your specific applications.
 
 ## Build Scope
+
 The build scope of zAppBuild is an application which is loosely defined as one or more Git repositories containing all the z/OS source files required to build the application.  There are no specific rules as to the structure of the repositories except that one repository must contain the high level `application-conf` folder provided by zAppBuild which contains all of the configuration properties for building the application programs.  
 
 **NOTE:** All source repositories that make up the application must be cloned on the build machine under a common *workspace*  directory prior to calling build.groovy.
 
 zAppBuild supports a number of build scenarios:
+
 * **Single Program** - Build a single program in the application.
 * **List of Programs** - Build a list of programs provided by a text file.
 * **Full Build** - Build all programs (or buildable files) of an application.
@@ -51,8 +59,9 @@ zAppBuild supports a number of build scenarios:
 * **Merge Build** - Build only changed programs which will be merged back into the mainBuildBranch by using a triple-dot git diff. 
 * **Scan Source** - Skip the actual building and only scan source files to store dependency data in collection (migration scenario).
 * **Scan Source + Outputs** - Skip the actual building and only scan source files and existing load modules to dependency data in source and output collection (migration scenario with static linkage scenarios).
-* **Build Preview** - Supplemental build option. Process all phases of the supplied build option, but will not execute the commands. A build report and a build result are generated with a specific status that excludes them in subsequent impact build calculations. 
+* **Build Preview** - Supplemental build option. Process all phases of the supplied build option, but will not execute the commands. A build report and a build result are generated with a specific status that excludes them in subsequent impact build calculations.
 
+To check how to get started with zAppBuild, see [Getting started with zAppbuild](./docs/GettingStarted.md) section.
 
 Instructions on invoking a zAppBuild is included in [docs/BUILD.md](docs/BUILD.md) as well as invocation samples for the above mentioned build scenarios including sample console logs.
 
@@ -63,6 +72,7 @@ zAppBuild comes with a set of reporting features. It helps development teams to
 Links to additional documentation is provided in the table below.  
 
 ## Repository Legend
+
 Folder/File | Description | Documentation Link
 --- | --- | ---
 build-conf | This folder contains global configuration properties used by build.groovy and language build scripts. | [build-conf/README.md](build-conf/README.md)

build-conf/ACBgen.properties

@@ -21,7 +21,7 @@ acbgen_loadPDS=${hlq}.ACBLIB
 acbgen_loadDatasets=${acbgen_loadPDS}
 acbgen_loadOptions=cyl space(5,5) dir(40) dsorg(PO) recfm(U) blksize(32760) dsntype(PDS)
 
-acbgen_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+acbgen_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 
 # List of output datasets to document deletions
 acbgen_outputDatasets=${acbgen_loadPDS}

build-conf/Assembler.properties

@@ -36,7 +36,7 @@ assembler_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(l
 
 #
 # temporary data set allocations
-assembler_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+assembler_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 
 #
 # sysadata data set creation options
@@ -67,14 +67,23 @@ assembler_linkEditor=IEWBLINK
 # additional libraries for linkEdit SYSLIB concatenation, comma-separated, see definitions in application-conf
 # assembler_linkEditSyslibConcatenation=
 
-# assembler_dependenciesDatasetMapping - an optional dbb property mapping to map dependencies to different target datasets
-#  this property is used when dependencies are copied to the different build libraries, e.q dclgens going into to a dedicated library
-#  note, that a dependency file needs to match a single rule, target dataset definitions need to exist
+# assembler_dependenciesDatasetMapping - an optional property mapping to map dependencies to different target datasets
+#  this property is used when dependencies are copied to the different build libraries, e.q assembler copy going into to a dedicated library
 #
-#  sample:
-#   assembler_dependenciesDatasetMapping = assembler_macroPDS :: **/macro/*.mac
+#  default copies all dependencies into the dependency dataset definition
+#   assembler_dependenciesDatasetMapping = assembler_macroPDS
+#
+#  sample to override for a subset of dependencies using DBB file properties:
 #   assembler_dependenciesDatasetMapping = assembler_cpyPDS :: **/copy/*.asmcpy
 #
-#  default copies all dependencies into the dependency dataset definition which was previously passed to the utilities/BuildUitilities.copySourceFiles method
-#   assembler_dependenciesDatasetMapping = assembler_macroPDS :: **/* 
-assembler_dependenciesDatasetMapping = assembler_macroPDS :: **/*
+assembler_dependenciesDatasetMapping = assembler_macroPDS
+
+# assembler_dependenciesCopyMode defines the copy mode to copy files from the build workspace into the target libraries
+#  default sets the mode to TEXT transfer
+#   assembler_dependenciesCopyMode = TEXT
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#  copy resolved object decks as binary object:
+#   assembler_dependenciesCopyMode = BINARY :: **/*.OBJ
+#
+assembler_dependenciesCopyMode = TEXT

build-conf/BMS.properties

@@ -23,7 +23,7 @@ bms_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(library)
 bms_loadDatasets=${bms_loadPDS}
 bms_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
 
-bms_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+bms_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 
 # List of output datasets to document deletions
 bms_outputDatasets=${bms_cpyPDS},${bms_loadPDS}

build-conf/CPP.properties

@@ -0,0 +1,61 @@
+# Releng properties used by language/cc.groovy
+
+#
+# Comma separated list of required build properties for language/cc.groovy
+cpp_requiredBuildProperties=cpp_srcPDS,cpp_objPDS,cpp_loadPDS, \
+cpp_compiler,cpp_linkEditor,cpp_tempOptions,cpp_compileMaxRC,SCCNCMP\
+
+
+#
+# CC compiler name
+cpp_compiler=CCNDRVR
+
+# CC linker name
+cpp_linkEditor=IEWBLINK
+
+#
+# cc source data sets
+cpp_srcPDS=${hlq}.CPP
+cpp_headerPDS=${hlq}.CPP.H
+cpp_objPDS=${hlq}.OBJ
+
+#
+# cc load data sets
+cpp_loadPDS=${hlq}.LOAD
+
+# List the data sets that need to be created and their creation options
+cpp_srcDatasets=${cpp_srcPDS},${cpp_headerPDS},${cpp_objPDS}
+
+cpp_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(library)
+
+cpp_loadDatasets=${cpp_loadPDS}
+cpp_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
+
+cpp_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
+cpp_tempListOptions=cyl space(15,15)  lrecl(137) recfm(v,b) new
+
+#
+# List of output datasets to document deletions
+cpp_outputDatasets=${cpp_loadPDS}
+
+# CC scanner language hint
+dbb.DependencyScanner.languageHint=C :: **/*.c, **/*.cpp, **/*.h
+
+# cpp_dependenciesDatasetMapping - an optional dbb property mapping to map dependencies to different target datasets
+#  this property is used when dependencies are copied to the different build libraries, e.q dclgens going into to a dedicated library
+#  note, that a dependency file needs to match a single rule
+#
+#  default copies all dependencies into the dependency dataset definition
+#   cpp_dependenciesDatasetMapping = cpp_headerPDS
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#  copy dclgens into a different library:
+#   cpp_dependenciesDatasetMapping = cpp_dclgensPDS :: **/dclgens/*.h
+cpp_dependenciesDatasetMapping = cpp_headerPDS
+
+# cpp_dependenciesCopyMode defines the copy mode to copy files from the build workspace into the target libraries
+#  default sets the mode to TEXT transfer
+#   cpp_dependenciesCopyMode = TEXT
+#
+# Can be overridden for a subset of dependencies using DBB file properties.
+cpp_dependenciesCopyMode = TEXT

build-conf/Cobol.properties

@@ -23,6 +23,12 @@ cobol_objPDS=${hlq}.OBJ
 cobol_dbrmPDS=${hlq}.DBRM
 cobol_BMS_PDS=${team}.BMS.COPY
 
+#
+# data sets used to pass in external dependencies 
+# both for compile and link steps 
+cobol_import_cpyPDS=${hlq}.COPY.IMP
+cobol_import_objPDS=${hlq}.OBJ.IMP
+
 #
 # COBOL load data sets
 cobol_loadPDS=${hlq}.LOAD
@@ -38,13 +44,17 @@ cobol_testcase_loadPDS=${hlq}.TEST.LOAD
 #
 # List the data sets that need to be created and their creation options
 cobol_srcDatasets=${cobol_srcPDS},${cobol_cpyPDS},${cobol_objPDS},${cobol_dbrmPDS}
+# When zAppBuild is enabled to resolve dependencies from external archives,
+#  the datasets referenced by 'cobol_cpyImpPDS' and 'cobol_objImpPDS' need to be added to be created
+# cobol_srcDatasets=${cobol_srcPDS},${cobol_cpyPDS},${cobol_objPDS},${cobol_dbrmPDS},${cobol_import_cpyPDS},${cobol_import_objPDS}
+
 cobol_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(library)
 
 cobol_loadDatasets=${cobol_loadPDS}
 cobol_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
 
-cobol_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
-cobol_printTempOptions=cyl space(5,5) unit(vio) blksize(133) lrecl(133) recfm(f,b) new
+cobol_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
+cobol_printTempOptions=cyl space(5,5) unit(vio) lrecl(133) recfm(f,b) new
 
 
 #
@@ -81,12 +91,42 @@ cobol_dependenciesAlternativeLibraryNameMapping=
 
 # cobol_dependenciesDatasetMapping - an optional dbb property mapping to map dependencies to different target datasets
 #  this property is used when dependencies are copied to the different build libraries, e.q dclgens going into to a dedicated library
-#  note, that a dependency file needs to match a single rule
 #
-#  sample:
-#   cobol_dependenciesDatasetMapping = cobol_cpyPDS :: **/copybook/*.cpy
+#  default copies all dependencies into the dependency dataset definition
+#   cobol_dependenciesDatasetMapping = cobol_cpyPDS
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#  copy dclgens into a different library:
 #   cobol_dependenciesDatasetMapping = cobol_dclgenPDS :: **/dclgens/*.cpy
 #
-#  default copies all dependencies into the dependency dataset definition which was previously passed to the utilities/BuildUitilities.copySourceFiles method
-#   cobol_dependenciesDatasetMapping = cobol_cpyPDS :: **/* 
-cobol_dependenciesDatasetMapping = cobol_cpyPDS :: **/*
+cobol_dependenciesDatasetMapping = cobol_cpyPDS
+# target the object dataset for any dependencies with file extension OBJ
+cobol_dependenciesDatasetMapping = cobol_import_objPDS :: **/*.OBJ
+
+# cobol_compileDependenciesDatasets - comma-separated list of additional datasets containing
+#  dependencies resolved from the 'import' directory external for the compile step
+#   OR 
+#  dependencies that got loaded into different target datasets via the
+#   cobol_dependenciesDatasetMapping mapping
+#
+# these data sets are then added to the SYSLIB concatenation for the compile step
+# 
+cobol_compileDependenciesDatasets = ${cobol_import_cpyPDS}
+
+# cobol_linkEditDependenciesDatasets - comma-separated list of datasets containing
+#  dependencies resolved from the 'import' directory external for the linkEdit step
+#
+# These data sets are then added to the SYSLIB concatenation
+# 
+cobol_linkEditDependenciesDatasets = ${cobol_import_objPDS}
+
+# cobol_dependenciesCopyMode defines the copy mode to copy files from the build workspace into the target libraries
+#  default sets the mode to TEXT transfer
+#   cobol_dependenciesCopyMode = TEXT
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#  copy resolved object decks as binary object:
+#   cobol_dependenciesCopyMode = BINARY :: **/*.OBJ
+cobol_dependenciesCopyMode = TEXT
+# target the object dataset for any dependencies with file extension OBJ
+cobol_dependenciesCopyMode = BINARY :: **/*.OBJ

build-conf/DBDgen.properties

@@ -23,7 +23,7 @@ dbdgen_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(library)
 dbdgen_loadDatasets=${dbdgen_loadPDS}
 dbdgen_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
 
-dbdgen_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+dbdgen_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 
 # Allocation of SYSMLSD Dataset used for extracting Compile Messages to Remote Error List
 dbdgen_compileErrorFeedbackXmlOptions=tracks space(200,40) dsorg(PS) blksize(27998) lrecl(16383) recfm(v,b) new keep

build-conf/Easytrieve.properties

@@ -36,10 +36,30 @@ easytrieve_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(libr
 easytrieve_loadDatasets=${easytrieve_loadPDS}
 easytrieve_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
 
-easytrieve_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
-easytrieve_printTempOptions=cyl space(5,5) unit(vio) blksize(133) lrecl(133) recfm(f,b) new
+easytrieve_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
+easytrieve_printTempOptions=cyl space(5,5) unit(vio) lrecl(133) recfm(f,b) new
 
 #
 # List of output datasets to document deletions
 easytrieve_outputDatasets=${easytrieve_loadPDS}
 
+# easytrieve_dependenciesDatasetMapping - an optional property mapping to map dependencies to different target datasets
+#  this property is used when dependencies are copied to the different build libraries, e.q easytrieve copys going into to a dedicated library
+#
+#  default copies all dependencies into the dependency dataset definition
+#   easytrieve_dependenciesDatasetMapping = easytrieve_cpyPDS
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#   easytrieve_dependenciesDatasetMapping = easytrieve_macPDS :: **/copy/*.etzmac
+#
+easytrieve_dependenciesDatasetMapping = easytrieve_cpyPDS
+
+# easytrieve_dependenciesCopyMode defines the copy mode to copy files from the build workspace into the target libraries
+#  default sets the mode to TEXT transfer
+#   easytrieve_dependenciesCopyMode = TEXT
+#
+#  sample to override for a subset of dependencies using DBB file properties:
+#   easytrieve_dependenciesCopyMode = BINARY :: **/*.OBJ
+#
+easytrieve_dependenciesCopyMode = TEXT
+

build-conf/GlobalOptions.properties

@@ -1,5 +1,5 @@
 # Releng properties used by zAppBuild
 
 # Default allocation options for in-stream DD statements
-global_instreamDataTempAllocation=tracks space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+global_instreamDataTempAllocation=tracks space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 

build-conf/LinkEdit.properties

@@ -27,7 +27,7 @@ linkedit_srcOptions=cyl space(1,1) lrecl(80) dsorg(PO) recfm(F,B) dsntype(librar
 linkedit_loadDatasets=${linkedit_loadPDS}
 linkedit_loadOptions=cyl space(1,1) dsorg(PO) recfm(U) blksize(32760) dsntype(library)
 
-linkedit_tempOptions=cyl space(5,5) unit(vio) blksize(80) lrecl(80) recfm(f,b) new
+linkedit_tempOptions=cyl space(5,5) unit(vio) lrecl(80) recfm(f,b) new
 
 #
 # List of output datasets to document deletions