Merge pull request #593 from oracle-devrel/pob-parkinglot

Parking Lot pattern OIC Gen3 solution template initial version

Author: MarcelStraka

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/LICENSE

@@ -0,0 +1,35 @@
+Copyright (c) 2023 Oracle and/or its affiliates.
+
+The Universal Permissive License (UPL), Version 1.0
+
+Subject to the condition set forth below, permission is hereby granted to any
+person obtaining a copy of this software, associated documentation and/or data
+(collectively the "Software"), free of charge and under any and all copyright
+rights in the Software, and any and all patent rights owned or freely
+licensable by each licensor hereunder covering either (i) the unmodified
+Software as contributed to or provided by such licensor, or (ii) the Larger
+Works (as defined below), to deal in both
+
+(a) the Software, and
+(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+one is included with the Software (each a "Larger Work" to which the Software
+is contributed by such licensors),
+
+without restriction, including without limitation the rights to copy, create
+derivative works of, display, perform, and distribute the Software and make,
+use, sell, offer for sale, import, export, have made, and have sold the
+Software and the Larger Work(s), and to sublicense the foregoing rights on
+either these or other terms.
+
+This license is subject to the following condition:
+The above copyright notice and either this complete permission notice or at
+a minimum a reference to the UPL must be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/README.md

@@ -0,0 +1,20 @@
+# OIC Resilient Batch Processing
+ 
+This asset contains accelerator type OIC Gen3 implementation of the resilient batch processing on top of Parking Lot Pattern. This pattern address one specific need when using Oracle Integration Cloud OIC for processing many batches. The repository provides the OIC Gen3 project with sample integrations and ATP scripts required for the implementation described in the [README.md](./files/oic-parkinglot/README.md).
+
+# When to use this asset?
+ 
+Use this asset as an example respectively as the template for the resilient batch services solution. It is good to update the model based on implemented business, capabilities of the target systems and operatioonal and monitoring requirements.
+ 
+# How to use this asset?
+ 
+This asset is provided as the resilient batch processing solution template.
+Follow the instructions in the [README.md](./files/oic-parkinglot/README.md).
+ 
+# License
+
+Copyright (c) 2023 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/files/Batch to API.pdf

@@ -0,0 +1,14 @@
+Copyright (c) 2023, Oracle and/or its affiliates. 
+Licensed under The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+Pull requests can be made under The Oracle Contributor Agreement (OCA).
+
+For pull requests to be accepted, the bottom of your commit message must have the following line using your name and e-mail address as it appears in the OCA Signatories list.
+Signed-off-by: Peter Obert <[email protected]>
+
+
+This can be automatically added to pull requests by committing with:
+git commit --signoff
+
+
+Only pull requests from committers that can be verified as having signed the OCA can be accepted.

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/files/oic-parkinglot/CONTRIBUTING.md

@@ -0,0 +1,36 @@
+Copyright (c) 2023, Oracle and/or its affiliates.
+
+The Universal Permissive License (UPL), Version 1.0
+
+Subject to the condition set forth below, permission is hereby granted to any
+person obtaining a copy of this software, associated documentation and/or data
+(collectively the "Software"), free of charge and under any and all copyright
+rights in the Software, and any and all patent rights owned or freely
+licensable by each licensor hereunder covering either (i) the unmodified
+Software as contributed to or provided by such licensor, or (ii) the Larger
+Works (as defined below), to deal in both
+
+(a) the Software, and
+(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+one is included with the Software (each a "Larger Work" to which the Software
+is contributed by such licensors),
+
+without restriction, including without limitation the rights to copy, create
+derivative works of, display, perform, and distribute the Software and make,
+use, sell, offer for sale, import, export, have made, and have sold the
+Software and the Larger Work(s), and to sublicense the foregoing rights on
+either these or other terms.
+
+This license is subject to the following condition:
+The above copyright notice and either this complete permission notice or at
+a minimum a reference to the UPL must be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/files/oic-parkinglot/LICENSE.txt

@@ -0,0 +1,78 @@
+-- ADMIN:Parking12345#
+-- Copyright (c) 2020, Oracle and/or its affiliates.
+-- Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
+--------------------------------------------------------
+--  DDL for Table PAYLOAD_PARKING_LOT_TAB
+--------------------------------------------------------
+create sequence "ADMIN"."sqname_id_seq" start with 1 increment by 1 nocache;
+drop table "ADMIN"."PAYLOAD_PARKING_LOT_TAB";
+CREATE TABLE "ADMIN"."PAYLOAD_PARKING_LOT_TAB" 
+   (	"ID" NUMBER(*,0) DEFAULT "ADMIN"."sqname_id_seq".nextval, -- BATCHID
+	"GRPID" VARCHAR2(40 BYTE) COLLATE "USING_NLS_COMP", 
+	"GRPTYPE" VARCHAR2(40 BYTE) COLLATE "USING_NLS_COMP", 
+	"GRPSEQ" NUMBER(*,0), 
+	"PAYLOAD_CLOB" CLOB COLLATE "USING_NLS_COMP", 
+	"PAYLOAD" VARCHAR2(10000 BYTE) COLLATE "USING_NLS_COMP", 
+	"FILENAME" VARCHAR2(40 BYTE) COLLATE "USING_NLS_COMP", 
+	"DIRECTORY" VARCHAR2(200 BYTE) COLLATE "USING_NLS_COMP", 
+	"STATUS" VARCHAR2(20 BYTE) COLLATE "USING_NLS_COMP", 
+		-- NEW - new request/batch
+		-- SUBMITTED - batch processing triggered
+		-- PROCESSING - batch contitions passed and processing started
+		-- FAILED - request/batch processing failed
+		-- ERROR_RETRY - request/batch ready to be re-processed
+    "CREATIONTIME" TIMESTAMP(3) NOT NULL,
+    "STATUSTIME"   TIMESTAMP(3) NOT NULL,
+    "ERROR" VARCHAR2(2000 BYTE) COLLATE "USING_NLS_COMP"
+   )  DEFAULT COLLATION "USING_NLS_COMP" ;
+
+CREATE UNIQUE INDEX "ADMIN"."PK_ID" ON "ADMIN"."PAYLOAD_PARKING_LOT_TAB" ("ID") 
+  ;
+ALTER TABLE "ADMIN"."PAYLOAD_PARKING_LOT_TAB" ADD CONSTRAINT "PK_ID" PRIMARY KEY ("ID");
+-- 
+--drop table "ADMIN"."PAYLOAD_ERRORS_TAB";
+create sequence "ADMIN"."error_id_seq" start with 1 increment by 1 nocache;
+CREATE TABLE "ADMIN"."PAYLOAD_ERRORS_TAB" 
+   ( "ERRID" INT DEFAULT "ADMIN"."error_id_seq".nextval, 
+	"BATCHID" INT NOT NULL,
+	"GRPID" VARCHAR2(40 BYTE),
+	"GRPTYPE" VARCHAR2(40 BYTE),
+      "GRPSEQ" INT,
+	"PAYLOAD_CLOB" CLOB, 
+	"PAYLOAD" VARCHAR2(10000 BYTE) COLLATE "USING_NLS_COMP", 
+	"STATUS" VARCHAR2(20 BYTE) COLLATE "USING_NLS_COMP", 
+       -- NEW after registering by initial BATCH
+       -- READY after fixing the payload
+       -- Whatewer else after failure
+	"ERROR" VARCHAR2(2000 BYTE) COLLATE "USING_NLS_COMP",
+      "CREATIONTIME" TIMESTAMP(3) NOT NULL,
+  	"STATUSTIME"   TIMESTAMP(3) NOT NULL
+   )  ;
+CREATE UNIQUE INDEX "ADMIN"."PKERR_ID" ON "ADMIN"."PAYLOAD_ERRORS_TAB" ("ERRID");
+ALTER TABLE "ADMIN"."PAYLOAD_ERRORS_TAB" ADD CONSTRAINT "PKERR_ID" PRIMARY KEY ("ERRID");
+
+--drop table "ADMIN"."BATCH_STATISTICS";
+CREATE TABLE "ADMIN"."BATCH_STATISTICS" 
+   ( 	
+    "BATCHID" INT NOT NULL,
+	"GRPID" VARCHAR2(40 BYTE),
+	"GRPTYPE" VARCHAR2(40 BYTE),
+      "GRPSEQ" INT,
+	"TOTAL_ROWS" INT,
+	"PROCESSED_ROWS" INT,
+	"SUCCEEDED_ROWS" INT,
+	"FAILED_ROWS" INT,
+	"CREATIONTIME" TIMESTAMP(3) NOT NULL,
+  	"STATUSTIME"   TIMESTAMP(3) NOT NULL,
+	"STATUS" VARCHAR2(20 BYTE) COLLATE "USING_NLS_COMP", 
+	"ERROR" VARCHAR2(2000 BYTE) COLLATE "USING_NLS_COMP"
+ );
+CREATE UNIQUE INDEX "ADMIN"."BATCH_STATISTICS_PK_ID" ON "ADMIN"."BATCH_STATISTICS" ("BATCHID");
+ALTER TABLE "ADMIN"."BATCH_STATISTICS" ADD CONSTRAINT "BATCHID" PRIMARY KEY ("BATCHID");
+
+
+------------------------------------------------------------------------------------------------------
+
+SELECT * FROM ADMIN.PAYLOAD_PARKING_LOT_TAB ORDER BY ID DESC;
+SELECT * FROM ADMIN.BATCH_STATISTICS  ORDER BY BATCHID DESC;
+SELECT * FROM ADMIN.PAYLOAD_ERRORS_TAB ORDER BY ERRID DESC;

ai-and-app-modernisation/app-integration-and-automation/oracle-integration-cloud/04-oic-resilient-batch/files/oic-parkinglot/OIC_GEN3_PROJECT/BATCH_PARKINGLOT.car

@@ -0,0 +1,140 @@
+# cloud.asset.integration-oic-parkinglotpattern-batch
+
+## Introduction
+
+In this Project, we will illustrate a simple practical example of using the parking lot pattern with Oracle Integration Cloud (OIC) and batch processing style.
+The example will use Autonomous Transaction Processing ([ATP]((https://docs.oracle.com/en/cloud/paas/atp-cloud/index.html))) database for a parking lot table.
+
+This project implements extension of the Resilient Asynchronous Integrations suitable for processing, managing and monitoring pattern based batch style integrations. 
+
+The solution is build as the project for [OIC Gen3](https://docs.oracle.com/en-us/iaas/application-integration/index.html). The solution artifacts contains [cloud.asset.integration-oic-parkinglotpattern](https://github.com/oracle/cloud-asset-integration-oic-parkinglotpattern-sample) artifacts migrated to OIC Gen3 as well.
+
+Refer to the blog - [OIC Advanced Error Handling ](https://www.ateam-oracle.com/advanced-error-handling-oic), for a detailed discussion of the Integration design pattern and solution.
+
+Refer to best practice article - [Best practices for building resilient asynchronous integrations](https://docs.oracle.com/en/solutions/best-practices-resilient-asynch-integrations/index.html#GUID-B18DDA79-78FD-4767-BEE6-DB213B5EC073).
+
+The repository provides the OIC Gen3 sample integrations and ATP scripts required for the implementation described in the blog.
+
+## How to Install and Run
+* Download the Car file (within zip archives) from this project and import into target OIC Gen3 environment
+* Configure the required connections and activate the integrations
+* Download the parking lot table creation DDL script file and execute on target database.
+* Refer to the blog - [OIC Advanced Error Handling ](https://www.ateam-oracle.com/advanced-error-handling-oic), for more detailed instructions.
+
+### High Level Design 
+ Figure below shows the high level design of solution using OIC flows and [ATP DB](https://docs.oracle.com/en/cloud/paas/atp-cloud/index.html) for parking lot table. The OIC solution use Persister, Dispatcher and Processor flows.
+ 
+ ![High Level Design Diagram](./images/OICextended_parkinglot_EH_v0.1.png)
+
+  **Batch processing scenario** is implemented to reflect following processing steps:
+  1. HCM produces CSV File with Employee Bank Connection Information
+  2. Batch “Requestor” integration flow will download file from source e.e.g. File Server/Storage/UCM and Register the Batch Information (opt. de-batch)
+  3. Batch Dispatcher will select from the parked batches which process is ready to start and submits batch
+  4. Batch Processor process the CSV File and documents processing, handles error in Payroll API provisioning and Notifies Result
+
+ ![Batch processing scenario](./images/HCM_To_Payroll.drawio.png)
+
+ ### Solution Components
+The [Parking Lot pattern](https://www.ateam-oracle.com/oic-error-handling-guide) here uses an ATP table to store request metadata and status information. The OIC Flows in the design are inherited from the original OIC Gen2 resilient services
+ * Request Persister
+ * Scheduled Dispatcher
+ * Async Processor
+
+**For the resilient batch processing additional OIC Flows added**
+ * Batch Requests Persister responsible for the Batch Registration and parking the processing
+  *SR_BulkDownload_RequestPersister_ATP*
+     - Batch Request Registration Provider template schedulable service which scans "Remote FTP Server, transfers the batch CSV file to Local FTP and registers/parks the Batch in the ATP Parking Lot Table.
+
+ * Batch Dispatcher for CSV Batch responsible for the submitting the best candidate from the registered and parked batches
+  *SR_ScheduledDispatcher_CSVBatch*
+     - Dispatcher fetches all the ready/parked batches for the CSV Batch Group Type and triggers one way batch processors. Updates Batch Status to Submitted
+
+ * Async Batch Processor responsible for the batch processing in our case it is HCM To Payroll provisioning
+   *SR_OneWay_Processor_HCM_To_Payroll*
+   - Example of the one-way processor that processes all the submitted batch rows. In this case it it is example of HCM Bank Connections to Payroll. It invokes Payroll Mock Service which represents Payroll API for the employee bank connection
+
+ * Resubmission Processor for the fixed and ready payloadsprovisioning
+
+   *SR_ScheduledDispatcher_PayrollErrors*
+   - Example of the Resubmission Endpoint for the Failed processed rows from the Batch processing. It fetches READY state records form the Failed Payloads table and invokes the row processing service.
+
+![Batch processing sequence scenario](./images/HCM_To_Payroll.seq.png)
+ 
+### Artifacts:
+
+#### DDL script for ATP table
+Parking lot table creation DDL script for sample 
+
+#### OIC Flows - Integration Archive - OIC Gen3 Project File (CAR)
+Download the CAR file from this project and import into target OIC environment. Configure the required connections before the integrations activation
+
+#### sample account json file
+Sample json file for a simple account creation request - resilient single payload service
+
+#### sample csv files 
+Sample csv files for demonstrating batch with/without errors
+
+#### opaque schema file
+XSD schema file used for persisting json payload into Db parking lot table using Stage file
+ 
+Refer to the blog - [OIC Advanced Error Handling ](https://www.ateam-oracle.com/advanced-error-handling-oic), for a detailed discussion of the Integration design pattern and solution.
+
+Refer to the blog - [Batch to API - Design for Resiliency Using Oracle Integration](https://www.linkedin.com/posts/ingpeterobert_oic-parkingabrlot-oraclecloud-activity-7065351235192680449-8Bk2?utm_source=share&utm_medium=member_desktop), for the resilient batch metamodel.
+
+## Payload examples of the last mile API - Batch processor and Payroll Bank Connection API
+
+### ./payloads/hcm_bnk_connsX4long.csv - sample file which should pass with no errors
+
+### ./payloads/hcm_bnk_connsNoError4long.csv - sample file containing errors(small employee numbers)
+
+### ./payloads/*.json
+
+#### { "PayrollAPIRequest":
+	{
+	"employeeNumber":123456789,
+	"iban":"GB29 NWBK 6016 1331 9268 19",
+	"bankIdentifier": "NWBK",
+	"branchIdentifier": "601613",
+	"accountNumber": "31926819"
+	}
+}
+
+#### {  "PayrollAPIResponse":
+	{
+	"status": "SUCCESS",
+	"errorCode":"BRS-200",
+	"errorMessage": "Possible Error"
+	}
+}
+
+
+#
+----------------------------------------------------------
+## Batch processing finished - notification email template
+
+<html>
+<title>Batch Processing Statistics</title>
+
+<head>
+</head>
+
+<body>
+    <h1>Batch Processing Statistics</h1>
+    <div>---------------------------------</div>
+    <div>BatchID: {batchid}</div>
+<div>---------------------------------</div>
+<div>Filename: {filename}</div>
+<div>Employee connections rows: {rows}</div>
+<div>Processed rows: {aPICalls}</div>
+<div>Succeeded API Calls: {aPISuccess}</div>
+<div>Failed API Calls: {aPIFailed}</div>
+<div>ErrorFile: {errorFile}</div>
+</body>
+
+</html>
+
+#
+
+Copyright (c) 2023, Oracle and/or its affiliates.
+Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
+