RHIDP-8641: Building and Deploying Serverless Workflows

artifacts/attributes.adoc

@@ -151,6 +151,8 @@
 :observability-category-link: {product-docs-link}/#Observability
 :ocp-docs-link: link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-version}
 :odf-docs-link: link:https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/{ocp-version}
+:orchestrator-book-link: {product-docs-link}/html-single/orchestrator_in_red_hat_developer_hub/index
+:orchestrator-book-title: Orchestrator in {product}
 :osd-docs-link: link:https://docs.redhat.com/en/documentation/openshift_dedicated/{osd-version}
 :release-notes-book-link: {product-docs-link}/html-single/red_hat_developer_hub_release_notes/index
 :release-notes-book-title: {product} release notes

assemblies/assembly-building-and-deploying-serverless-workflows.adoc

@@ -0,0 +1,48 @@
+:_mod-docs-content-type: ASSEMBLY
+
+ifndef::context[]
+[id="assembly-building-and-deploying-serverless-workflows"]
+endif::[]
+ifdef::context[]
+[id="assembly-building-and-deploying-serverless-workflows"]
+endif::[]
+:context: orchestrator-rhdh
+= Build and deploy serverless workflows
+
+To deploy a workflow and make it available in the Orchestrator plugin, follow these main steps:
+
+* Building workflow images
+* Generating workflow manifests
+* Deploying workflows to a cluster
+
+This process moves the workflow from your local machine to deployment on a cluster.
+
+// why build workflow images
+include::modules/orchestrator/con-why-build-workflow-images.adoc[leveloffset=+1]
+
+// project structure
+include::modules/orchestrator/con-project-structure-overview.adoc[leveloffset=+2]
+
+// creating and running workflows locally
+include::modules/orchestrator/proc-creating-and-running-workflows.adoc[leveloffset=+2]
+
+// building locally and generating artifacts
+include::modules/orchestrator/proc-building-locally.adoc[leveloffset=+1]
+
+// the script and its uses
+include::modules/orchestrator/con-build-sh-script-and-its-uses.adoc[leveloffset=+2]
+
+// environment variables supported by the script
+include::modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc[leveloffset=+2]
+
+// required tools
+include::modules/orchestrator/con-required-tools.adoc[leveloffset=+2]
+
+// building the 01_basic workflow
+include::modules/orchestrator/proc-building-the-01-basic-workflow.adoc[leveloffset=+2]
+
+// generated workflow manifests
+include::modules/orchestrator/con-generated-workflow-manifests.adoc[leveloffset=+1]
+
+// deploy workflows on a cluster
+include::modules/orchestrator/proc-deploying-workflows-on-a-cluster.adoc[leveloffset=+1]

modules/orchestrator/con-build-sh-script-and-its-uses.adoc

@@ -0,0 +1,35 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-build-sh-script-and-its-uses.adoc_{context}"]
+= The `build-sh` script functionality and important flags
+
+The `build-sh` script does the following in order:
+
+* Generates workflow manifests using the `kn-workflow` CLI.
+* Builds the workflow image using `podman` or `docker`.
+* Optional: The script pushes the images to an image registry and deploys the workflow using `kubectl`.
+
+You can review the script configuration options and see available flags and their functions by accessing the help menu:
+
+[source,bash]
+----
+./scripts/build.sh [flags]
+----
+
+The following flags are essential for running the script:
+
+[cols="1,3", options="header"]
+|===
+|Flag |Description
+|`-i`, `--image` (required) |Full image path, for example, `quay.io/orchestrator/demo:latest`.
+|`-w`, `--workflow-directory` |Workflow source directory (default is the current directory).
+|`-m`, `--manifests-directory` |Where to save generated manifests.
+|`--push` |Push the image to the registry.
+|`--deploy` |Deploy the workflow.
+|`-h`, `--help` |Show the help message.
+|===
+
+[TIP]
+====
+The script also supports builder and runtime image overrides, namespace targeting, and persistence flags.
+====

modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc

@@ -0,0 +1,28 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-environment-variables-supported-by-the-build-script.adoc_{context}"]
+= Environment variables supported by the build script
+
+The `build-sh` script supports the following environment variables to customize the workflow build process without modifying the script itself:
+
+`QUARKUS_EXTENSIONS`::
+
+The `QUARKUS_EXTENSIONS` variable specifies additional link:https://quarkus.io/extensions/[Quarkus] extensions required by the workflow. This variable takes the format of a comma-separated list of fully qualified extension IDs as shown in the following example:
++
+[source,yaml]
+----
+export QUARKUS_EXTENSIONS="io.quarkus:quarkus-smallrye-reactive-messaging-kafka"
+----
++
+Add Kafka messaging support or other integrations at build time.
+
+`MAVEN_ARGS_APPEND`::
+
+The `MAVEN_ARGS_APPEND` variable appends additional arguments to the *Maven build* command. This variable takes the format of a string of Maven CLI arguments as shown in the following example:
++
+[source,yaml]
+----
+export MAVEN_ARGS_APPEND="-DmaxYamlCodePoints=35000000"
+----
++
+Control build behavior. For example, set `maxYamlCodePoints` parameter that controls the maximum input size for YAML input files to 35000000 characters (~33MB in UTF-8).

modules/orchestrator/con-generated-workflow-manifests.adoc

@@ -0,0 +1,74 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-generated-workflow-manifests.adoc_{context}"]
+= Generated workflow manifests
+
+The following example is an illustration of what is generated under the `01_basic/manifests`:
+
+[source,yaml]
+----
+01_basic/manifests
+├── 00-secret_basic-secrets.yaml
+├── 01-configmap_basic-props.yaml
+├── 02-configmap_01-basic-resources-schemas.yaml
+└── 03-sonataflow_basic.yaml
+----
+
+Here is a quick overview of each of these manifests:
+
+`00-secret_basic-secrets.yaml`::
+Contains secrets from `01_basic/src/main/resources/secret.properties`.
+Values are not required at this stage as you can set them later after applying CRs or when using GitOps.
+
+[Important]
+====
+In OpenShift Serverless Logic `v1.36`, after updating a secret, you must manually restart the workflow Pod for changes to apply.
+====
+
+`01-configmap_basic-props.yaml`::
+Holds application properties from application.properties.
+Any change to this ConfigMap triggers an automatic Pod restart.
+
+`02-configmap_01-basic-resources-schemas.yaml`::
+Contains JSON schemas from src/main/resources/schemas.
++
+[NOTE]
+====
+You do not need to deploy certain configuration resources when using the GitOps profile.
+====
+
+`03-sonataflow_basic.yaml`::
+The SonataFlow custom resource (CR) that defines the workflow.
+A few important parts:
+
+.The image and secrets mounting example:
+[source,yaml]
+----
+podTemplate:
+  container:
+    image: quay.io/orchestrator/demo-basic
+    resources: {}
+    envFrom:
+      - secretRef:
+          name: basic-secrets
+----
+
+.The persistence configuration example:
+[source,yaml,subs="+quotes"]
+----
+persistence:
+  postgresql:
+    secretRef:
+      name: `sonataflow-psql-postgresql` #  Replace with the Secret name of your deployment.
+      userKey: `__<your_postgres_username>__` # Replace with the key for your deployment.
+      passwordKey: `__<your_postgres_password>__` # Replace with the password of your deployment.
+    serviceRef:
+      name: `sonataflow-psql-postgresql` #  Replace with the Secret name of your deployment.
+      port: 5432
+      databaseName: sonataflow
+      databaseSchema: basic
+----
+
+If you must connect to an external database, replace `serviceRef` with a `jdbcUrl`. See link:https://docs.redhat.com/en/documentation/red_hat_openshift_serverless/1.36/html-single/serverless_logic/index#serverless-logic-managing-persistence[Managing workflow persistence].
+
+By default, the script generates all the manifests without a namespace. You can specify a namespace to the script by using the `--namespace` flag if you know the target namespace in advance. Otherwise, you must provide the namespace when applying the manifests to the cluster. See link:https://docs.redhat.com/en/documentation/red_hat_openshift_serverless/1.36/html-single/serverless_logic/index#serverless-logic-configuring-workflow-services[Configuring workflow services].

modules/orchestrator/con-project-structure-overview.adoc

@@ -0,0 +1,32 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-project-structure-overview.adoc_{context}"]
+= Project structure overview
+
+The project utilizes a *Quarkus project layout* (Maven project structure). This structure is illustrated by the following `01_basic` workflow example:
+
+[source, yaml]
+----
+01_basic
+├── pom.xml
+├── README.md
+└── src
+    └── main
+        ├── docker
+        │   ├── Dockerfile.jvm
+        │   ├── Dockerfile.legacy-jar
+        │   ├── Dockerfile.native
+        │   └── Dockerfile.native-micro
+        └── resources
+            ├── application.properties
+            ├── basic.svg
+            ├── basic.sw.yaml
+            ├── schemas
+            │   ├── basic__main-schema.json
+            │   └── workflow-output-schema.json
+            └── secret.properties
+----
+
+The main workflow resources are located under the `src/main/resources/` directory.
+
+The `kn-workflow CLI` generated this project structure. You can try generating the structure yourself by following the _Getting Started guide_. For more information on the Quarkus project, see link:https://quarkus.io/guides/getting-started[Creating your first application].

modules/orchestrator/con-required-tools.adoc

@@ -0,0 +1,17 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-required-tools.adoc_{context}"]
+= Required tools
+
+To run the `build-sh` script locally and manage the workflow lifecycle, you must install the following command-line tools:
+
+[cols="1,3", options="header"]
+|===
+|Tool |Conceptual Purpose.
+|podman or docker |Container runtime required for building the workflow images.
+|`kubectl` |Kubernetes CLI.
+|`yq` |YAML processor.
+|`jq` |JSON  processor.
+|`curl`, `git`, `find`, `which`| Shell utilities.
+|`kn-workflow` |CLI for generating workflow manifests.
+|===

modules/orchestrator/con-why-build-workflow-images.adoc

@@ -0,0 +1,11 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="con-why-build-workflow-images.adoc_{context}"]
+= Why build workflow images?
+
+While the OpenShift Serverless Logic Operator supports the building of workflows dynamically, this approach is primarily for experimentation.
+For production deployments, building images is the preferred method due to the following reasons:
+
+Production readiness:: Prebuilt images can be scanned, secured, and tested before going live.
+GitOps compatibility:: The Orchestrator relies on a central OpenShift Serverless Logic Operator instance to track workflows and their state. To use this tracking service, you must deploy workflows with the `gitops` profile, which expects a prebuilt image.
+Testing and quality:: Building an image gives you more control over the testing process.

modules/orchestrator/proc-building-locally.adoc

@@ -0,0 +1,35 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-building-locally.adoc_{context}"]
+= Building workflow images locally
+
+You can use the build script (`build.sh`) to build workflow images. You can run it either locally or inside a container. This section highlights how build workflow images locally.
+
+.Procedure
+
+. Clone the project as shown in the following example:
++
+[source, yaml]
+----
+git clone [email protected]:rhdhorchestrator/orchestrator-demo.git
+cd orchestrator-demo
+----
+
+. Check the help menu of the script:
++
+[source,yaml]
+----
+./scripts/build.sh --help
+----
+
+. Run the `build.sh` script, providing the required flags, for instance, the image path (`-i`), workflow source directory (`-w`), and manifests output directory (`-m`).
+
+[IMPORTANT]
+====
+You must specify the full target image path with a tag as shown in the following example:
+
+[source,yaml]
+----
+./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests
+----
+====

modules/orchestrator/proc-building-the-01-basic-workflow.adoc

@@ -0,0 +1,27 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-building-the-01-basic-workflow.adoc_{context}"]
+= Building the `01_basic` workflow
+
+To run the script from the root directory of the repository, you must use the `-w` flag to point to the workflow directory. Additionally, specify the output directory with the `-m` flag.
+
+.Prerequisites
+
+* You have specified the target image using a tag.
+
+.Procedure
+
+. Run the following command:
++
+[source,bash]
+----
+./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests
+----
++
+This build command produces the following two artifacts:
+
+* A workflow image and Kubernetes manifests: `quay.io/orchestrator/demo-basic:test` and tagged as `latest`.
+* Kubernetes manifests under: `01_basic/manifests/`
+
+. Optional: You can add the `--push` flag to automatically push the image after building. Otherwise, pushing manually is mandatory before deploying.
+

modules/orchestrator/proc-creating-and-running-workflows.adoc

@@ -0,0 +1,43 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="proc-creating-and-running-workflows.adoc_{context}"]
+= Creating and running your serverless workflow project locally
+
+The `kn-workflow` CLI is an essential tool that generates workflow manifests and project structures. To ensure successful development and immediate testing, begin developing a new serverless workflow locally by following these steps:
+
+.Procedure
+. Use the `kn-workflow` CLI to create a new workflow project, which adheres to the Quarkus structure as shown in the following example:
++
+[source,bash]
+----
+kn-workflow quarkus create --name <specify project name, for example ,00_new_project>
+----
+
+. Edit the workflow, add schema and specific files and run it locally from project folder as shown in the following example:
++
+[source,bash]
+----
+kn-workflow quarkus run
+----
+. Run the workflow locally using the `kn-workflow run` which pulls the following image:
++
+[source,yaml]
+----
+registry.redhat.io/openshift-serverless-1/logic-swf-devmode-rhel8:1.36.0
+----
+
+. For building the workflow image, the `kn-workflow` CLI pulls the following images:
++
+[source,yaml]
+----
+registry.redhat.io/openshift-serverless-1/logic-swf-builder-rhel8:1.36.0-8
+registry.access.redhat.com/ubi9/openjdk-17:1.21-2
+----
+
+[role="_additional-resources"]
+.Additional resources
+
+* link:https://openshift-knative.github.io/docs/docs/latest/serverless-logic/about.html[About OpenShift Serverless Logic]
+* link:https://redhat-scholars.github.io/serverless-workflow/osl/index.html[OpenShift Serverless Logic Tutorial]
+* {configuring-book-link}#running-behind-a-proxy[Running {product} behind a corporate proxy]
+* link:https://docs.redhat.com/en/documentation/red_hat_build_of_quarkus/3.15/html-single/getting_started_with_red_hat_build_of_quarkus/index#proc_online-maven_quarkus-getting-started[Using the Red Hat-hosted Quarkus repository]