diff --git a/artifacts/attributes.adoc b/artifacts/attributes.adoc index 951a196846..fdb2e7334d 100644 --- a/artifacts/attributes.adoc +++ b/artifacts/attributes.adoc @@ -151,6 +151,8 @@ :observability-category-link: {product-docs-link}/#Observability :ocp-docs-link: link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-version} :odf-docs-link: link:https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/{ocp-version} +:orchestrator-book-link: {product-docs-link}/html-single/orchestrator_in_red_hat_developer_hub/index +:orchestrator-book-title: Orchestrator in {product} :osd-docs-link: link:https://docs.redhat.com/en/documentation/openshift_dedicated/{osd-version} :release-notes-book-link: {product-docs-link}/html-single/red_hat_developer_hub_release_notes/index :release-notes-book-title: {product} release notes diff --git a/assemblies/assembly-building-and-deploying-serverless-workflows.adoc b/assemblies/assembly-building-and-deploying-serverless-workflows.adoc new file mode 100644 index 0000000000..619b4b8481 --- /dev/null +++ b/assemblies/assembly-building-and-deploying-serverless-workflows.adoc @@ -0,0 +1,45 @@ +:_mod-docs-content-type: ASSEMBLY + +ifdef::context[] +[id="assembly-building-and-deploying-serverless-workflows"] +endif::[] +:context: orchestrator-rhdh += Build and deploy serverless workflows + +To deploy a workflow and make it available in the Orchestrator plugin, follow these main steps: + +* Building workflow images +* Generating workflow manifests +* Deploying workflows to a cluster + +This process moves the workflow from your local machine to deployment on a cluster. + +// why build workflow images +include::modules/orchestrator/con-benefits-of-workflow-images.adoc[leveloffset=+1] + +// project structure +include::modules/orchestrator/con-project-structure-overview.adoc[leveloffset=+2] + +// creating and running workflows locally +include::modules/orchestrator/proc-creating-and-running-workflows.adoc[leveloffset=+2] + +// building locally and generating artifacts +include::modules/orchestrator/proc-building-locally.adoc[leveloffset=+1] + +// the script and its uses +include::modules/orchestrator/con-build-sh-script-and-its-uses.adoc[leveloffset=+2] + +// environment variables supported by the script +include::modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc[leveloffset=+2] + +// required tools +include::modules/orchestrator/con-required-tools.adoc[leveloffset=+2] + +// building the 01_basic workflow +include::modules/orchestrator/proc-building-the-01-basic-workflow.adoc[leveloffset=+2] + +// generated workflow manifests +include::modules/orchestrator/con-generated-workflow-manifests.adoc[leveloffset=+1] + +// deploy workflows on a cluster +include::modules/orchestrator/proc-deploying-workflows-on-a-cluster.adoc[leveloffset=+1] \ No newline at end of file diff --git a/modules/orchestrator/con-benefits-of-workflow-images.adoc b/modules/orchestrator/con-benefits-of-workflow-images.adoc new file mode 100644 index 0000000000..743c7e31ed --- /dev/null +++ b/modules/orchestrator/con-benefits-of-workflow-images.adoc @@ -0,0 +1,11 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-benefits-of-workflow-images.adoc_{context}"] += Benefits of workflow images + +While the OpenShift Serverless Logic Operator supports the building of workflows dynamically, this approach is primarily for experimentation. +For production deployments, building images is the preferred method due to the following reasons: + +* Production readiness: Prebuilt images can be scanned, secured, and tested before going live. +* GitOps compatibility: The Orchestrator relies on a central OpenShift Serverless Logic Operator instance to track workflows and their state. To use this tracking service, you must deploy workflows with the `gitops` profile, which expects a prebuilt image. +* Testing and quality: Building an image gives you more control over the testing process. \ No newline at end of file diff --git a/modules/orchestrator/con-build-sh-script-and-its-uses.adoc b/modules/orchestrator/con-build-sh-script-and-its-uses.adoc new file mode 100644 index 0000000000..45ce77b4e3 --- /dev/null +++ b/modules/orchestrator/con-build-sh-script-and-its-uses.adoc @@ -0,0 +1,35 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-build-sh-script-and-its-uses.adoc_{context}"] += The `build-sh` script functionality and important flags + +The `build-sh` script does the following tasks in order: + +* Generates workflow manifests using the `kn-workflow` CLI. +* Builds the workflow image using `podman` or `docker`. +* Optional: The script pushes the images to an image registry and deploys the workflow using `kubectl`. + +You can review the script configuration options and see available flags and their functions by accessing the help menu: + +[source,bash] +---- +./scripts/build.sh [flags] +---- + +The following flags are essential for running the script: + +[cols="1,3", options="header"] +|=== +|Flag |Description +|`-i`, `--image` |Required: Full image path, for example, `quay.io/orchestrator/demo:latest` +|`-w`, `--workflow-directory` |Workflow source directory (default is the current directory) +|`-m`, `--manifests-directory` |Where to save generated manifests +|`--push` |Push the image to the registry +|`--deploy` |Deploy the workflow +|`-h`, `--help` |Show the help message +|=== + +[TIP] +==== +The script also supports builder and runtime image overrides, namespace targeting, and persistence flags. +==== \ No newline at end of file diff --git a/modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc b/modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc new file mode 100644 index 0000000000..d114fc18ab --- /dev/null +++ b/modules/orchestrator/con-environment-variables-supported-by-the-build-script.adoc @@ -0,0 +1,28 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-environment-variables-supported-by-the-build-script.adoc_{context}"] += Environment variables supported by the build script + +The `build-sh` script supports the following environment variables to customize the workflow build process without modifying the script itself: + +`QUARKUS_EXTENSIONS`:: + +The `QUARKUS_EXTENSIONS` variable specifies additional link:https://quarkus.io/extensions/[Quarkus] extensions required by the workflow. This variable takes the format of a comma-separated list of fully qualified extension IDs as shown in the following example: ++ +[source,yaml] +---- +export QUARKUS_EXTENSIONS="io.quarkus:quarkus-smallrye-reactive-messaging-kafka" +---- ++ +Add Kafka messaging support or other integrations at build time. + +`MAVEN_ARGS_APPEND`:: + +The `MAVEN_ARGS_APPEND` variable appends additional arguments to the *Maven build* command. This variable takes the format of a string of Maven CLI arguments as shown in the following example: ++ +[source,yaml] +---- +export MAVEN_ARGS_APPEND="-DmaxYamlCodePoints=35000000" +---- ++ +Control build behavior. For example, set `maxYamlCodePoints` parameter that controls the maximum input size for YAML input files to 35000000 characters (~33MB in UTF-8). \ No newline at end of file diff --git a/modules/orchestrator/con-generated-workflow-manifests.adoc b/modules/orchestrator/con-generated-workflow-manifests.adoc new file mode 100644 index 0000000000..5315700804 --- /dev/null +++ b/modules/orchestrator/con-generated-workflow-manifests.adoc @@ -0,0 +1,76 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-generated-workflow-manifests.adoc_{context}"] += Generated workflow manifests + +The following example is an illustration of what is generated under the `01_basic/manifests`: + +[source,yaml] +---- +01_basic/manifests +├── 00-secret_basic-secrets.yaml +├── 01-configmap_basic-props.yaml +├── 02-configmap_01-basic-resources-schemas.yaml +└── 03-sonataflow_basic.yaml +---- + +`00-secret_basic-secrets.yaml`:: +Contains secrets from `01_basic/src/main/resources/secret.properties`. +Values are not required at this stage as you can set them later after applying CRs or when using GitOps. + +[Important] +==== +In OpenShift Serverless Logic `v1.36`, after updating a secret, you must manually restart the workflow Pod for changes to apply. +==== + +`01-configmap_basic-props.yaml`:: +Holds application properties from application.properties. +Any change to this ConfigMap triggers an automatic Pod restart. + +`02-configmap_01-basic-resources-schemas.yaml`:: +Contains JSON schemas from src/main/resources/schemas. ++ +[NOTE] +==== +You do not need to deploy certain configuration resources when using the GitOps profile. +==== + +`03-sonataflow_basic.yaml`:: +The SonataFlow custom resource (CR) that defines the workflow. ++ +[source,yaml] +---- +podTemplate: + container: + image: quay.io/orchestrator/demo-basic + resources: {} + envFrom: + - secretRef: + name: basic-secrets +---- ++ +[source,yaml,subs="+quotes"] +---- +persistence: + postgresql: + secretRef: + name: `sonataflow-psql-postgresql` + userKey: `____` + passwordKey: `____` + serviceRef: + name: `sonataflow-psql-postgresql` + port: 5432 + databaseName: sonataflow + databaseSchema: basic +---- ++ +where: + +`postgresql:secretRef:name`:: Enter the Secret name for your deployment. +`postgresql:secretRef:userKey`:: Enter the key for your deployment. +`postgresql:secretRef:passwordKey`:: Enter the password for your deployment. +`postgresql:serviceRef:name`:: Enter the Service name for your deployment. ++ +If you must connect to an external database, replace `serviceRef` with `jdbcUrl`. See link:https://docs.redhat.com/en/documentation/red_hat_openshift_serverless/1.36/html-single/serverless_logic/index#serverless-logic-managing-persistence[Managing workflow persistence]. + +By default, the script generates all the manifests without a namespace. You can specify a namespace to the script by using the `--namespace` flag if you know the target namespace in advance. Otherwise, you must provide the namespace when applying the manifests to the cluster. See link:https://docs.redhat.com/en/documentation/red_hat_openshift_serverless/1.36/html-single/serverless_logic/index#serverless-logic-configuring-workflow-services[Configuring workflow services]. \ No newline at end of file diff --git a/modules/orchestrator/con-project-structure-overview.adoc b/modules/orchestrator/con-project-structure-overview.adoc new file mode 100644 index 0000000000..fb83bbf427 --- /dev/null +++ b/modules/orchestrator/con-project-structure-overview.adoc @@ -0,0 +1,32 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-project-structure-overview.adoc_{context}"] += Project structure overview + +The project utilizes *Quarkus project layout* (Maven project structure). This structure is illustrated by the following `01_basic` workflow example: + +[source, yaml] +---- +01_basic +├── pom.xml +├── README.md +└── src + └── main + ├── docker + │ ├── Dockerfile.jvm + │ ├── Dockerfile.legacy-jar + │ ├── Dockerfile.native + │ └── Dockerfile.native-micro + └── resources + ├── application.properties + ├── basic.svg + ├── basic.sw.yaml + ├── schemas + │ ├── basic__main-schema.json + │ └── workflow-output-schema.json + └── secret.properties +---- + +The main workflow resources are located under the `src/main/resources/` directory. + +The `kn-workflow CLI` generated this project structure. You can try generating the structure yourself by following the _Getting Started guide_. For more information on the Quarkus project, see link:https://quarkus.io/guides/getting-started[Creating your first application]. \ No newline at end of file diff --git a/modules/orchestrator/con-required-tools.adoc b/modules/orchestrator/con-required-tools.adoc new file mode 100644 index 0000000000..4c1081b31b --- /dev/null +++ b/modules/orchestrator/con-required-tools.adoc @@ -0,0 +1,17 @@ +:_mod-docs-content-type: CONCEPT + +[id="con-required-tools.adoc_{context}"] += Required tools + +To run the `build-sh` script locally and manage the workflow lifecycle, you must install the following command-line tools: + +[cols="1,3", options="header"] +|=== +|Tool |Conceptual Purpose. +|podman or docker |Container runtime required for building the workflow images. +|`kubectl` |Kubernetes CLI. +|`yq` |YAML processor. +|`jq` |JSON processor. +|`curl`, `git`, `find`, `which`| Shell utilities. +|`kn-workflow` |CLI for generating workflow manifests. +|=== \ No newline at end of file diff --git a/modules/orchestrator/proc-building-locally.adoc b/modules/orchestrator/proc-building-locally.adoc new file mode 100644 index 0000000000..4ac66621d1 --- /dev/null +++ b/modules/orchestrator/proc-building-locally.adoc @@ -0,0 +1,35 @@ +:_mod-docs-content-type: PROCEDURE + +[id="proc-building-locally.adoc_{context}"] += Building workflow images locally + +You can use the build script (`build.sh`) to build workflow images. You can run it either locally or inside a container. This section highlights how build workflow images locally. + +.Procedure + +. Clone the project as shown in the following example: ++ +[source, yaml] +---- +git clone git@github.com:rhdhorchestrator/orchestrator-demo.git +cd orchestrator-demo +---- + +. Check the help menu of the script: ++ +[source,yaml] +---- +./scripts/build.sh --help +---- + +. Run the `build.sh` script, providing the required flags, for instance, the image path (`-i`), workflow source directory (`-w`), and manifests output directory (`-m`). + ++ +[IMPORTANT] +==== +You must specify the full target image path with a tag as shown in the following example: +[source,yaml] +---- +./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests +---- +==== \ No newline at end of file diff --git a/modules/orchestrator/proc-building-the-01-basic-workflow.adoc b/modules/orchestrator/proc-building-the-01-basic-workflow.adoc new file mode 100644 index 0000000000..6dad50f109 --- /dev/null +++ b/modules/orchestrator/proc-building-the-01-basic-workflow.adoc @@ -0,0 +1,27 @@ +:_mod-docs-content-type: PROCEDURE + +[id="proc-building-the-01-basic-workflow.adoc_{context}"] += Building the `01_basic` workflow + +To run the script from the root directory of the repository, you must use the `-w` flag to point to the workflow directory. Additionally, specify the output directory with the `-m` flag. + +.Prerequisites + +* You have specified the target image using a tag. + +.Procedure + +. Run the following command: ++ +[source,bash] +---- +./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests +---- ++ +This build command produces the following two artifacts: + +* A workflow image and Kubernetes manifests: `quay.io/orchestrator/demo-basic:test` and tagged as `latest`. +* Kubernetes manifests under: `01_basic/manifests/` + +. Optional: You can add the `--push` flag to automatically push the image after building. Otherwise, pushing manually is mandatory before deploying. + diff --git a/modules/orchestrator/proc-creating-and-running-workflows.adoc b/modules/orchestrator/proc-creating-and-running-workflows.adoc new file mode 100644 index 0000000000..f470822d8c --- /dev/null +++ b/modules/orchestrator/proc-creating-and-running-workflows.adoc @@ -0,0 +1,43 @@ +:_mod-docs-content-type: PROCEDURE + +[id="proc-creating-and-running-workflows.adoc_{context}"] += Creating and running your serverless workflow project locally + +The `kn-workflow` CLI is an essential tool that generates workflow manifests and project structures. To ensure successful development and immediate testing, begin developing a new serverless workflow locally by completing the following steps: + +.Procedure +. Use the `kn-workflow` CLI to create a new workflow project, which adheres to the Quarkus structure as shown in the following example: ++ +[source,bash] +---- +kn-workflow quarkus create --name +---- + +. Edit the workflow, add schema and specific files, and run it locally from project folder as shown in the following example: ++ +[source,bash] +---- +kn-workflow quarkus run +---- +. Run the workflow locally using the `kn-workflow run` which pulls the following image: ++ +[source,yaml] +---- +registry.redhat.io/openshift-serverless-1/logic-swf-devmode-rhel8:1.36.0 +---- + +. For building the workflow image, the `kn-workflow` CLI pulls the following images: ++ +[source,yaml] +---- +registry.redhat.io/openshift-serverless-1/logic-swf-builder-rhel8:1.36.0-8 +registry.access.redhat.com/ubi9/openjdk-17:1.21-2 +---- + +[role="_additional-resources"] +.Additional resources + +* link:https://openshift-knative.github.io/docs/docs/latest/serverless-logic/about.html[About OpenShift Serverless Logic] +* link:https://redhat-scholars.github.io/serverless-workflow/osl/index.html[OpenShift Serverless Logic Tutorial] +* {configuring-book-link}#running-behind-a-proxy[Running {product} behind a corporate proxy] +* link:https://docs.redhat.com/en/documentation/red_hat_build_of_quarkus/3.15/html-single/getting_started_with_red_hat_build_of_quarkus/index#proc_online-maven_quarkus-getting-started[Using the Red Hat-hosted Quarkus repository] \ No newline at end of file diff --git a/modules/orchestrator/proc-deploying-workflows-on-a-cluster.adoc b/modules/orchestrator/proc-deploying-workflows-on-a-cluster.adoc new file mode 100644 index 0000000000..29cbcf7b11 --- /dev/null +++ b/modules/orchestrator/proc-deploying-workflows-on-a-cluster.adoc @@ -0,0 +1,108 @@ +:_mod-docs-content-type: PROCEDURE + +[id="proc-deploying-workflows-on-a-cluster.adoc_{context}"] += Deploying workflows on a cluster + +You can deploy the workflow on a cluster, since the image is pushed to the image registry and the deployment manifests are available. + +.Prerequisites + +* You have an {ocp-short} cluster with the following versions of components installed: + +** {product} ({product-very-short}) `v1.7` +** Orchestrator plugins `v1.7.1` +** OpenShift Serverless `v1.36` +** OpenShift Serverless Logic `v1.36` ++ +For instructions on how to install these components, see the {orchestrator-book-link}#con-orchestrator-plugin-components.adoc_orchestrator-rhdh[Orchestrator plugin components on {ocp-short}]. + +* You must apply the workflow manifests in a namespace that contains a `SonataflowPlatform` custom resource (CR), which manages the link:https://docs.redhat.com/en/documentation/red_hat_openshift_serverless/1.36/html-single/serverless_logic/index#serverless-logic-configuring-workflow-services[supporting services]. + +.Procedure + +. Use the `kubectl create` command specifying the target namespace to apply the Kubernetes manifests as shown in the following example: ++ +[source,bash] +---- +kubectl create -n -f ./01_basic/manifests/. +---- + +. After deployment, monitor the status of the workflow pods as shown in the following example: ++ +[source,yaml] +---- +kubectl get pods -n -l app=basic +---- ++ +The pod may initially appear in an `Error` state because of missing or incomplete configuration in the Secret or ConfigMap. + +. Inspect the Pod logs as shown in the following example: ++ +[source,yaml] +---- +oc logs -n basic-f7c6ff455-vwl56 +---- ++ +The following code is an example of the output: ++ +[source,yaml] +---- +SRCFG00040: The config property quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token is defined as the empty String ("") which the following Converter considered to be null: io.smallrye.config.Converters$BuiltInConverter +java.lang.RuntimeException: Failed to start quarkus +... +Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties +---- ++ +The error indicates a missing property: `quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token`. + +. In such a case where the logs show the `ConfigurationException: Failed to read configuration properties` error or indicate a missing value, retrieve the ConfigMap as shown in the following example: ++ +[source,yaml] +---- +oc get -n configmaps basic-props -o yaml +---- ++ +The following code is an example of the sample output: ++ +[source,yaml] +---- +apiVersion: v1 +data: + application.properties: | + # Backstage notifications service + quarkus.rest-client.notifications.url=${BACKSTAGE_NOTIFICATIONS_URL} + quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token=${NOTIFICATIONS_BEARER_TOKEN} +... +---- ++ +Resolve the placeholders using values provided using a Secret. + +. You must edit the corresponding Secret and provide appropriate base64-encoded values to resolve the placeholders in `application.properties` as shown in the following example: ++ +[source,yaml] +---- +kubectl edit secrets -n basic-secrets +---- +. Restart the workflow Pod for Secret changes to take effect in OpenShift Serverless Logic `v1.36`. + +.Verification + +. Verify the deployment status by checking the Pods again as shown in the following example: ++ +[source,yaml] +---- +oc get pods -n -l app=basic +---- ++ +The expected status for a successfully deployed workflow Pod is as shown in the following example: ++ +[source,yaml] +---- +NAME READY STATUS RESTARTS AGE +basic-f7c6ff455-grkxd 1/1 Running 0 47s +---- + +. Once the Pod is in the `Running` state, the workflow now appears in the Orchestrator plugin inside the {product}. + +.Next steps +* Inspect the provided build script to extract the actual steps and implement them in your preferred CI/CD tool, for example, GitHub Actions, GitLab CI, Jenkins, and Tekton. \ No newline at end of file diff --git a/titles/orchestrator/master.adoc b/titles/orchestrator/master.adoc index 97584be038..f483c3021e 100644 --- a/titles/orchestrator/master.adoc +++ b/titles/orchestrator/master.adoc @@ -8,4 +8,6 @@ include::artifacts/attributes.adoc[] include::assemblies/assembly-orchestrator-rhdh.adoc[leveloffset=+1] -include::assemblies/assembly-install-rhdh-orchestrator.adoc[leveloffset=+1] \ No newline at end of file +include::assemblies/assembly-building-and-deploying-serverless-workflows.adoc[leveloffset=+1] + +include::assemblies/assembly-install-rhdh-orchestrator.adoc[leveloffset=+1]