You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on May 14, 2025. It is now read-only.
You can launch a task from a stream by using the https://github.com/spring-cloud-stream-app-starters/tasklauncher-dataflow/blob/master/spring-cloud-starter-stream-sink-task-launcher-dataflow/README.adoc[`task-launcher-dataflow`] sink.
1062
+
You can launch a task from a stream by using the https://github.com/spring-cloud/spring-cloud-dataflow/tree/main/spring-cloud-dataflow-tasklauncher/README.adoc[`task-launcher-dataflow`] sink which is provided as a part of the Spring Cloud Data Flow project.
1063
1063
The sink connects to a Data Flow server and uses its REST API to launch any defined task.
1064
-
The sink accepts a https://github.com/spring-cloud-stream-app-starters/tasklauncher-dataflow/blob/master/spring-cloud-starter-stream-sink-task-launcher-dataflow/README.adoc#payload[JSON payload] representing a `task launch request`, which provides the name of the task to launch and may include command line arguments and deployment properties.
1064
+
The sink accepts a https://github.com/spring-cloud/spring-cloud-dataflow/tree/main/spring-cloud-dataflow-tasklauncher/README.adoc#payload[JSON payload] representing a `task launch request`, which provides the name of the task to launch and may include command line arguments and deployment properties.
1065
1065
1066
-
The https://github.com/spring-cloud-stream-app-starters/core/blob/master/common/app-starters-task-launch-request-common/README.adoc[`app-starters-task-launch-request-common`] component, in conjunction with Spring Cloud Stream https://docs.spring.io/spring-cloud-stream/docs/current-snapshot/reference/htmlsingle/#_functional_composition[functional composition], can transform the output of any source or processor to a task launch request.
1066
+
The https://github.com/spring-cloud/stream-applications/tree/main/functions/function/task-launch-request-function/README.adoc[`task-launch-request-function`] component, in conjunction with Spring Cloud Stream https://docs.spring.io/spring-cloud-stream/docs/current-snapshot/reference/htmlsingle/#_functional_composition[functional composition], can transform the output of any source or processor to a task launch request.
1067
1067
1068
-
Adding a dependency to `app-starters-task-launch-request-common` auto-configures a `java.util.function.Function` implementation, registered through https://cloud.spring.io/spring-cloud-function/[Spring Cloud Function] as a `taskLaunchRequest`.
1068
+
Adding a dependency to `task-launch-request-function` auto-configures a `java.util.function.Function` implementation, registered through https://cloud.spring.io/spring-cloud-function/[Spring Cloud Function] as a `taskLaunchRequest`.
1069
1069
1070
-
For example, you can start with the https://github.com/spring-cloud-stream-app-starters/time/tree/master/spring-cloud-starter-stream-source-time[time] source, add the following dependency, build it, and register it as a custom source. We call it `time-tlr` in this example:
1070
+
For example, you can start with the https://github.com/spring-cloud/stream-applications/tree/main/applications/source/time-source[time] source, add the following dependency, build it, and register it as a custom source.
1071
1071
1072
1072
====
1073
1073
[source,xml]
@@ -1079,14 +1079,16 @@ For example, you can start with the https://github.com/spring-cloud-stream-app-s
1079
1079
----
1080
1080
====
1081
1081
1082
-
TIP: https://start-scs.cfapps.io/[Spring Cloud Stream Initializr] provides a great starting point for creating streamapplications.
1082
+
To build the application follow the instructions https://github.com/spring-cloud/stream-applications#building-stream-applications[here].
1083
1083
1084
-
Next, <<applications.adoc#applications, register>> the `task-launcher-dataflow` sink and create a task (we use the provided timestamp task):
1084
+
This will create an `apps` directory that contains `time-source-rabbit` and `time-source-kafka` directories in the `<stream app project>/applications/source/time-source` directory. In each of these you will see a target directory that contains a `time-source-<binder>-<version>.jar`. Now register the `time-source` jar (use the appropriate binder jar) with SCDF as a time source named `timestamp-tlr`.
1085
+
1086
+
Next, register the `task-launcher-dataflow` sink with SCDF and create a task definition `timestamp-task`. Once this is complete create the stream definition as shown below:
@@ -1128,43 +1130,30 @@ This pattern may be applied to any source to launch a task in response to any ev
1128
1130
A composed task can be launched with the `task-launcher-dataflow` sink, as discussed <<spring-cloud-dataflow-launch-tasks-from-stream, here>>.
1129
1131
Since we use the `ComposedTaskRunner` directly, we need to set up the task definitions for the composed task runner itself, along with the composed tasks, prior to the creation of the composed task launching stream.
1130
1132
Suppose we wanted to create the following composed task definition: `AAA && BBB`.
1131
-
The first step would be to create the task definitions, as shown in the following example:
1133
+
The first step would be to create the task definition, as shown in the following example:
This application launches a registered task definition using the Data Flow Server https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-launching[REST API].
5
+
6
+
== Input
7
+
8
+
Launch request args including:
9
+
10
+
* the task name (required and created as a task with the target Data Flow Server)
11
+
* deployment properties (key value pairs, optional).
12
+
* program arguments for the task (a list, optional).
13
+
14
+
=== Headers:
15
+
16
+
* `Content-Type: application/json`
17
+
18
+
=== Payload:
19
+
20
+
A JSON document:
21
+
22
+
[source,json]
23
+
----
24
+
{
25
+
"name":"foo",
26
+
"deploymentProps": {"key1":"val1","key2":"val2"},
27
+
"args":["--debug", "--foo", "bar"]
28
+
}
29
+
----
30
+
31
+
minimally,
32
+
33
+
[source,json]
34
+
----
35
+
{"name":"foo"}
36
+
----
37
+
38
+
== Output
39
+
40
+
N/A (launches task on the SCDF server's task platform).
41
+
42
+
=== Options
43
+
44
+
The **$$Spring Cloud Data Flow Task Launcher$$** $$sink$$ supports the following configuration properties:
45
+
46
+
//tag::configuration-properties[]
47
+
$$platform-name$$:: $$The Spring Cloud Data Flow platform to use for launching tasks.$$ *($$String$$, default: `$$default$$`)*
$$trigger.initial-delay$$:: $$The initial delay in milliseconds.$$ *($$Integer$$, default: `$$1000$$`)*
59
+
$$trigger.max-period$$:: $$The maximum polling period in milliseconds. Will be set to period if period > maxPeriod.$$ *($$Integer$$, default: `$$30000$$`)*
60
+
$$trigger.period$$:: $$The polling period in milliseconds.$$ *($$Integer$$, default: `$$1000$$`)*
61
+
//end::configuration-properties[]
62
+
63
+
== Using the TaskLauncher
64
+
The Dataflow tasklauncher is a sink that consumes `TaskLaunchRequest` messages, as described above, and launches a task using the configured Spring Cloud Data Flow server (given by `--spring.cloud.dataflow.client.server-uri`).
65
+
The task launcher periodically polls its input source for launch requests but will pause polling when the platform has reached it's concurrent task execution limit, given by `spring.cloud.dataflow.task.platform.<platform-type>.accounts[<account-name>].maximum-concurrent-tasks`.
66
+
This prevents the SCDF deployer's deployment platform from exhausting its resources under heavy task load.
67
+
The poller is scheduled using a `DynamicPeriodicTrigger`. By default the initial polling rate is 1 second, but may be configured to any duration. When polling is paused, or if there are no launch requests present, the trigger period will increase, applying exponential backoff, up to a configured maximum (30 seconds by default).
68
+
69
+
The SCDF server may be configured to launch tasks on multiple platforms.
70
+
Each task launcher instance is configured for a single platform, given by the `platformName` property (`default` if not specified).
71
+
This limitation is enforced because if the server has multiple task platforms configured, it may be the case that some of its task platforms are at the limit and some are not.
72
+
In this situation, we can only consume the next launch request if we know for which task platform it is targeted.
73
+
For this reason, if the SCDF server is configured for multiple task platforms (or a single non-default platform), we assume that all launch requests are targeted for that platform.
74
+
The task launcher will set the required deployment property `spring.cloud.dataflow.task.platformName` if the request does not provide it.
75
+
76
+
NOTE: If the request includes the deployment property `spring.cloud.dataflow.task.platformName`, and the value is not the same as the tasklauncher's `platformName`, the task launcher will throw an exception.
77
+
78
+
To launch tasks on multiple platforms, you must configure a task launcher instance per platform and use a https://github.com/spring-cloud-stream-app-starters/router/tree/master/spring-cloud-starter-stream-sink-router[router sink], or https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#partitioning[partitioning strategy], to route requests to the correct instance.
79
+
80
+
NOTE: When the poller is paused it puts pressure
81
+
on the message broker so some tuning will be necessary in extreme cases to balance resource utilization.
82
+
83
+
=== Client Authentication
84
+
85
+
If the SCDF server requires authentication, the client must pass credentials with authorization to launch a task.
86
+
The Data Flow client supports both basic and OAuth2 authentication.
87
+
88
+
For basic authentication set the username and password:
For OAuth2 authentication, set the `client-id`, `client-secret`, and `token-uri` at a minimum. These values correspond to values set in the SCDF server's OAuth2 configuration.
95
+
For more details, see https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#configuration-local-security[the Security section in the Data Flow reference].
0 commit comments