You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:description: You can use Redpanda Console to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed.
1
+
= Manage Data Transforms in {ui}
2
+
:description: Use {ui} to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed.
3
+
// tag::single-source[]
3
4
4
5
{description}
5
6
6
7
== Prerequisites
7
8
8
9
Before you begin, ensure that you have the following:
9
10
11
+
ifndef::env-cloud[]
10
12
- Redpanda Console must be xref:console:config/connect-to-redpanda.adoc[connected to a Redpanda cluster].
11
13
- Redpanda Console must be xref:console:config/connect-to-redpanda.adoc#admin[configured to connect to the Redpanda Admin API].
14
+
endif::[]
12
15
- xref:develop:data-transforms/configure.adoc#enable-transforms[Data transforms enabled] in your Redpanda cluster.
13
16
- At least one transform function deployed to your Redpanda cluster.
14
17
15
18
[[monitor]]
16
19
== Monitor transform functions
17
20
18
-
To monitor transform functions in Redpanda Console:
21
+
To monitor transform functions:
19
22
20
23
. Navigate to the *Transforms* menu.
21
-
. Click on the name of a transform function to view detailed information:
24
+
. Click the name of a transform function to view detailed information:
22
25
- The partitions that the function is running on
23
26
- The broker (node) ID
24
27
- Any lag (the amount of pending records on the input topic that have yet to be processed by the transform)
25
28
26
29
[[logs]]
27
30
== View logs
28
31
29
-
To view logs for a transform function in Redpanda Console:
32
+
To view logs for a transform function:
30
33
31
34
. Navigate to the *Transforms* menu.
32
35
. Click on the name of a transform function.
33
36
. Click the *Logs* tab to see the logs.
34
37
35
-
Redpanda Console displays a limited number of logs for transform functions. To view the full history of logs, use the xref:develop:data-transforms/monitor.adoc#logs[`rpk` command-line tool].
38
+
{ui} displays a limited number of logs for transform functions. To view the full history of logs, use the xref:develop:data-transforms/monitor.adoc#logs[`rpk` command-line tool].
36
39
37
40
[[delete]]
38
41
== Delete transform functions
39
42
40
-
To delete a transform function in Redpanda Console:
43
+
To delete a transform function:
41
44
42
45
1. Navigate to the *Transforms* menu.
43
46
2. Find the transform function you want to delete from the list.
@@ -50,4 +53,6 @@ Deleting a transform function will remove it from the cluster and stop any furth
// Do not put page aliases in the single-sourced content
4
-
// tag::single-source[]
5
4
:description: Learn how to filter Kafka records using custom JavaScript code within {ui}.
5
+
// tag::single-source[]
6
6
7
7
You can use push-down filters in {ui} to search through large Kafka topics that may contain millions of records. Filters are JavaScript functions executed on the backend, evaluating each record individually. Your function must return a boolean:
:description: Learn how {ui} deserializes messages.
4
+
// tag::single-source[]
5
5
6
6
In Redpanda, the messages exchanged between producers and consumers contain raw bytes. Schemas work as an agreed-upon format, like a contract, for producers and consumers to serialize and deserialize those messages. If a producer breaks this contract, consumers can fail.
:description: Perform common Schema Registry management operations in the {ui}.
5
+
// tag::single-source[]
6
6
7
7
In {ui}, the *Schema Registry* menu lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to.
8
8
9
9
[NOTE]
10
10
====
11
-
The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with {ui}. This section describes Schema Registry operations available in {ui}.
11
+
The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with the UI. This section describes Schema Registry operations available in the UI.
Copy file name to clipboardExpand all lines: modules/develop/pages/data-transforms/build.adoc
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,20 @@
1
1
= Develop Data Transforms
2
2
:description: Learn how to initialize a data transforms project and write transform functions in your chosen language.
3
3
:page-categories: Development, Stream Processing, Data Transforms
4
+
// tag::single-source[]
4
5
5
6
{description}
6
7
7
8
== Prerequisites
8
9
9
10
You must have the following development tools installed on your host machine:
10
11
12
+
ifdef::env-cloud[]
13
+
* The xref:manage:rpk/rpk-install.adoc[`rpk` command-line client] installed.
14
+
endif::[]
15
+
ifndef::env-cloud[]
11
16
* The xref:get-started:rpk-install.adoc[`rpk` command-line client] installed on your host machine and configured to connect to your Redpanda cluster.
17
+
endif::[]
12
18
* For Golang projects, you must have at least version 1.20 of https://go.dev/doc/install[Go^].
13
19
* For Rust projects, you must have the latest stable version of https://rustup.rs/[Rust^].
14
20
* For JavaScript and TypeScript projects, you must have the https://nodejs.org/en/download/package-manager[latest long-term-support release of Node.js^].
@@ -46,9 +52,7 @@ For example, if you choose `tinygo-no-goroutines`, the following project files a
46
52
The `transform.go` file contains a boilerplate transform function.
47
53
The `transform.yaml` file specifies the configuration settings for the transform function.
48
54
49
-
See also:
50
-
51
-
- xref:develop:data-transforms/configure.adoc[]
55
+
See also: xref:develop:data-transforms/configure.adoc[]
52
56
53
57
== Build transform functions
54
58
@@ -284,7 +288,7 @@ See also:
284
288
- xref:develop:data-transforms/monitor#logs[View logs for transform functions]
285
289
- xref:develop:data-transforms/monitor.adoc[Monitor data transforms]
Copy file name to clipboardExpand all lines: modules/develop/pages/data-transforms/configure.adoc
+15-7Lines changed: 15 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,7 @@
1
1
= Configure Data Transforms
2
2
:description: pass:q[Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment.]
3
3
:page-categories: Development, Stream Processing, Data Transforms
4
+
// tag::single-source[]
4
5
5
6
{description}
6
7
@@ -41,7 +42,7 @@ env:
41
42
42
43
You can set the name of the transform function, environment variables, and input and output topics on the command-line when you deploy the transform. These command-line settings take precedence over those specified in the `transform.yaml` file.
43
44
44
-
See xref:develop:data-transforms/deploy.adoc[].
45
+
See xref:develop:data-transforms/deploy.adoc[]
45
46
46
47
[[built-in]]
47
48
=== Built-In environment variables
@@ -62,19 +63,20 @@ This section covers how to configure the Wasm engine environment using Redpanda
62
63
63
64
To use data transforms, you must enable it for a Redpanda cluster using the xref:reference:properties/cluster-properties.adoc#data_transforms_enabled[`data_transforms_enabled`] property.
64
65
65
-
[[resources]]
66
+
ifndef::env-cloud[]
66
67
=== Configure memory resources for data transforms
67
68
68
69
Redpanda reserves memory for each transform function within the broker. You need enough memory for your input record and output record to be in memory at the same time.
69
70
70
-
Set the following properties based on the number of functions you have and the amount of memory you anticipate needing.
71
+
Set the following based on the number of functions you have and the amount of memory you anticipate needing.
71
72
72
73
- xref:reference:properties/cluster-properties.adoc#data_transforms_per_core_memory_reservation[`data_transforms_per_core_memory_reservation`]: Increase this setting if you plan to deploy a large number of data transforms or if your transforms are memory-intensive. Reducing it may limit the number of concurrent transforms.
73
-
74
74
- xref:reference:properties/cluster-properties.adoc#data_transforms_per_function_memory_limit[`data_transforms_per_function_memory_limit`]: Adjust this setting if individual transform functions require more memory to process records efficiently. Reducing it may cause memory errors in complex transforms.
75
75
76
76
The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. When that limit is hit, Redpanda cannot allocate memory for the VM and the transforms stay in `errored` states.
77
+
endif::[]
77
78
79
+
ifndef::env-cloud[]
78
80
[[binary-size]]
79
81
=== Configure maximum binary size
80
82
@@ -88,25 +90,31 @@ Increase this setting if your Wasm binaries are larger than the default limit. S
88
90
You can set the interval at which data transforms commit their progress using the xref:reference:properties/cluster-properties.adoc#data_transforms_commit_interval_ms[`data_transforms_commit_interval_ms`] property.
89
91
90
92
Adjust this setting to control how frequently the transform function's progress is committed. Shorter intervals may provide more frequent progress updates but can increase load. Longer intervals reduce load but may delay progress updates.
93
+
endif::[]
91
94
92
95
[[log]]
93
96
=== Configure transform logging
97
+
The following properties configure logging for data transforms:
94
98
95
-
Redpanda provides several properties to configure logging for data transforms:
96
-
99
+
ifndef::env-cloud[]
97
100
- xref:reference:properties/cluster-properties.adoc#data_transforms_logging_buffer_capacity_bytes[`data_transforms_logging_buffer_capacity_bytes`]: Increase this value if your transform logs are large or if you need to buffer more log data before flushing. Reducing this value may cause more frequent log flushing.
98
101
99
102
- xref:reference:properties/cluster-properties.adoc#data_transforms_logging_flush_interval_ms[`data_transforms_logging_flush_interval_ms`]: Adjust this value to control how frequently logs are flushed to the `transform_logs` topic. Shorter intervals provide more frequent log updates but can increase load. Longer intervals reduce load but may delay log updates.
103
+
endif::[]
100
104
101
105
- xref:reference:properties/cluster-properties.adoc#data_transforms_logging_line_max_bytes[`data_transforms_logging_line_max_bytes`]: Increase this value if your log messages are frequently truncated. Setting this value too low may truncate important log information.
102
106
107
+
ifndef::env-cloud[]
103
108
[[runtime-limit]]
104
109
=== Configure runtime limits
105
110
106
111
You can set the maximum runtime for starting up a data transform and the time it takes for a single record to be transformed using the xref:reference:properties/cluster-properties.adoc#data_transforms_runtime_limit_ms[`data_transforms_runtime_limit_ms`] property.
107
112
108
113
Adjust this value only if your transform functions need more time to process each record or to start up.
:description: Learn how Redpanda data transforms work.
4
+
// tag::single-source[]
5
5
6
6
Redpanda provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. Redpanda does this directly inside the broker, eliminating the need to manage a separate stream processing environment or use third-party tools.
7
7
@@ -22,7 +22,9 @@ To execute a transform function, Redpanda uses just-in-time (JIT) compilation to
22
22
When you deploy a data transform to a Redpanda broker, it stores the Wasm bytecode and associated metadata, such as input and output topics and environment variables. The broker then replicates this data across the cluster using internal Kafka topics. When the data is distributed, each shard runs its own instance of the transform function. This process includes several resource management features:
23
23
24
24
- Each shard can run only one instance of the transform function at a time to ensure efficient resource utilization and prevent overload.
25
+
ifndef::env-cloud[]
25
26
- Memory for each function is reserved within the broker with the `data_transforms_per_core_memory_reservation` and `data_transforms_per_function_memory_limit` properties. See xref:develop:data-transforms/configure.adoc#resources[Configure memory for data transforms].
27
+
endif::[]
26
28
- CPU time is dynamically allocated to the Wasm runtime to ensure that the code does not run forever and cannot block the broker from handling traffic or doing other work, such as Tiered Storage uploads.
27
29
28
30
== Flow of data transforms
@@ -74,6 +76,8 @@ This section outlines the limitations of data transforms. These constraints are
0 commit comments