Skip to content

Commit 04dc080

Browse files
micheleRPpaulohtb6
andauthored
DOC-1575 RPCN vs Wasm decision matrix (#402)
* DOC-1575 RPCN vs Wasm decision matrix * minor coderabbit suggestions * fix Tip title * Update modules/develop/pages/data-transforms/build.adoc Co-authored-by: Paulo Borges <[email protected]> * Update modules/get-started/pages/cloud-overview.adoc Co-authored-by: Paulo Borges <[email protected]> * Update modules/get-started/pages/cloud-overview.adoc Co-authored-by: Paulo Borges <[email protected]> * minor edits --------- Co-authored-by: Paulo Borges <[email protected]>
1 parent 0f40322 commit 04dc080

File tree

2 files changed

+55
-2
lines changed

2 files changed

+55
-2
lines changed

modules/develop/pages/data-transforms/build.adoc

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,18 @@
33

44
NOTE: Data transforms are supported on BYOC and Dedicated clusters running Redpanda version 24.3 and later.
55

6+
:tip-caption: When to use Redpanda Connect instead
7+
8+
[TIP]
9+
====
10+
Data transforms do not access external networks or disks, and are best for lightweight data preparation (filtering, scrubbing, schema/format conversion). Use xref:develop:connect/about.adoc[Redpanda Connect] when you need any of the following:
11+
12+
* External integration (HTTP services, databases, cloud storage) for enrichment or fan-out to third-party systems
13+
* Batching or windowed processing for grouping/aggregation
14+
* Prebuilt processors and connectors to reduce custom code
15+
16+
====
17+
18+
:tip-caption: Tip
19+
620
include::ROOT:develop:data-transforms/build.adoc[tag=single-source]

modules/get-started/pages/cloud-overview.adoc

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -290,9 +290,48 @@ Dedicated::
290290

291291
== Redpanda Connect and Kafka Connect
292292

293-
xref:develop:connect/about.adoc[Redpanda Connect] is integrated into Redpanda Cloud and available as a fully-managed service. Choose from a range of connectors, processors, and other components to quickly build and deploy streaming data pipelines or AI applications from the Cloud UI or using the link:/api/doc/cloud-dataplane/group/endpoint-redpanda-connect-pipeline[Data Plane API]. Comprehensive metrics, monitoring, and per pipeline scaling are also available. To start using Redpanda Connect, xref:develop:connect/connect-quickstart.adoc[try this quickstart].
293+
xref:develop:connect/about.adoc[Redpanda Connect] lets you compose pipelines from a rich library of inputs, processors, and outputs with strong metrics, logging, and per-pipeline scaling. To try it, see the xref:develop:connect/connect-quickstart.adoc[quickstart].
294294

295-
xref:develop:managed-connectors/index.adoc[Kafka Connect] is disabled by default on all new clusters. To unlock this feature for your BYOC or Dedicated clusters, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. Note that when Kafka Connect is enabled, there is a node running for Kafka Connect, even if connectors are not used.
295+
xref:develop:managed-connectors/index.adoc[Kafka Connect] is disabled by default on all new clusters. To unlock this feature for your BYOC or Dedicated cluster, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. When enabled, a Kafka Connect node runs even if no connectors are configured.
296+
297+
=== Redpanda Connect vs data transforms
298+
299+
xref:develop:data-transforms/how-transforms-work.adoc[Data transforms] (Wasm) provide lightweight, per-record changes between Redpanda topics with minimal latency. Transforms run inside the broker, map one input topic to one or more output topics, and are intentionally sandboxed (no external network or disk access). They're ideal for validation, redaction, format/schema conversion, and simple routing.
300+
301+
[cols="1,2,2",options="header",stripes=none]
302+
|===
303+
| | Data transforms | Redpanda Connect
304+
305+
| Best for
306+
| Simple, stateless, per-record normalization inside Redpanda
307+
| Enrichment/lookup with external services; multi-stage flows
308+
309+
| External I/O
310+
| Not permitted (sandboxed)
311+
| Native (HTTP/database/object storage)
312+
313+
| Topology
314+
| 1:1 or 1:N (no cross-topic fan-in)
315+
| Fan-in and fan-out; multi-step pipelines
316+
317+
| Ordering
318+
| Preserves per-partition order
319+
| Per-partition order can be preserved; configure parallelism and batching accordingly
320+
321+
| Scale & isolation
322+
| Shares broker CPU/memory; best for lightweight operations
323+
| Scales independently; isolates heavy work from brokers
324+
325+
| Failure handling
326+
| You code routing/error behavior
327+
| Built-in retries/backoff and DLQ patterns
328+
|===
329+
330+
[TIP]
331+
====
332+
* Use data transforms for simple, in-broker, per-record changes with minimal latency.
333+
* Use Redpanda Connect if your pipeline must talk to external systems (HTTP services, databases, cloud storage), or when you need advanced flow control, such as batching and windowed processing.
334+
====
296335

297336
== Redpanda Cloud vs Self-Managed feature compatibility
298337

0 commit comments

Comments
 (0)