Skip to content

Commit a559481

Browse files
authored
Correct outdated performance tuning guidance in redpanda_migrator input and output documentation (#342)
1 parent 6f705e3 commit a559481

File tree

2 files changed

+24
-5
lines changed

2 files changed

+24
-5
lines changed

modules/components/pages/inputs/redpanda_migrator.adoc

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,22 @@ For capabilities, guarantees, scheduling, and examples, see the output documenta
5656
When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair.
5757

5858
== Performance tuning for high throughput
59-
For workloads with high message rates or large messages, adjust these fields:
6059

61-
- `partition_buffer_bytes`: Increase to 10MB or higher (default: 1MB)
62-
- `max_yield_batch_bytes`: Increase to 100MB or higher (default: 10MB)
60+
For workloads with high message rates or large messages, adjust the following settings to optimize throughput:
6361

64-
Increasing these values allows the consumer to buffer more data per partition and yield larger batches, reducing overhead and improving throughput. Higher values increase memory usage—monitor system resources accordingly.
62+
On this input component:
63+
64+
- `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size
65+
- `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded
66+
67+
On the paired `redpanda_migrator` output component:
68+
69+
- `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster)
70+
71+
[NOTE]
72+
====
73+
Setting `max_yield_batch_bytes` over 1MB is counter-productive unless you change the broker settings to allow bigger messages or batches. The `partition_buffer_bytes` setting allows for partition readahead.
74+
====
6575

6676
== Metrics
6777
This input emits an `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic. This metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. Monitor this metric to track migration progress and detect bottlenecks.

modules/components/pages/outputs/redpanda_migrator.adoc

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,16 @@ When using multiple migrator pairs in a pipeline, match the `label` field exactl
4848

4949
== Performance tuning
5050

51-
For high-throughput workloads, adjust buffer sizes on the input component. xref:components:inputs/redpanda_migrator.adoc#performance-tuning[See input docs for tuning advice.]
51+
For high-throughput workloads, adjust the following settings:
52+
53+
On this output component:
54+
55+
- `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster)
56+
57+
On the paired xref:components:inputs/redpanda_migrator.adoc#performance-tuning[`redpanda_migrator` input component]:
58+
59+
- `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size
60+
- `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded
5261

5362
== Synchronization details
5463

0 commit comments

Comments
 (0)