DOC-961 New fields metrics, lag, and wait fields redpanda and redpanda_common inputs (#156)

Author: asimms41

modules/components/pages/inputs/redpanda.adoc

@@ -59,11 +59,13 @@ input:
     rack_id: ""
     start_from_oldest: true
     fetch_max_bytes: 50MiB
+    fetch_max_wait: 5s
     fetch_min_bytes: 1B
     fetch_max_partition_bytes: 1MiB
     consumer_group: "" # No default (optional)
     commit_period: 5s
     partition_buffer_bytes: 1MB
+    topic_lag_refresh_period: 5s
     auto_replay_nacks: true
 ```
 
@@ -120,6 +122,10 @@ Records are processed and delivered from each partition in the same batches as t
 
 You can break batches down further using the xref:components:processors/split.adoc[`split`] processor.
 
+== Metrics
+
+This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
+
 == Metadata
 
 This input adds the following metadata fields to each message:
@@ -128,6 +134,7 @@ This input adds the following metadata fields to each message:
 - `kafka_topic`
 - `kafka_partition`
 - `kafka_offset`
+- `kafka_lag`
 - `kafka_timestamp_ms`
 - `kafka_timestamp_unix`
 - `kafka_tombstone_message`
@@ -460,7 +467,7 @@ endif::[]
 
 === `sasl[].aws.credentials.role`
 
-he role ARN to assume.
+The role ARN to assume.
 
 
 *Type*: `string`
@@ -556,6 +563,14 @@ If individual records are larger than the `fetch_max_bytes` value, brokers will
 
 *Default*: `50MiB`
 
+=== `fetch_max_wait`
+
+The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`).
+
+*Type*: `string`
+
+*Default*: `5s`
+
 === `fetch_min_bytes`
 
 The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`.
@@ -604,6 +619,14 @@ Each buffer can grow slightly beyond this value.
 
 *Default*: `1MB`
 
+=== `topic_lag_refresh_period`
+
+The interval between refresh cycles. During each cycle, this input queries the Repanda Connect server to calculate the topic lag - the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
+
+*Type*: `string`
+
+*Default*: `5s`
+
 === `auto_replay_nacks`
 
 Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure.

modules/components/pages/inputs/redpanda_common.adoc

@@ -48,11 +48,13 @@ input:
     rack_id: ""
     start_from_oldest: true
     fetch_max_bytes: 50MiB
+    fetch_max_wait: 5s
     fetch_min_bytes: 1B
     fetch_max_partition_bytes: 1MiB
     consumer_group: "" # No default (optional)
     commit_period: 5s
     partition_buffer_bytes: 1MB
+    topic_lag_refresh_period: 5s
     auto_replay_nacks: true
 ```
 
@@ -135,6 +137,10 @@ Records are processed and delivered from each partition in the same batches as t
 
 You can break batches down further using the xref:components:processors/split.adoc[`split`] processor.
 
+== Metrics
+
+This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
+
 == Metadata
 
 This input adds the following metadata fields to each message:
@@ -143,6 +149,7 @@ This input adds the following metadata fields to each message:
 - `kafka_topic`
 - `kafka_partition`
 - `kafka_offset`
+- `kafka_lag`
 - `kafka_timestamp_ms`
 - `kafka_timestamp_unix`
 - `kafka_tombstone_message`
@@ -222,6 +229,14 @@ If individual records are larger than the `fetch_max_bytes` value, brokers will
 
 *Default*: `50MiB`
 
+=== `fetch_max_wait`
+
+The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`).
+
+*Type*: `string`
+
+*Default*: `5s`
+
 === `fetch_min_bytes`
 
 The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`.
@@ -271,6 +286,14 @@ Each buffer can grow slightly beyond this value.
 
 *Default*: `1MB`
 
+=== `topic_lag_refresh_period`
+
+The interval between refresh cycles. During each cycle, this input queries the Repanda Connect server to calculate the topic lag - the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
+
+*Type*: `string`
+
+*Default*: `5s`
+
 === `auto_replay_nacks`
 
 Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure.

modules/components/pages/inputs/redpanda_migrator.adoc

@@ -90,7 +90,7 @@ Specify a consumer group for this input to consume one or more topics and automa
 
 == Metrics
 
-This input emits a `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic.
+This input emits an `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group.
 
 == Metadata