Documentation for batch's event metrics (current and average) (#18017)

Update the list of logstash settings with the new pipeline.batch.metrics.sampling_mode to control the batch size ones.

Co-authored-by: Rob Bavey <[email protected]>

Author: andsel

docs/reference/logstash-settings-file.md

@@ -48,6 +48,7 @@ The `logstash.yml` file includes these settings.
 | `pipeline.workers` | The number of workers that will, in parallel, execute the filter and outputstages of the pipeline. This setting uses the[`java.lang.Runtime.getRuntime.availableProcessors`](https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.md#availableProcessors())value as a default if not overridden by `pipeline.workers` in `pipelines.yml` or`pipeline.workers` from `logstash.yml`.  If you have modified this setting andsee that events are backing up, or that the CPU is not saturated, considerincreasing this number to better utilize machine processing power. | Number of the host’s CPU cores |
 | `pipeline.batch.size` | The maximum number of events an individual worker thread will collect from inputs  before attempting to execute its filters and outputs.  Larger batch sizes are generally more efficient, but come at the cost of increased memory  overhead. You may need to increase JVM heap space in the `jvm.options` config file.  See [Logstash Configuration Files](/reference/config-setting-files.md) for more info. | `125` |
 | `pipeline.batch.delay` | When creating pipeline event batches, how long in milliseconds to wait for  each event before dispatching an undersized batch to pipeline workers. | `50` |
+| `pipeline.batch.metrics.sampling_mode` {applies_to}`stack: preview 9.2.0`| Controls frequency of collection of batch size metrics. These metrics measure the actual number of events and byte size of batches processed through a pipeline. This can be helpful to tune `pipeline.batch.size` to reflect the actual batch sizes processed. <br><br>Note: This feature is in **technical preview** and may change in the future.<br><br>Current options are:<br><br>* `disabled`: disabling the collection.<br>* `minimal`:  calculate based on a subset of batches.(default)<br>* `full`: calculate based on every processed batch.<br> | `minimal` |
 | `pipeline.unsafe_shutdown` | When set to `true`, forces Logstash to exit during shutdown even if there are still inflight events  in memory. By default, Logstash will refuse to quit until all received events  have been pushed to the outputs. Enabling this option can lead to data loss during shutdown. | `false` |
 | `pipeline.plugin_classloaders` | (Beta) Load Java plugins in independent classloaders to isolate their dependencies. | `false` |
 | `pipeline.ordered` | Set the pipeline event ordering. Valid options are:<br><br>* `auto`. Automatically enables ordering if the `pipeline.workers` setting is `1`, and disables otherwise.<br>* `true`. Enforces ordering on the pipeline and prevents Logstash from starting if there are multiple workers.<br>* `false`. Disables the processing required to preserve order. Ordering will not be guaranteed, but you save the processing cost of preserving order.<br> | `auto` |

docs/reference/tuning-logstash.md

@@ -50,6 +50,7 @@ Make sure you’ve read the [Performance troubleshooting](/reference/performance
 If you plan to modify the default pipeline settings, take into account the following suggestions:
 
 * The total number of inflight events is determined by the product of the  `pipeline.workers` and `pipeline.batch.size` settings. This product is referred to as the *inflight count*. Keep the value of the inflight count in mind as you adjust the `pipeline.workers` and `pipeline.batch.size` settings. Pipelines that intermittently receive large events at irregular intervals require sufficient memory to handle these spikes. Set the JVM heap space accordingly in the `jvm.options` config file (See [Logstash Configuration Files](/reference/config-setting-files.md) for more info).
+* {applies_to}`stack: preview 9.2.0` Consider enabling the metering of batch sizes using the setting `pipeline.batch.metrics.sampling_mode` to help you understand the actual batch sizes being processed by your pipeline. This setting can be useful tuning the `pipeline.batch.size` setting. For more details see [logstash.yml](/reference/logstash-settings-file.md).
 * Measure each change to make sure it increases, rather than decreases, performance.
 * Ensure that you leave enough memory available to cope with a sudden increase in event size. For example, an application that generates exceptions that are represented as large blobs of text.
 * The number of workers may be set higher than the number of CPU cores since outputs often spend idle time in I/O wait conditions.

docs/static/spec/openapi/logstash-api.yaml

@@ -810,6 +810,7 @@ paths:
             - stats for each configured filter or output stage
             - info about config reload successes and failures (when [config reload](https://www.elastic.co/guide/en/logstash/current/reloading-config.html) is enabled)
             - info about the persistent queue (when [persistent queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) are enabled)
+            - metrics related to processed batch sizes. Includes the size in bytes and the number of events of batches processed in this pipeline. (when setting [pipeline.batch.metrics.sampling_mode](https://www.elastic.co/docs/reference/logstash/logstash-settings-file.html) is not `disabled`).
 
           content:
             application/json:
@@ -821,6 +822,15 @@ paths:
               example:
                 pipelines:
                   beats-es:
+                    batch:
+                      event_count:
+                        current: 78
+                        average:
+                          lifetime: 115
+                      byte_size:
+                        current: 32767
+                        average:
+                          lifetime: 14820
                     events:
                       duration_in_millis: 365495
                       in: 216610
@@ -1095,6 +1105,13 @@ paths:
                   value:
                     pipelines:
                       heartbeat-ruby-stdout:
+                        batch:
+                          event_count:
+                            average:
+                              lifetime: 115
+                          byte_size:
+                            average:
+                              lifetime: 14820
                         events:
                           queue_push_duration_in_millis: 159
                           in: 45