[Logstash integration] Add current and peak connections metrics of elastic-agent input plugin. (#15148)

* Saves current and peak connections metrics of elastic-agent input plugin and visualizes on dashboard.

* Update the type migration version which was snapshot before.

* Unify current and peak connections graphs into single view.

* Rename y-axis title.

* Rename the dashboard title and panel.

* Specify Logstash versions which expose these metrics.

Author: mashhurs

packages/logstash/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.8.0"
+  changes:
+    - description: Adds a dashboard for monitoring current and peak connections in the elastic_agent or beats input plugins. These metrics are available only in Logstash versions 8.18.7, 8.19.4, 9.0.7, 9.1.4, or later. Older versions do not expose them.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14727
 - version: "2.7.4"
   changes:
     - description: |

packages/logstash/data_stream/plugins/agent/stream/cel.yml.hbs

@@ -50,7 +50,8 @@ program: |
       :
         [],
       "counter_map": has(body.pipelines[pipeline_name].vertices) ?
-        body.pipelines[pipeline_name].vertices.map(vertex, has(vertex.long_counters), vertex.long_counters.map(counter, {
+        body.pipelines[pipeline_name].vertices.map(vertex, has(vertex.long_counters) || has(vertex.double_gauges),
+        ((has(vertex.long_counters) ? vertex.long_counters : []) + (has(vertex.double_gauges) ? vertex.double_gauges : [])).map(counter, {
           "plugin_id": vertex.id,
           "name": counter.name,
           "value": counter.value

packages/logstash/data_stream/plugins/fields/fields.yml

@@ -109,6 +109,22 @@
                   type: scaled_float
                   metric_type: gauge
                   description: throughput of this input plugin
+            - name: metrics
+              type: group
+              description: Plugin specific metrics
+              fields:
+                - name: beats
+                  type: group
+                  description: elastic_agent or beats input plugin specific metrics
+                  fields:
+                    - name: peak_connections
+                      type: long
+                      description: Number of peak connections
+                      metric_type: counter
+                    - name: current_connections
+                      type: long
+                      description: Current connections count
+                      metric_type: counter
         - name: filter
           type: group
           description: Information about filter plugins

packages/logstash/docs/README.md

@@ -687,6 +687,8 @@ pipeline collection period, and setting it to an appropriate value.
 | logstash.pipeline.plugin.input.flow.throughput.current | throughput of this input plugin | scaled_float |  | gauge |
 | logstash.pipeline.plugin.input.flow.throughput.last_1_minute | throughput of this input plugin | scaled_float |  | gauge |
 | logstash.pipeline.plugin.input.id | Id of input plugin | keyword |  |  |
+| logstash.pipeline.plugin.input.metrics.beats.current_connections | Current connections count | long |  | counter |
+| logstash.pipeline.plugin.input.metrics.beats.peak_connections | Number of peak connections | long |  | counter |
 | logstash.pipeline.plugin.input.name | Name of input plugin | keyword |  |  |
 | logstash.pipeline.plugin.input.source.column |  | keyword |  |  |
 | logstash.pipeline.plugin.input.source.id |  | keyword |  |  |