[Helm] Add Logstash Output to Elastic Agent Standalone

changelog/fragments/1760710876-add-support-for-logstash-output-to-elastic-agent-standalone-helm-chart.yaml

@@ -0,0 +1,45 @@
+# REQUIRED
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: add support for logstash output to elastic agent standalone helm chart
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# description:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# impact:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# action:
+
+# REQUIRED for all kinds
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component:
+
+# AUTOMATED
+# OPTIONAL to manually add other PR URLs
+# PR URL: A link the PR that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+# pr: https://github.com/owner/repo/1234
+
+# AUTOMATED
+# OPTIONAL to manually add other issue URLs
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+# issue: https://github.com/owner/repo/1234

deploy/helm/elastic-agent/examples/kubernetes-logstash-output/README.md

@@ -0,0 +1,96 @@
+# Example Kubernetes Integration with Logstash output
+
+In this example we install the built-in `kubernetes` integration with the default built-in values and a different agent output with ssl settings that allow to connect to a Logstash cluster that is exposed with mtls using a self-signed certificate.
+
+## Prerequisites:
+1. Build the dependencies of the Helm chart
+    ```console
+    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+    helm dependency build ../../
+    ```
+2. A k8s secret that contains the client cert and key for [mtls to a logstash cluster](https://www.elastic.co/docs/reference/fleet/secure-logstash-connections) stored in the files tls.crt and tls.key.
+    ```console
+    kubectl create secret tls cert-secret \
+       --cert=tls.crt \
+       --key=tls.key
+    ```
+
+3. `kubernetes` integration assets installed through Kibana ([Kibana - Install and uninstall Elastic Agent integration assets](https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html))
+4. The CA certificate, that validates the self-signed server certificate, stored in the file `ca.crt`. Example content of the `ca.crt` file.
+     ```
+     -----BEGIN CERTIFICATE-----
+     MIIDSjCCAjKgAwIBAgIRALfMeXFmYLUW4HaNXLzfP4cwDQYJKoZIhvcNAQELBQAw
+     LzETMBEGA1UECxMKbW9uaXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRw
+     MB4XDTI0MTIxMTEwMTMzNVoXDTI1MTIxMTEwMjMzNVowLzETMBEGA1UECxMKbW9u
+     aXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRwMIIBIjANBgkqhkiG9w0B
+     AQEFAAOCAQ8AMIIBCgKCAQEAsljXOJrCsvZGHr2SroKUGJOnJwtz8VTx2spQ96OO
+     8Q+Tw8gX5C32bjplwAeQsnZ7i5YRRLneaG6NXJuaUEDefsKeG6jdN/bjce+Sz5xm
+     U6guXe3TuIyk0+UoFtOzZ1lYUNk6lg9+60iOllRO3xI7SwxqKAaC4KKs7QL1jQCR
+     Q14QedcPrS4v76OT+TJvYWrbTFLtYYvfJDGop5EE90v7iB5j0ehSLjfC2R4CD5Kr
+     OSYJrGqnhnznbUUjulVqCkPKmgZdcvcIBn4NnZlN6oYzwhRHSSj6r3sy11j3A6SA
+     7KeG+IlY+LmRtrj85tiRJ3pXz1FD2d/Mf6cNI6lBGRrNZwIDAQABo2EwXzAOBgNV
+     HQ8BAf8EBAMCAoQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1Ud
+     EwEB/wQFMAMBAf8wHQYDVR0OBBYEFMgVU7RwXciOOz18FcQDTQZXy9gIMA0GCSqG
+     SIb3DQEBCwUAA4IBAQCgOSe2s3Xc0QKR+86xmoAADpoe7SFT0Yyh3rMjL+0p02m3
+     CqrILqCRNFu9az8gc47hUt9Crb1BXmTR0Sb23M1NvGmR2D2K7CLp/SvkAP6RlB4M
+     dZ70UKw4ohq+VSSSiLOoHYdlH46xtunLL31GLYRwD+OgeKAc5pwqWgZkndzxrouB
+     uNyoxB5NGvaVUqIouILQ9V2fvraCNf+RxuQ0AaPxdt/CNpFaXpbJBuXJCphlydu0
+     KztVqRv5EZjuYpcXDfGP9BEvMy6o895H4iG0M2wb2e3WEDo6jH5pecZfc4yz8iae
+     jLwbOPbWqOGRkxTMLOV6Q1dtr09zf2SuOQuxm7F2
+     -----END CERTIFICATE-----
+     ```
+5. A logstash instance running with the server tls certificates that match the client set used by elastic agent. Running a pipeline similar to the below:
+
+```
+    input {
+      elastic_agent {
+        port => 5044
+        ssl_enabled => true
+        ssl_certificate_authorities => ["/opt/logstash/ssl/ca.crt"]
+        ssl_certificate => "/opt/logstash/ssl/logstash.crt"
+        ssl_key => "/opt/logstash/ssl/logstash.pkcs8.key"
+        ssl_client_authentication => "required"
+      }
+    }
+    filter {
+       elastic_integration {
+         cloud_id => "${CLOUD_ID}"
+         username => "${ES_USER}"
+         password => "${ES_PASSWORD}"
+       }
+    }
+    output {
+        elasticsearch {
+            cloud_id => "${CLOUD_ID}"
+            user => "${ES_USER}"
+            password => "${ES_PASSWORD}"
+            data_stream => "true"
+            manage_template => "false"
+        }
+    }
+```
+## Run:
+```console
+helm install elastic-agent ../../ \
+     -f ./agent-kubernetes-values.yaml \
+     --set outputs.test.hosts[0]="logstash-agent.default.svc.cluster.local:5044" \
+     --set outputs.test.type=Logstash \
+     --set-file outputs.test.ssl.certificateAuthorities[0].value=ca.crt \
+     --set outputs.test.ssl.certificate.valueFromSecret.key=tls.crt \
+     --set outputs.test.ssl.certificate.valueFromSecret.name=cert-secret \
+     --set outputs.test.ssl.key.valueFromSecret.key=tls.key \
+     --set outputs.test.ssl.key.valueFromSecret.name=cert-secret \
+     --set agent.presets.perNode.agent.monitoring.use_output=test \
+     --set agent.presets.clusterWide.agent.monitoring.use_output=test \
+     --set kubernetes.output=test
+```
+
+## Validate:
+
+1. `kube-state metrics` is installed with this command `kubectl get deployments -n kube-system kube-state-metrics`.
+2. Kubernetes data ship to the Logstash Cluster of the `test` output.
+3. The Kibana `kubernetes`-related dashboards should start showing up the respective info.
+
+## Note:
+
+1. If you want to disable kube-state-metrics installation with the elastic-agent Helm chart, you can set `kube-state-metrics.enabled=false` in the Helm chart. The helm chart will use the value of `kubernetes.state.host` to configure the elastic-agent input.

deploy/helm/elastic-agent/examples/kubernetes-logstash-output/agent-kubernetes-values.yaml

@@ -0,0 +1,73 @@
+kubernetes:
+  enabled: true
+  output: test
+  namespace: test
+  scheduler:
+    enabled: true
+  controller_manager:
+    enabled: true
+
+outputs:
+  test:
+    type: Logstash
+    hosts:
+    - myhost:5443
+    loadbalance: true
+    ttl: "10s"
+    slow_start: false
+    timeout: "30s"
+
+    workers: 5
+    pipelining: 1
+    bulk_max_size: 1600
+    queue.mem.flush.timeout: "30s"
+    queue.mem.flush.min_events: 1600
+    queue.mem.events:  3200
+    max_retries: 3
+    compression_level: 5
+    backoff.max: "30s"
+    backoff.init: "1s"
+    allow_older_versions: false
+    ssl:
+      certificate: 
+        valueFromSecret: 
+            name: cert-secret
+            key: cert
+      key: 
+        valueFromSecret:
+            name: cert-secret
+            key: key
+      verificationMode: certificate
+      certificateAuthorities:
+        - value: |-
+            -----BEGIN CERTIFICATE-----
+            MIIDSjCCAjKgAwIBAgIRALx9cV4kYFWe1ZtMXLzfX4cwDQYJKoZIhvcNAQELBQAw
+            LzETMBEGA1UECxMKbW9uaXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRw
+            MB4XDTI1MDQwODEyMDAwMFoXDTI2MDQwODEyMDAwMFowLzETMBEGA1UECxMKbW9u
+            aXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRwMIIBIjANBgkqhkiG9w0B
+            AQEFAAOCAQ8AMIIBCgKCAQEAwk7mDcV4hO6WKnz2z2gC+ODb2xF6aP6sB5o3rZ2F
+            R0vbPY9EYI56bxSB3zT5mx0I3XsdKy8JK3AeY32K5CPv7tkHeGqTxJfYKwKXWpO4
+            rN9i3PtZZ3ZX5iPBiGUKuZJH2HQolZt/8OoyH5n3FEs/zNVbrXL6RuMjVzRGyJhR
+            UeYpXib35SKvwWWNHh9fHV1m8WUdJxvEY9ohkry0ENj6tzB9Om+yKq5Scdw3hC5h
+            X1f6+fY7ElZjVQ2lXnyrRHKpLjcDhI9cRm7r1qeKcHQMxz4Rzk3NYOBfMiEn+pns
+            OLqUz9tK0q3VgbIUlFe94CEZ2DDkRpsIqyfRTUutOwIDAQABo2EwXzAOBgNVHQ8B
+            Af8EBAMCAoQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB
+            /wQFMAMBAf8wHQYDVR0OBBYEFDUYT4RsXgjJJz38XfQHTTZVr8gOMA0GCSqGSIb3
+            DQEBCwUAA4IBAQCvIXs1b8Pe4W5PbqQtnAV+wsyq4XqB7kFNTfZBx8eh5RMqG4y8
+            pTRSDM3I+hMI1K2XpMyUM4dyaZPz2yxOZ7UgvxFwLdHDb9uYaXJ38Rm3FxV9Z+Lr
+            ZPbXce2po5WT1em8MyxLgTStbMaXYHlsk09Fj4hF9Gy93bz4iVR7cUbDgkZd7ZPP
+            uN2bDD9kU+1iyqYv44YGbEv0UTKhrT37GkBzZ3H4cGy8sKnuwz7Z0I8v+FZzKN4L
+            Knk2MCWxZQ6t4nnqmkFoK4nSlAmW5Fnj7c2YwPYsZnscEcFlS0YDNmwC8wvDdDHL
+            tD0YrMZxZDh0LAGWAW72rT+mSPCWq5eB2ZtS
+            -----END CERTIFICATE-----
+agent:
+  unprivileged: true
+  presets:
+    perNode:
+      agent:
+        monitoring:
+          use_output: test
+    clusterWide:
+      agent:
+        monitoring:
+          use_output: test