Merge pull request #2248 from splunk/urbiz-OD6444-collector-k8s-tls

[6444]: Collector for K8s - TLS config

Author: aurbiztondo-splunk

gdi/opentelemetry/collector-kubernetes/collector-configuration-tutorial-k8s/about-collector-config-tutorial.rst

@@ -1,7 +1,7 @@
 .. _about-collector-configuration-tutorial-k8s:
 
 *****************************************************************************************
-Tutorial: Configure the Splunk Distribution of OpenTelemetry Collector on Kubernetes
+Tutorial: Configure the Splunk Distribution of the OpenTelemetry Collector on Kubernetes
 *****************************************************************************************
 
 .. meta::
@@ -14,7 +14,7 @@ Tutorial: Configure the Splunk Distribution of OpenTelemetry Collector on Kubern
    collector-config-tutorial-start
    collector-config-tutorial-edit
 
-The Splunk Distribution of OpenTelemetry Collector is a :new-page:`distribution <https://docs.splunk.com/Splexicon:Distribution>` of the OpenTelemetry Collector that includes components, installers, and default settings so that it's ready to work with Splunk Observability Cloud.
+The Splunk Distribution of the OpenTelemetry Collector is a :new-page:`distribution <https://docs.splunk.com/Splexicon:Distribution>` of the OpenTelemetry Collector that includes components, installers, and default settings so that it's ready to work with Splunk Observability Cloud.
 
 Follow this tutorial for a walkthrough of configuring the Splunk Distribution of OpenTelemetry Collector to collect telemetry in common situations.
 

gdi/opentelemetry/collector-kubernetes/collector-configuration-tutorial-k8s/collector-config-tutorial-edit.rst

@@ -26,7 +26,6 @@ Download the default :new-page:`values.yaml <https://github.com/signalfx/splunk-
 
 Take a moment to read through the values.yaml file and examine its structure. Notice how each section configures the Collector for different targets, such as Splunk Observability Cloud and Splunk Cloud Platform. The comments in the file contain useful indications as to which values you can use and what's their effect.
 
-
 Configure the Splunk HEC endpoint and token
 ============================================
 
@@ -156,7 +155,10 @@ This completes the tutorial. You created a local Kubernetes cluster, configured
 
 To learn more about the Collector installation and components, see the following resources:
 
-- :ref:`otel-install-k8s`
-- :ref:`otel-kubernetes-config`
-- :ref:`splunk-hec-exporter`
+* :ref:`kubernetes-helm-architecture`
+* :ref:`otel-install-k8s`
+* :ref:`otel-kubernetes-config`
+* :ref:`kubernetes-config-add` 
+* :ref:`splunk-hec-exporter`
+
 

gdi/opentelemetry/collector-kubernetes/collector-kubernetes-intro.rst

@@ -18,8 +18,9 @@ Get started with the Collector for Kubernetes
    Install with YAML manifests <install-k8s-manifests.rst>
    Kubernetes (EKS Add-on) <install-k8s-addon-eks.rst>
    Configure with Helm <kubernetes-config.rst>
-   Advanced config <kubernetes-config-advanced.rst>
+   Add components and data sources <kubernetes-config-add.rst>
    Configure logs and events <kubernetes-config-logs.rst>
+   Advanced configuration <kubernetes-config-advanced.rst>
    Default Kubernetes metrics <metrics-ootb-k8s.rst>
    Upgrade <kubernetes-upgrade.rst>
    Uninstall <kubernetes-uninstall.rst>
@@ -55,12 +56,13 @@ Optionally, you can also:
     <h2>Configure the Collector for Kubernetes<a name="k8s-configure" class="headerlink" href="#k8s-configure" title="Permalink to this headline">¶</a></h2>
   </embed>
 
-To configure the Collector, see:
+To configure the Collector, including adding additional components or activating automatic discovery, see:
 
 * :ref:`otel-kubernetes-config`
-* :ref:`otel-kubernetes-config-advanced`
-* :ref:`kubernetes-config-logs`
+* :ref:`kubernetes-config-add` 
 * :ref:`discovery-mode-k8s`
+* :ref:`kubernetes-config-logs`
+* :ref:`otel-kubernetes-config-advanced`
 
 .. raw:: html
 

gdi/opentelemetry/collector-kubernetes/kubernetes-config-add.rst

@@ -0,0 +1,207 @@
+.. _kubernetes-config-add:
+
+**********************************************************************************************
+Configure the Collector for Kubernetes with Helm: Add components and data sources
+**********************************************************************************************
+
+.. meta::
+      :description: Optional configurations for the Splunk Distribution of OpenTelemetry Collector for Kubernetes: Add components or new data sources.
+
+Read on to learn how to add additional components or data sources to your Collector for Kubernetes config.
+
+For other config options, see:
+
+* :ref:`otel-kubernetes-config`
+* :ref:`discovery-mode-k8s`
+* :ref:`kubernetes-config-logs`
+* :ref:`otel-kubernetes-config-advanced`
+
+For a practical example of how to configure the Collector for Kubernetes see :ref:`about-collector-configuration-tutorial-k8s`.
+
+.. _otel-kubernetes-config-add-components:
+
+Add additional components to the configuration
+======================================================
+
+To use any additional OTel component, integration or legacy monitor, add it the relevant configuration sections in the values.yaml file. Depending on your requirements, you might want to include it in the ``agent.config`` or the ``clusterReceiver.config`` section of the values.yaml. See more at :ref:`helm-chart-components`.
+
+For a full list of available components and how to configure them, see :ref:`otel-components`. For a list of available application integrations, see :ref:`monitor-data-sources`.
+
+How to collect data: agent or cluster receiver?
+-----------------------------------------------------------------------------
+
+Read the following table to decide which option to chose to collect your data:
+
+.. list-table:: 
+  :header-rows: 1
+  :width: 100%
+  :widths: 20 40 40 
+
+  * - 
+    - Collect via the Collector agent 
+    - Collect via the Collector cluster receiver 
+
+  * - Where is data collected?
+    - At the node level.
+    - At the Kubernetes service level, through a single point.
+
+  * - Advantages
+    - * Granularity: This option ensures that you capture the complete picture of your cluster's performance and health. 
+      * Fault tolerance: If a node becomes isolated or experiences issues, its metrics are still being collected independently. This gives you visibility into problems affecting individual nodes.
+    - Simplicity: This option simplifies the setup and management. 
+
+  * - Considerations
+    - Complexity: Managing and configuring agents on each node can increase operational complexity, specifically agent config file management.
+    - Uncomplete data: This option might result in a partial view of your cluster's health and performance. If the service collects metrics only from a subset of nodes, you might miss critical metrics from parts of your cluster.
+
+  * - Use cases
+    - - Use this in environments where you need detailed insights into each node's operations. This allows better issue diagnosing and optimizing performance. 
+      - Use this to collect metrics from application pods that have multiple replicas that can be running on multiple nodes.
+    - Use this in environments where operational simplicity is a priority, or if your cluster is already simple and has only 1 node.
+
+Example: Add the MySQL receiver
+-----------------------------------------------------------------------------
+
+This example shows how to add the :ref:`mysql-receiver` to your configuration file.
+
+Add the MySQL receiver in the ``agent`` section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use the Collector agent daemonset to collect ``mysql`` metrics from every node the agent is deployed to, add this to your configuration:
+
+.. code:: yaml
+
+  agent:
+    config:
+      receivers:
+        mysql:
+          endpoint: localhost:3306
+          ...
+
+Add the MySQL receiver in the ``clusterReceiver`` section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use the Collector cluster receiver deployment to collect ``mysql`` metrics from a single endpoint, add this to your configuration:
+
+.. code:: yaml
+
+  clusterReceiver:
+    config:
+      receivers:
+        mysql:
+          endpoint: mysql-k8s-service:3306
+          ...
+
+Example: Add the Rabbit MQ monitor
+-----------------------------------------------------------------------------
+
+This example shows how to add the :ref:`rabbitmq` integration to your configuration file.
+
+Add RabbitMQ in the ``agent`` section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to activate the RabbitMQ monitor in the Collector agent daemonset, add ``mysql`` to the ``receivers`` section of your agent section in the configuration file:
+
+.. code:: yaml
+
+  agent:
+    config:
+      receivers:
+        smartagent/rabbitmq:
+          type: collectd/rabbitmq
+          host: localhost
+          port: 5672
+          username: otel
+          password: ${env:RABBITMQ_PASSWORD}
+
+Next, include the receiver in the ``metrics`` pipeline of the ``service`` section of your configuration file:
+
+.. code:: yaml
+
+  service:
+    pipelines:
+      metrics:
+        receivers:
+          - smartagent/rabbitmq
+
+Add RabbitMQ in the ``clusterReceiver`` section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Similarly, if you want to activate the RabbitMQ monitor in the cluster receiver, add ``mysql`` to the ``receivers`` section of your cluster receiver section in the configuration file:
+
+.. code:: yaml
+
+  clusterReceiver:
+    config:
+      receivers:
+        smartagent/rabbitmq:
+          type: collectd/rabbitmq
+          host: rabbitmq-service
+          port: 5672
+          username: otel
+          password: ${env:RABBITMQ_PASSWORD}
+
+Next, include the receiver in the ``metrics`` pipeline of the ``service`` section of your configuration file:
+
+.. code:: yaml
+
+  service:
+    pipelines:
+      metrics:
+        receivers:
+          - smartagent/rabbitmq
+
+Activate discovery mode on the Collector
+============================================
+
+Use the discovery mode of the Splunk Distribution of OpenTelemetry Collector to detect metric sources and create
+a configuration based on the results.
+
+See :ref:`discovery-mode-k8s` for instructions on how to activate discovery mode in the Helm chart.
+
+.. _otel-kubernetes-config-resources:
+
+Add additional telemetry sources
+===========================================
+
+Use the ``autodetect`` configuration option to activate additional telemetry sources.
+
+Set ``autodetect.prometheus=true`` if you want the Collector to scrape Prometheus metrics from pods that have generic Prometheus-style annotations. Add the following annotations on pods to allow a fine control of the scraping process:
+
+* ``prometheus.io/scrape: true``: The default configuration scrapes all pods. If set to ``false``, this annotation excludes the pod from the scraping process.
+* ``prometheus.io/path``: The path to scrape the metrics from. The default value is ``/metrics``.
+* ``prometheus.io/port``: The port to scrape the metrics from. The default value is ``9090``.
+
+If the Collector is running in an Istio environment, set ``autodetect.istio=true`` to make sure that all traces, metrics, and logs reported by Istio are collected in a unified manner.
+
+For example, use the following configuration to activate automatic detection of both Prometheus and Istio telemetry sources:
+
+.. code-block:: yaml
+
+  splunkObservability:
+    accessToken: xxxxxx
+    realm: us0
+  clusterName: my-k8s-cluster
+  autodetect:
+    istio: true
+    prometheus: true
+
+.. _otel-kubernetes-deactivate-telemetry:
+
+Deactivate particular types of telemetry
+============================================
+
+By default, OpenTelemetry sends only metrics and traces to Splunk Observability Cloud and sends only logs to Splunk Platform. You can activate or deactivate any kind of telemetry data collection for a specific destination. 
+
+For example, the following configuration allows the Collector to send all collected telemetry data to Splunk Observability Cloud and the Splunk Platform if you've properly configured them:
+
+.. code-block:: yaml
+
+  splunkObservability:
+    metricsEnabled: true
+    tracesEnabled: true
+    logsEnabled: true
+  splunkPlatform:
+    metricsEnabled: true
+    logsEnabled: true
+

gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst

@@ -75,7 +75,8 @@ The following table shows which Kubernetes distributions support control plane m
 
 .. list-table::
   :header-rows: 1
-  :width: 60%
+  :width: 100%
+  :widths: 50 50
 
   * - Supported
     - Unsupported
@@ -134,6 +135,7 @@ The following example shows how to connect to a nonstandard API server that uses
                 useHTTPS: true
                 useServiceAccount: false
 
+.. _kubernetes-config-advanced-non-root:
 
 Run the container in non-root user mode
 ==================================================
@@ -151,6 +153,91 @@ To run the container in ``non-root`` user mode, use ``agent.securityContext`` to
 
 .. note:: Running the collector agent for log collection in non-root mode is not currently supported in CRI-O and OpenShift environments at this time. For more details, see the :new-page:`related GitHub feature request issue <https://github.com/signalfx/splunk-otel-collector-chart/issues/891>`.
 
+.. _kubernetes-config-advanced-tls-certificates:
+
+Configure custom TLS certificates
+==================================================
+
+If your organization requires custom TLS certificates for secure communication with the Collector, follow these steps: 
+
+1. Create a Kubernetes secret containing the Root CA certificate, TLS certificate, and private key files
+---------------------------------------------------------------------------------------------------------------------
+
+Store your custom CA certificate, key, and cert files in a Kubernetes secret in the same namespace as the your Splunk Helm chart. 
+
+For example, you can run this command:
+
+.. code-block:: bash
+
+  kubectl create secret generic my-custom-tls --from-file=ca.crt=/path/to/custom_ca.crt --from-file=apiserver.key=/path/to/custom_key.key --from-file=apiserver.crt=/path/to/custom_cert.crt -n <namespace>
+
+.. Note:: You are responsible for externally managing this secret, which is not part of the Splunk Helm chart deployment.  
+
+2. Mount the secret in the Splunk Helm Chart
+-----------------------------------------------------------------------------
+
+Apply this configuration to the ``agent``, ``clusterReceiver``, or ``gateway`` using the following Helm values:
+
+* ``agent.extraVolumes``, ``agent.extraVolumeMounts``
+* ``clusterReceiver.extraVolumes``, ``clusterReceiver.extraVolumeMounts``
+* ``gateway.extraVolumes``, ``gateway.extraVolumeMounts``
+
+Learn more about Helm components at :ref:`helm-chart-components`.
+
+For example:
+
+.. code-block:: yaml
+
+  agent:
+    extraVolumes:
+      - name: custom-tls
+        secret:
+          secretName: my-custom-tls
+    extraVolumeMounts:
+      - name: custom-tls
+        mountPath: /etc/ssl/certs/
+        readOnly: true
+
+  clusterReceiver:
+    extraVolumes:
+      - name: custom-tls
+        secret:
+          secretName: my-custom-tls
+    extraVolumeMounts:
+      - name: custom-tls
+        mountPath: /etc/ssl/certs/
+        readOnly: true
+
+  gateway:
+    extraVolumes:
+      - name: custom-tls
+        secret:
+          secretName: my-custom-tls
+    extraVolumeMounts:
+      - name: custom-tls
+        mountPath: /etc/ssl/certs/
+        readOnly: true
+
+3. Override your TLS configuration
+-----------------------------------------------------------------------------
+
+Update the TLS configuration for specific Collector components, such as the agent's ``kubeletstatsreceiver``, to use the mounted certificate, key, and CA files. 
+
+For example:
+
+.. code-block:: yaml
+
+  agent:
+    config:
+      receivers:
+        kubeletstats:
+          auth_type: "tls"
+          ca_file: "/etc/ssl/certs/custom_ca.crt"
+          key_file: "/etc/ssl/certs/custom_key.key"
+          cert_file: "/etc/ssl/certs/custom_cert.crt"
+          insecure_skip_verify: true
+
+.. note:: To skip certificate checks, you can disable secure TLS checks per component. This option is not recommended for production environments due to security standards.
 
 Collect network telemetry using eBPF
 ==================================================