Merge pull request #2738 from splunk/urbiz-OD7149-remove-fluentd

[7149]: Remove Fluentd in the Colector for L/W

Author: aurbiztondo-splunk

_includes/requirements/collector-linux.rst

@@ -1,10 +1,10 @@
 The Collector supports the following Linux distributions and versions:
 
-* Amazon Linux: 2, 2023. Log collection with Fluentd is not currently supported for Amazon Linux 2023.
+* Amazon Linux: 2, 2023. 
 * CentOS: 7, 8, 9
 * Red Hat: 7, 8, 9
 * Oracle: 8, 9
 * Debian: 11, 12
-* SUSE: 12, 15 for version 0.34.0 or higher. Log collection with Fluentd is not currently supported.
+* SUSE: 12, 15 for version 0.34.0 or higher. 
 * Ubuntu: 16.04, 18.04, 20.04, 22.04, and 24.04
 * Rocky Linux: 8, 9

admin/authentication/allow-services.rst

@@ -216,9 +216,6 @@ If you're unable to allow all URLs as described in :ref:`allow-urls`, ensure tha
    # RUM ingest endpoint 
    rum-ingest.<YOUR_REALM>.signalfx.com/v1/rum
    
-   # For td-agent/Fluentd on Linux and Windows
-   packages.treasuredata.com
-   
    # For DEB/RPM collector packages
    splunk.jfrog.io 
    jfrog-prod-use1-shared-virginia-main.s3.amazonaws.com

gdi/opentelemetry/automatic-discovery/linux/linux-backend.rst

@@ -89,8 +89,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                             curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
                             sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
 
-                        .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option.
-
                         The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/java.conf``.
 
                         To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment <env>`` option. Replace ``<env>`` with the desired attribute value, for example, ``prod``, as shown in the following example:
@@ -125,8 +123,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                         
                         The ``systemd`` instrumentation automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``.
 
-                        .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. 
-
                         To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment <env>`` option. Replace ``<env>`` with the desired attribute value, for example, ``prod``, as shown in the following example:
 
                         .. code-block:: bash
@@ -230,7 +226,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                                 curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
                                 sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
 
-                            .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. 
 
                         The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/node.conf``.
 
@@ -257,8 +252,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                             
                             The ``systemd`` zero-code instrumentation method automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``.
 
-                            .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option.
-
                         You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:
 
                         .. code-block:: bash
@@ -309,8 +302,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                             curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
                             sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>
 
-                        .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option.
-
                         The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/dotnet.conf``.
 
                         To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment <env>`` option. Replace ``<env>`` with the desired attribute value, for example, ``prod``, as shown in the following example:
@@ -345,8 +336,6 @@ Using the installer script, you can install and activate zero-code instrumentati
                                 
                         The ``systemd`` instrumentation automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``.
 
-                        .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. 
-
                         To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment <env>`` option. Replace ``<env>`` with the desired attribute value, for example, ``prod``, as shown in the following example:
 
                         .. code-block:: bash

gdi/opentelemetry/collector-linux/collector-configuration-tutorial/collector-config-tutorial-start.rst

@@ -66,7 +66,6 @@ After you've installed the Collector, navigate to /etc/otel/collector to find th
    .
    |-- agent_config.yaml
    |-- config.d
-   |-- fluentd
    |-- gateway_config.yaml
    |-- splunk-otel-collector.conf
    |-- splunk-otel-collector.conf.example

gdi/opentelemetry/collector-linux/deployments-linux-ansible.rst

@@ -96,14 +96,6 @@ The following table describes the variables that can be configured for this role
      - The amount of allocated memory in MiB. The default value is ``512``, or 500 x 2^20 bytes, of memory .
    * - ``splunk_ballast_size_mib``
      - ``splunk_ballast_size_mib`` is deprecated starting on Collector version 0.97.0. If you're using it, see :ref:`how to update your configuration <collector-upgrade-memory-ballast>`.
-   * - ``install_fluentd``
-     - The option to install or manage Fluentd and dependencies for log collection. The dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries or development tools. The default value is ``false``.
-   * - ``td_agent_version``
-     - The version of td-agent (Fluentd package) that is installed. The default value is ``3.3.0`` for Debian jessie, ``3.7.1`` for Debian stretch, and ``4.3.0`` for other distros.
-   * - ``splunk_fluentd_config``
-     - The path to the Fluentd configuration file on the remote host. The default location is ``/etc/otel/collector/fluentd/fluent.conf``.
-   * - ``splunk_fluentd_config_source``
-     - The source path to a Fluentd configuration file on your control host that is uploaded and set in place of the value set in ``splunk_fluentd_config`` on remote hosts. Use this variable to submit a custom Fluentd configuration, for example, ``./custom_fluentd_config.conf``. The default value is ``""``, which means that nothing is copied and the configuration file set with ``splunk_otel_collector_config`` is used.
 
 .. _ansible-zero-config:
 

gdi/opentelemetry/collector-linux/deployments-linux-chef.rst

@@ -112,18 +112,6 @@ For Linux, the cookbook accepts the attributes described in the following table:
    * - ``package_stage``
      - The Collector package repository stage to use. Can be ``release``, ``beta``, or ``test``.
      - ``release``
-   * - ``with_fluentd``
-     - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools.
-     - ``false``
-   * - ``fluentd_version``
-     -  Version of the td-agent (Fluentd) package to install 
-     -  ``3.7.1`` for Debian stretch and ``4.3.1`` for all other Linux distros 
-   * - ``fluentd_config_source``
-     - Source path to the Fluentd configuration file. This file is copied to the ``$fluentd_config_dest`` path on the node. See the :new-page:`source attribute <https://puppet.com/docs/puppet/latest/types/file.html#file-attribute-source>` of the file resource for the supported value types. The default source file is provided by the Collector package. Only applicable if ``$with_fluentd`` is set to ``true``.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
-   * - ``fluentd_config_dest``
-     - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
 
 .. _chef-zero-config:
 

gdi/opentelemetry/collector-linux/deployments-linux-puppet.rst

@@ -101,18 +101,6 @@ The class accepts the parameters described in the following table:
    * - ``service_user and $service_group``
      - Sets the user or group ownership for the Collector service. The user or group is created if they do not exist.
      - ``splunk-otel-collector``
-   * - ``with_fluentd``
-     - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools.
-     - ``false``
-   * - ``fluentd_config_source``
-     - Source path to the Fluentd configuration file. This file is copied to the ``$fluentd_config_dest`` path on the node. See the :new-page:`source attribute <https://puppet.com/docs/puppet/latest/types/file.html#file-attribute-source>` of the file resource for the supported value types. The default source file is provided by the Collector package. Only applicable if ``$with_fluentd`` is set to ``true``.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
-   * - ``fluentd_config_dest``
-     - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
-   * - ``manage_repo`` 
-     - In cases where the Collector and Fluentd apt/yum repositories are managed externally, set this to ``false`` to deactivate management of the repositories by this module. If set to ``false``, the externally managed repositories should provide the ``splunk-otel-collector`` and ``td-agent`` packages. Also, the apt (``/etc/apt/sources.list.d/splunk-otel-collector.list`` and ``/etc/apt/sources.list.d/splunk-td-agent.list``) and yum (``/etc/yum.repos.d/splunk-otel-collector.repo`` and ``/etc/yum.repos.d/splunk-td-agent.repo``) repository definition files are deleted if they exist in order to avoid any conflicts.
-     - ``true``
 
 .. _puppet-zero-config:
 

gdi/opentelemetry/collector-linux/deployments-linux-salt.rst

@@ -110,21 +110,6 @@ For Linux, the formula accepts the attributes described in the following table:
    * - ``service_user`` and ``$service_group``
      - Sets the user or group ownership for the Collector service. The user or group is created if they do not exist.
      - ``splunk-otel-collector``
-   * - ``install_fluentd``
-     - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools.
-     - ``false``
-   * - ``td_agent_version``
-     -  Version of the td-agent (Fluentd) package to install 
-     -  ``4.3.0`` 
-   * - ``splunk_fluentd_config``
-     - The path to the Fluentd configuration file on the remote host.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
-   * - ``splunk_fluentd_config_source``
-     - The source path to a Fluentd configuration file on your control host that is uploaded and set in place of the ``splunk_fluentd_config`` file on remote hosts. To use a custom Fluentd configuration file, add the configuration file into the Salt dir. For example, ``salt://templates/td_agent.conf``. 
-     - ``""`` meaning that nothing is copied and the existing ``splunk_fluentd_config`` file is used.
-   * - ``fluentd_config_dest``
-     - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``.
-     - ``/etc/otel/collector/fluentd/fluent.conf``
 
 .. _salt-zero-config:
 

gdi/opentelemetry/collector-linux/install-linux-manual.rst

@@ -89,7 +89,6 @@ See also:
 
 * :ref:`linux-packages-post`
 * :ref:`linux-packages-auto`
-* :ref:`linux-packages-fluentd`
 
 .. _linux-packages-rpm:
 
@@ -164,7 +163,6 @@ See also:
 
 * :ref:`linux-packages-post`
 * :ref:`linux-packages-auto`
-* :ref:`linux-packages-fluentd`
 
 .. _linux-packages:
 
@@ -213,7 +211,6 @@ See also:
 
 * :ref:`linux-packages-post`
 * :ref:`linux-packages-auto`
-* :ref:`linux-packages-fluentd`
 
 .. _linux-packages-post:
 
@@ -305,35 +302,6 @@ The ``splunk-otel-auto-instrumentation`` deb/rpm package installs and supports c
 
 To learn more, see :ref:`linux-backend-auto-discovery`.
 
-.. _linux-packages-fluentd:
-
-Install and configure Fluentd for log collection
---------------------------------------------------------------
-
-If you require log collection, perform the following steps to install Fluentd and forward collected log events to the Collector. This requires root privileges.
-
-#. Install, configure, and start the Collector as described in :ref:`linux-packages-repo`. The Collector's default configuration file listens for log events on ``127.0.0.1:8006`` and sends them to Splunk Observability Cloud.
-
-#. Install the ``td-agent`` package appropriate for the Linux distribution/version of the target system. Find the package in :new-page:`Fluentd installation <https://docs.fluentd.org/installation>`. 
-
-   * If necessary, install the ``capng_c`` plugin and dependencies to enable Linux capabilities, for example ``cap_dac_read_search`` and/or ``cap_dac_override``. This requires ``td-agent`` version 4.1 or higher. See :new-page:`Linux capabilities <https://docs.fluentd.org/deployment/linux-capability>`.
-
-   * If necessary, install the ``fluent-plugin-systemd`` plugin to collect log events from the systemd journal. See :new-page:`Fluent plugin systemd <https://github.com/fluent-plugin-systemd/fluent-plugin-systemd>`.
-
-#. Configure Fluentd to collect log events and forward them to the Collector:
-
-   * Option 1: Update the default config file at /etc/td-agent/td-agent.conf provided by the Fluentd package to collect the desired log events and forward them to ``127.0.0.1:8006``.
-
-   * Option 2: The installed Collector package provides a custom Fluentd config file /etc/otel/collector/fluentd/fluent.conf to collect log events from many popular services and forwards them to ``127.0.0.1:8006``. To use these files, you need to override the default config file path for the Fluentd service. To do this, copy the systemd environment file from /etc/otel/collector/fluentd/splunk-otel-collector.conf to /etc/systemd/system/td-agent.service.d/splunk-otel-collector.conf.
-
-#. Ensure that the ``td-agent`` service user/group has permissions to access to the config file(s) from the previous step.
-
-#. Restart the Fluentd service to apply the changes by running ``systemctl restart td-agent``.
-
-#. View Fluentd service logs and errors in /var/log/td-agent/td-agent.log.
-
-See :new-page:`Fluentd configuration <https://docs.fluentd.org/configuration>` for general Fluentd configuration details.
-
 .. _linux-docker:
 
 Docker

gdi/opentelemetry/collector-linux/install-linux.rst

@@ -41,7 +41,6 @@ Included packages
 The installer script deploys and configures these elements:
 
 * The Splunk Distribution of the OpenTelemetry Collector for Linux
-* Fluentd, using the td-agent. Turned off by default. See :ref:`fluentd-manual-config-linux` and :ref:`fluentd-receiver` for more information
 * JMX metric gatherer
 
 .. _linux-scripts:
@@ -87,15 +86,15 @@ To configure proxy settings to install and run the OpenTelemetry Collector, see
 Use configured repos
 --------------------------------
 
-By default, apt/yum/zypper repo definition files are created to download the package and Fluentd deb/rpm packages from
+By default, apt/yum/zypper repo definition files are created to download the package from
 :new-page:`https://splunk.jfrog.io/splunk <https://splunk.jfrog.io/splunk>` and :new-page:`https://packages.treasuredata.com <https://packages.treasuredata.com>`, respectively.
 
-To skip these steps and use configured repos on the target system that provide the ``splunk-otel-collector`` and ``td-agent`` deb/rpm packages, specify the ``--skip-collector-repo`` or ``--skip-fluentd-repo`` options. For example:
+To skip these steps and use configured repos on the target system that provide the ``splunk-otel-collector`` packages, use the ``--skip-collector-repo``  option. For example:
 
 .. code-block:: bash
 
    curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
-   sudo sh /tmp/splunk-otel-collector.sh --realm $SPLUNK_REALM --skip-collector-repo --skip-fluentd-repo \
+   sudo sh /tmp/splunk-otel-collector.sh --realm $SPLUNK_REALM --skip-collector-repo \
     -- $SPLUNK_ACCESS_TOKEN
 
 .. _configure-auto-instrumentation-linux:
@@ -156,7 +155,7 @@ To use host bindings, run this command:
 Options of the installer script of the Collector for Linux
 ==================================================================
 
-The Linux installer script supports the following options for the Collector, automatic discovery with zero-code instrumentation for back-end services, and Fluentd.
+The Linux installer script supports the following options for the Collector, automatic discovery with zero-code instrumentation for back-end services.
 
 To display all the configuration options supported by the script, use the ``-h`` flag.
 
@@ -282,29 +281,9 @@ Automatic discovery with zero-code instrumentation for back-end services
      - The ``splunk-otel-auto-instrumentation`` package version to install. Note: The minimum supported version for Java and Node.js zero-code instrumentation is 0.87.0, and the minimum supported version for .NET zero-code instrumentation is 0.99.0.
      - ``latest``
 
-Fluentd
---------------------------------------------------------------------
-
-.. list-table::
-   :header-rows: 1
-   :width: 100%
-   :widths: 30 40 30
-
-   * - Option
-     - Description
-     - Default value
-   * - ``--with[out]-fluentd``
-     - Whether to install and configure fluentd to forward log events to the Collector. See :ref:`fluentd-manual-config-linux` for more information.
-     - ``--without-fluentd``
-   * - ``--skip-fluentd-repo``
-     - By default, a apt/yum repo definition file will be created to download the fluentd deb/rpm package from ``https://packages.treasuredata.com``. Use this option to skip the previous step and use a pre-configured repo on the target system that provides the ``td-agent`` deb/rpm package.
-     -
-
 Next steps
 ==================================
 
-
-
 .. raw:: html
 
    <div class="include-start" id="gdi/collector-linux-next-steps.rst"></div>