From e07b222d1b63cd770a162c6958dff7558d7466ec Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Fri, 21 Mar 2025 13:14:15 -0400 Subject: [PATCH 01/16] Added new histogram information per SME feedback --- apm/span-tags/metricsets.rst | 154 +++++++++++++++++++++-------------- 1 file changed, 92 insertions(+), 62 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index 07ca6e8d0..05a3c0a8d 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -9,42 +9,6 @@ Learn about MetricSets in APM MetricSets are key performance indicators, like request rate, error rate, and request duration, that are calculated from traces and spans in Splunk APM. There are 2 categories of MetricSets: Troubleshooting MetricSets (TMS), used for high-cardinality troubleshooting, and Monitoring MetricSets (MMS), used for real-time monitoring. MetricSets are similar to the metric time series (MTS) used in Splunk Infrastructure Monitoring to populate charts and generate alerts. See :ref:`metric-time-series` to learn more. MetricSets are MTS that are specific to Splunk APM. -.. _troubleshooting-metricsets: - -Troubleshooting MetricSets -========================== - -Troubleshooting MetricSets (TMS) are metric time series (MTS) you can use for troubleshooting high-cardinality identities in APM. You can also use TMS to make historical comparisons across spans and workflows. - -Splunk APM indexes and creates Troubleshooting MetricSets for several span tags by default. For more details about each of these tags, see :ref:`apm-default-span-tags`. You can't modify or stop APM from indexing these span tags. - -You can also create custom TMS by indexing additional span tags and processes. To learn how to index span tags and processes to create new Troubleshooting MetricSets, see :ref:`apm-index-span-tags`. - -Available TMS metrics ------------------------ -Every TMS creates the following metrics, known as request, error, and duration (RED) metrics. RED metrics appear when you select a service in the service map. See :ref:`service-map` to learn more about using RED metrics in the service map. - -- Request rate -- Error rate -- Root cause error rate -- p50, p90, and p99 latency - -The measurement precision of Troubleshooting MetricSets is 10 seconds. Splunk APM reports quantiles from a distribution of metrics for each 10-second reporting window. - -Use TMS within Splunk APM ----------------------------------------- - -TMS appear on the service map and in Tag Spotlight. Use TMS to filter the service map and create breakdowns across the values of a given indexed span tag or process. - -See :ref:`apm-service-map` and :ref:`apm-tag-spotlight`. - -TMS retention period ------------------------------------ - -Splunk Observability Cloud retains TMS for the same amount of time as raw traces. By default, the retention period is 8 days. - -For more details about Troubleshooting MetricSets, see :ref:`apm-index-tag-tips`. - .. _monitoring-metricsets: Monitoring MetricSets @@ -76,38 +40,68 @@ MMS are available for the following APM components: - traces - workflows (Workflow metrics are created by default when you create a Business Workflow. Custom MMS are not available for Business Workflows.) -Each MMS includes 6 metrics for each component. For histogram MMS, there is a single metric for each component. Use the histogram functions to access the specific histogram bucket you want to use. +Monitoring MetricSets in APM are classified as histogram metrics. Previously, MMS were classified as either counter or gauge metrics. A histogram metric type represents a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide a noticeable advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. For each metric, there is 1 metric time series (MTS) with responses ``sf_error: true`` or ``sf_error: false``. .. list-table:: - :widths: 33 33 33 - :width: 100 - :header-rows: 1 - - * - Description - - MMS - - Histogram MMS - * - Request count - - ``.count`` - - ```` with a ``count`` function - * - Minimum request duration - - ``.duration.ns.min`` - - ```` with a ``min`` function - * - Maximum request duration - - ``.duration.ns.max`` - - ```` with a ``max`` function - * - Median request duration - - ``.duration.ns.median`` - - ```` with a ``median`` function - * - Percentile request duration - - ``.duration.ns.p90`` - - ```` with a ``percentile`` function and a percentile ``value`` - * - Percentile request duration - - ``.duration.ns.p99`` - - ```` with a ``percentile`` function and a percentile ``value`` + :widths: 33 33 33 + :width: 100 + :header-rows: 1 + + * - Description + - Histogram MMS + - MMS (deprecated) + * - Request count + - ```` with a ``count`` function + - ``.count`` + * - Minimum request duration + - ```` with a ``min`` function + - ``.duration.ns.min`` + * - Maximum request duration + - ```` with a ``max`` function + - ``.duration.ns.max`` + * - Median request duration + - ```` with a ``median`` function + - ``.duration.ns.median`` + * - Percentile request duration + - ```` with a ``percentile`` function and a percentile ``value`` + - ``.duration.ns.p90`` + * - Percentile request duration + - ```` with a ``percentile`` function and a percentile ``value`` + - ``.duration.ns.p99`` + + +Example histogram MetricSets in APM +--------------------------------------------- + +A histogram MTS uses the following syntax using Signalflow: + +.. code-block:: none + histogram(metric=[,filter=][,resolution=) +.. list-table:: + :widths: 33 33 33 + :width: 100 + :header-rows: 1 + + + * - Description + - Previous MMS syntax + - Histogram MMS syntax + * - Aggregate count of all MTS + - ``A = data('spans.count').sum().publish(label='A')`` + - ``A = histogram('spans').count().publish(label='A')`` + * - P90 percentile for single MTS + - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') + A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` + - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') + A = histogram('spans', filter=filter_).percentile(pct=90).publish(label='A')`` + * - Combined p90 for multiple services + - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts'), rollup='average').mean().publish(label='A')`` + - ``A = histogram('service.request', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts')).percentile(pct=90).publish(label='A')`` + Each MMS has a set of dimensions you can use to monitor and alert on service performance. .. _service-mms: @@ -197,6 +191,42 @@ MMS retention period Splunk Observability Cloud stores MMS for 13 months by default. +.. _troubleshooting-metricsets: + +Troubleshooting MetricSets +========================== + +Troubleshooting MetricSets (TMS) are metric time series (MTS) you can use for troubleshooting high-cardinality identities in APM. You can also use TMS to make historical comparisons across spans and workflows. + +Splunk APM indexes and creates Troubleshooting MetricSets for several span tags by default. For more details about each of these tags, see :ref:`apm-default-span-tags`. You can't modify or stop APM from indexing these span tags. + +You can also create custom TMS by indexing additional span tags and processes. To learn how to index span tags and processes to create new Troubleshooting MetricSets, see :ref:`apm-index-span-tags`. + +Available TMS metrics +----------------------- +Every TMS creates the following metrics, known as request, error, and duration (RED) metrics. RED metrics appear when you select a service in the service map. See :ref:`service-map` to learn more about using RED metrics in the service map. + +- Request rate +- Error rate +- Root cause error rate +- p50, p90, and p99 latency + +The measurement precision of Troubleshooting MetricSets is 10 seconds. Splunk APM reports quantiles from a distribution of metrics for each 10-second reporting window. + +Use TMS within Splunk APM +---------------------------------------- + +TMS appear on the service map and in Tag Spotlight. Use TMS to filter the service map and create breakdowns across the values of a given indexed span tag or process. + +See :ref:`apm-service-map` and :ref:`apm-tag-spotlight`. + +TMS retention period +----------------------------------- + +Splunk Observability Cloud retains TMS for the same amount of time as raw traces. By default, the retention period is 8 days. + +For more details about Troubleshooting MetricSets, see :ref:`apm-index-tag-tips`. + Comparing Monitoring MetricSets and Troubleshooting MetricSets ================================================================= From b835005b22ffaa942af74a54c015bd15d5989924 Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Fri, 21 Mar 2025 13:28:38 -0400 Subject: [PATCH 02/16] Added note --- apm/span-tags/metricsets.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index 05a3c0a8d..4467752ae 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -40,7 +40,7 @@ MMS are available for the following APM components: - traces - workflows (Workflow metrics are created by default when you create a Business Workflow. Custom MMS are not available for Business Workflows.) -Monitoring MetricSets in APM are classified as histogram metrics. Previously, MMS were classified as either counter or gauge metrics. A histogram metric type represents a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide a noticeable advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. +Monitoring MetricSets in APM are generated as histogram metrics. Previously, MMS were classified as either a counter or gauge metric type. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. For each metric, there is 1 metric time series (MTS) with responses ``sf_error: true`` or ``sf_error: false``. @@ -72,24 +72,23 @@ For each metric, there is 1 metric time series (MTS) with responses ``sf_error: - ``.duration.ns.p99`` -Example histogram MetricSets in APM +Example histogram metrics in APM --------------------------------------------- -A histogram MTS uses the following syntax using Signalflow: +A histogram MTS uses the following syntax using SignalFlow: .. code-block:: none histogram(metric=[,filter=][,resolution=) - +The following table displays example SignalFlow .. list-table:: :widths: 33 33 33 :width: 100 :header-rows: 1 - * - Description - - Previous MMS syntax - - Histogram MMS syntax + - Previous MMS function + - Histogram MMS function * - Aggregate count of all MTS - ``A = data('spans.count').sum().publish(label='A')`` - ``A = histogram('spans').count().publish(label='A')`` @@ -102,6 +101,8 @@ A histogram MTS uses the following syntax using Signalflow: - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts'), rollup='average').mean().publish(label='A')`` - ``A = histogram('service.request', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts')).percentile(pct=90).publish(label='A')`` +.. note:: Because an aggregation is applied on histogram(), to display all of the metric sets separately, each dimension needs to be applied as a groupby. + Each MMS has a set of dimensions you can use to monitor and alert on service performance. .. _service-mms: From 5b495c99f4eaf20cfe7bcc41ad73a13647c31ff5 Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Fri, 21 Mar 2025 13:39:00 -0400 Subject: [PATCH 03/16] Missing phrase --- apm/span-tags/metricsets.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index 4467752ae..ad8103ff2 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -80,7 +80,7 @@ A histogram MTS uses the following syntax using SignalFlow: .. code-block:: none histogram(metric=[,filter=][,resolution=) -The following table displays example SignalFlow +The following table displays example SignalFlow functions: .. list-table:: :widths: 33 33 33 :width: 100 From 3d30fa327af0f2d342d19fbfae8d217bf132db54 Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Fri, 21 Mar 2025 13:58:52 -0400 Subject: [PATCH 04/16] Fixed table formatting --- apm/span-tags/metricsets.rst | 49 ++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index ad8103ff2..f06f7845f 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -50,26 +50,26 @@ For each metric, there is 1 metric time series (MTS) with responses ``sf_error: :header-rows: 1 * - Description - - Histogram MMS - - MMS (deprecated) + - Histogram MMS + - MMS (deprecated) * - Request count - - ```` with a ``count`` function - - ``.count`` + - ```` with a ``count`` function + - ``.count`` * - Minimum request duration - - ```` with a ``min`` function - - ``.duration.ns.min`` + - ```` with a ``min`` function + - ``.duration.ns.min`` * - Maximum request duration - - ```` with a ``max`` function - - ``.duration.ns.max`` + - ```` with a ``max`` function + - ``.duration.ns.max`` * - Median request duration - - ```` with a ``median`` function - - ``.duration.ns.median`` + - ```` with a ``median`` function + - ``.duration.ns.median`` * - Percentile request duration - - ```` with a ``percentile`` function and a percentile ``value`` - - ``.duration.ns.p90`` + - ```` with a ``percentile`` function and a percentile ``value`` + - ``.duration.ns.p90`` * - Percentile request duration - - ```` with a ``percentile`` function and a percentile ``value`` - - ``.duration.ns.p99`` + - ```` with a ``percentile`` function and a percentile ``value`` + - ``.duration.ns.p99`` Example histogram metrics in APM @@ -78,28 +78,29 @@ Example histogram metrics in APM A histogram MTS uses the following syntax using SignalFlow: .. code-block:: none + histogram(metric=[,filter=][,resolution=) + The following table displays example SignalFlow functions: + .. list-table:: :widths: 33 33 33 :width: 100 :header-rows: 1 * - Description - - Previous MMS function - - Histogram MMS function + - Previous MMS function + - Histogram MMS function * - Aggregate count of all MTS - - ``A = data('spans.count').sum().publish(label='A')`` - - ``A = histogram('spans').count().publish(label='A')`` + - ``A = data('spans.count').sum().publish(label='A')`` + - ``A = histogram('spans').count().publish(label='A')`` * - P90 percentile for single MTS - - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') - A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` - - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') - A = histogram('spans', filter=filter_).percentile(pct=90).publish(label='A')`` + - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` + - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = histogram('spans', filter=filter_).percentile(pct=90).publish(label='A')`` * - Combined p90 for multiple services - - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts'), rollup='average').mean().publish(label='A')`` - - ``A = histogram('service.request', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts')).percentile(pct=90).publish(label='A')`` + - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts'), rollup='average').mean().publish(label='A')`` + - ``A = histogram('service.request', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts')).percentile(pct=90).publish(label='A')`` .. note:: Because an aggregation is applied on histogram(), to display all of the metric sets separately, each dimension needs to be applied as a groupby. From 5c0066ad976c0137f89d2a836d08fac649851984 Mon Sep 17 00:00:00 2001 From: jcatera-splunk Date: Fri, 21 Mar 2025 12:07:40 -0700 Subject: [PATCH 05/16] Update metricsets.rst Just making a small change to troubleshoot the broken table. I added a space at the end of the page. --- apm/span-tags/metricsets.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index f06f7845f..ce02125ea 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -236,4 +236,4 @@ Because endpoint-level and service-level MMS include a subset of the TMS metrics For example, values for ``checkout`` service metrics displayed in the host dashboard might be different from the metrics displayed in the service map because there are multiple span ``kind`` values associated with this service that the MMS that power the dashboard don't monitor. -To compare MMS and TMS directly, restrict your TMS to endpoint-only data by filtering to a specific endpoint. You can also break down the service map by endpoint. \ No newline at end of file +To compare MMS and TMS directly, restrict your TMS to endpoint-only data by filtering to a specific endpoint. You can also break down the service map by endpoint. From f1e63952d939cbf0d94d10a48fdb40ba5e23ad89 Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Mon, 31 Mar 2025 15:18:52 -0400 Subject: [PATCH 06/16] Added feedback from PM, put dimensions list in a table --- apm/span-tags/metricsets.rst | 93 ++++++++++++++++++++++++++++-------- 1 file changed, 73 insertions(+), 20 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index ce02125ea..84f135ff8 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -14,35 +14,82 @@ MetricSets are key performance indicators, like request rate, error rate, and re Monitoring MetricSets ===================== -Monitoring MetricSets (MMS) are metric time series (MTS) that power the real-time monitoring capabilities in Splunk APM, including charts and dashboards. MMS power the APM landing page and the dashboard view. MMS are also the metrics that detectors monitor to generate alerts. +Monitoring MetricSets (MMS) are metric time series (MTS) that power the monitoring capabilities in Splunk APM, including charts and dashboards. MMS power the APM landing page and the dashboard view. MMS are also the metrics that detectors monitor to generate alerts. MMS are available for a specific endpoint or for the aggregate of all endpoints in a service. -Endpoint-level MMS reflect the activity of a single endpoint in a service, while service-level MMS aggregate the activity of all of the endpoints in the service. MMS are limited to spans where the ``span.kind`` has a value of ``SERVER`` or ``CONSUMER``. +Endpoint-level MMS reflect the activity of a single endpoint in a service, while service-level MMS aggregate the activity of all of the endpoints in the service. MMS are created for spans where the ``span.kind`` has a value of ``SERVER`` or ``CONSUMER``. Spans might lack a ``kind`` value, or have a different ``kind`` value, in the following situations: * The span originates in self-initiating operations or inferred services * An error in instrumentation occurs. -In addition to the following default MMS, you can create custom MMS. See :ref:`cmms`. - .. _default-mms: Available default MMS metrics and dimensions ----------------------------------------------- -MMS are available for the following APM components: +MMS are available for the APM components listed in the following table. Each MMS also has a set of dimensions you can use to monitor and alert on service performance. In addition to the following default MMS, you can create custom MMS to deep dive on your MMS. See :ref:`cmms`. + +.. _service-mms: +.. _inferred-service-mms-dimensions: -- service.request -- spans -- inferred.services -- traces -- workflows (Workflow metrics are created by default when you create a Business Workflow. Custom MMS are not available for Business Workflows.) -Monitoring MetricSets in APM are generated as histogram metrics. Previously, MMS were classified as either a counter or gauge metric type. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. +.. list-table:: + :widths: 33 33 33 + :width: 100 + :header-rows: 1 -For each metric, there is 1 metric time series (MTS) with responses ``sf_error: true`` or ``sf_error: false``. + * - Metric name + - Dimensions + - Custom dimension available? (Yes/No) + * - ``service.request`` - the requests to endpoints in a service + - * ``sf_environment`` + * ``deployment.environment`` - This dimension is only available for histogram MMS. + * ``sf_service`` + * ``service.name`` - This dimension is only available for histogram MMS. + * ``sf_error`` + - Yes + * - ``inferred.services`` - + - * ``sf_service`` + * ``service.name`` - This dimension is only available for histogram MMS. + * ``sf_environment`` + * ``deployment.environment`` - This dimension is only available for histogram MMS. + * ``sf_error`` + * ``sf.kind`` + * ``sf_operation`` + * ``sf_httpMethod`` + - No + * - ``spans`` - the count of spans (a single operation) + - * ``sf_environment`` + * ``deployment.environment`` - This dimension is only available for histogram MMS. + * ``sf_service`` + * ``service.name`` - This dimension is only available for histogram MMS. + * ``sf_operation`` + * ``sf_kind`` + * ``sf_error`` + * ``sf_httpMethod``, where relevant + - Yes + * - ``traces`` - the count of traces (collection of spans that represents a transaction) + - * ``sf_environment`` + * ``deployment.environment`` - This dimension is only available for histogram MMS. + * ``sf_service`` + * ``service.name`` - This dimension is only available for histogram MMS. + * ``sf_operation`` + * ``sf_httpMethod`` + * ``sf_error`` + - No + * - ``workflows`` - created by default when you create a business workflow + - * ``sf_environment`` + * ``deployment.environment`` - This dimension is only available for histogram MMS. + * ``sf_workflow`` + * ``sf_error`` + - No + +Monitoring MetricSets in APM are generated as histogram metrics. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. For histogram MMS, there is a single metric for each component. + +Previously, MMS were classified as either a counter or gauge metric type. The previous MMS included 6 metrics for each component. .. list-table:: :widths: 33 33 33 @@ -72,7 +119,7 @@ For each metric, there is 1 metric time series (MTS) with responses ``sf_error: - ``.duration.ns.p99`` -Example histogram metrics in APM +Example metrics in APM --------------------------------------------- A histogram MTS uses the following syntax using SignalFlow: @@ -90,22 +137,26 @@ The following table displays example SignalFlow functions: :header-rows: 1 * - Description - - Previous MMS function - - Histogram MMS function + - Histogram MMS + - Previous MMS (deprecated) * - Aggregate count of all MTS - - ``A = data('spans.count').sum().publish(label='A')`` - ``A = histogram('spans').count().publish(label='A')`` + - ``A = data('spans.count').sum().publish(label='A')`` * - P90 percentile for single MTS - - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` - - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'apm-api-peanuts') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = histogram('spans', filter=filter_).percentile(pct=90).publish(label='A')`` + - ``filter_ = filter('sf_environment', 'environment1') and filter('sf_service', 'service 1') and filter('sf_operation', 'operation1') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` + - ``filter_ = filter('sf_environment', 'us1') and filter('sf_service', 'service1') and filter('sf_operation', 'POST /api/autosuggest/tagvalues') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A')`` * - Combined p90 for multiple services - - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts'), rollup='average').mean().publish(label='A')`` - - ``A = histogram('service.request', filter=filter('sf_service', 'apm-graphql', 'apm-api-peanuts')).percentile(pct=90).publish(label='A')`` + - ``A = histogram('service.request', filter=filter('sf_service', 'service 2', 'service 1')).percentile(pct=90).publish(label='A')`` + - ``A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'service 2', 'service 1'), rollup='average').mean().publish(label='A')`` .. note:: Because an aggregation is applied on histogram(), to display all of the metric sets separately, each dimension needs to be applied as a groupby. +Dimensions +===================== Each MMS has a set of dimensions you can use to monitor and alert on service performance. +In addition to the following default MMS, you can create custom MMS to deep dive on your MMS. See :ref:`cmms`. + .. _service-mms: Service dimensions @@ -128,6 +179,8 @@ Inferred service dimensions * ``deployment.environment`` - This dimension is only available for histogram MMS. * ``sf_error`` * ``sf.kind`` +* ``sf_operation`` +* ``sf_httpMethod`` .. _endpoint-mms: From aec9f4815cd20af7f0d41175022d688ee8bbc833 Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Mon, 31 Mar 2025 15:23:01 -0400 Subject: [PATCH 07/16] Added feedback to separate file --- apm/span-tags/metricsets.rst | 73 +----------------------------------- 1 file changed, 1 insertion(+), 72 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index 84f135ff8..7971b55d4 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -34,7 +34,7 @@ MMS are available for the APM components listed in the following table. Each MMS .. _service-mms: .. _inferred-service-mms-dimensions: - +.. _endpoint-mms: .. list-table:: :widths: 33 33 33 @@ -128,7 +128,6 @@ A histogram MTS uses the following syntax using SignalFlow: histogram(metric=[,filter=][,resolution=) - The following table displays example SignalFlow functions: .. list-table:: @@ -151,76 +150,6 @@ The following table displays example SignalFlow functions: .. note:: Because an aggregation is applied on histogram(), to display all of the metric sets separately, each dimension needs to be applied as a groupby. -Dimensions -===================== -Each MMS has a set of dimensions you can use to monitor and alert on service performance. - -In addition to the following default MMS, you can create custom MMS to deep dive on your MMS. See :ref:`cmms`. - -.. _service-mms: - -Service dimensions ---------------------------------- - -* ``sf_environment`` -* ``deployment.environment`` - This dimension is only available for histogram MMS. -* ``sf_service`` -* ``service.name`` - This dimension is only available for histogram MMS. -* ``sf_error`` - -.. _inferred-service-mms-dimensions: - -Inferred service dimensions ------------------------------- - -* ``sf_service`` -* ``service.name`` - This dimension is only available for histogram MMS. -* ``sf_environment`` -* ``deployment.environment`` - This dimension is only available for histogram MMS. -* ``sf_error`` -* ``sf.kind`` -* ``sf_operation`` -* ``sf_httpMethod`` - -.. _endpoint-mms: - -Span dimensions ----------------------------------------------- - -* ``sf_environment`` -* ``deployment.environment`` - This dimension is only available for histogram MMS. -* ``sf_service`` -* ``service.name`` - This dimension is only available for histogram MMS. -* ``sf_operation`` -* ``sf_kind`` -* ``sf_error`` -* ``sf_httpMethod``, where relevant - -Trace dimensions ---------------------------------- - -.. note:: Trace dimensions are not supported for custom MMS. - -* ``sf_environment`` -* ``deployment.environment`` - This dimension is only available for histogram MMS. -* ``sf_service`` -* ``service.name`` - This dimension is only available for histogram MMS. -* ``sf_operation`` -* ``sf_httpMethod`` -* ``sf_error`` - -Workflow dimensions ---------------------------------- - -Workflow metrics and dimensions are created by default when you create a Business Workflow. - -.. note:: Workflow dimensions are not supported for custom MMS. - -* ``sf_environment`` -* ``deployment.environment`` - This dimension is only available for histogram MMS. -* ``sf_workflow`` -* ``sf_error`` - Use MMS within Splunk APM ---------------------------------------- From ac11b883647903bb2d55a8c1c4606ad64d5fc53e Mon Sep 17 00:00:00 2001 From: Johannah Perez Date: Mon, 31 Mar 2025 15:26:18 -0400 Subject: [PATCH 08/16] Updated section location --- apm/span-tags/metricsets.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index 7971b55d4..fe2fec9da 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -25,6 +25,11 @@ Spans might lack a ``kind`` value, or have a different ``kind`` value, in the fo * The span originates in self-initiating operations or inferred services * An error in instrumentation occurs. +MMS retention period +----------------------------------- + +Splunk Observability Cloud stores MMS for 13 months by default. + .. _default-mms: Available default MMS metrics and dimensions @@ -170,11 +175,6 @@ Use MMS for alerting and real-time monitoring in Splunk APM. You can create char * - Monitor services in APM dashboards - :ref:`Track service performance using dashboards in Splunk APM` -MMS retention period ------------------------------------ - -Splunk Observability Cloud stores MMS for 13 months by default. - .. _troubleshooting-metricsets: Troubleshooting MetricSets From 4c0bd4cc819daa1f9b85c952ab72a405e4426865 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 10 Apr 2025 11:09:20 +0200 Subject: [PATCH 09/16] Remove Fluentd --- _includes/requirements/collector-linux.rst | 4 +- admin/authentication/allow-services.rst | 3 - .../linux/linux-backend.rst | 11 -- .../collector-config-tutorial-start.rst | 1 - .../deployments-linux-ansible.rst | 8 -- .../deployments-linux-chef.rst | 12 --- .../deployments-linux-puppet.rst | 12 --- .../deployments-linux-salt.rst | 15 --- .../collector-linux/install-linux-manual.rst | 32 ------ .../collector-linux/install-linux.rst | 29 +---- .../collector-linux/linux-config-logs.rst | 78 +------------- .../collector-linux/linux-uninstall.rst | 102 +++--------------- .../deployments-windows-ansible.rst | 8 -- .../deployments-windows-puppet.rst | 8 -- .../collector-windows/install-windows-msi.rst | 40 +------ .../collector-windows/install-windows.rst | 11 +- .../collector-windows/windows-config-logs.rst | 45 -------- .../collector-windows/windows-uninstall.rst | 2 +- gdi/opentelemetry/install-the-collector.rst | 18 ---- gdi/opentelemetry/opentelemetry.rst | 5 +- .../other-configuration-sources.rst | 2 +- gdi/opentelemetry/sizing.rst | 4 +- .../splunk-collector-troubleshooting.rst | 5 - gdi/opentelemetry/troubleshoot-logs.rst | 25 +---- 24 files changed, 35 insertions(+), 445 deletions(-) diff --git a/_includes/requirements/collector-linux.rst b/_includes/requirements/collector-linux.rst index 034ec1302..636ad4b47 100644 --- a/_includes/requirements/collector-linux.rst +++ b/_includes/requirements/collector-linux.rst @@ -1,10 +1,10 @@ The Collector supports the following Linux distributions and versions: -* Amazon Linux: 2, 2023. Log collection with Fluentd is not currently supported for Amazon Linux 2023. +* Amazon Linux: 2, 2023. * CentOS: 7, 8, 9 * Red Hat: 7, 8, 9 * Oracle: 8, 9 * Debian: 11, 12 -* SUSE: 12, 15 for version 0.34.0 or higher. Log collection with Fluentd is not currently supported. +* SUSE: 12, 15 for version 0.34.0 or higher. * Ubuntu: 16.04, 18.04, 20.04, 22.04, and 24.04 * Rocky Linux: 8, 9 diff --git a/admin/authentication/allow-services.rst b/admin/authentication/allow-services.rst index 75aca7921..8db9187b1 100644 --- a/admin/authentication/allow-services.rst +++ b/admin/authentication/allow-services.rst @@ -216,9 +216,6 @@ If you're unable to allow all URLs as described in :ref:`allow-urls`, ensure tha # RUM ingest endpoint rum-ingest..signalfx.com/v1/rum - # For td-agent/Fluentd on Linux and Windows - packages.treasuredata.com - # For DEB/RPM collector packages splunk.jfrog.io jfrog-prod-use1-shared-virginia-main.s3.amazonaws.com diff --git a/gdi/opentelemetry/automatic-discovery/linux/linux-backend.rst b/gdi/opentelemetry/automatic-discovery/linux/linux-backend.rst index 6afc5f967..811187cf6 100644 --- a/gdi/opentelemetry/automatic-discovery/linux/linux-backend.rst +++ b/gdi/opentelemetry/automatic-discovery/linux/linux-backend.rst @@ -89,8 +89,6 @@ Using the installer script, you can install and activate zero-code instrumentati curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm -- - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. - The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/java.conf``. To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment `` option. Replace ```` with the desired attribute value, for example, ``prod``, as shown in the following example: @@ -125,8 +123,6 @@ Using the installer script, you can install and activate zero-code instrumentati The ``systemd`` instrumentation automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``. - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. - To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment `` option. Replace ```` with the desired attribute value, for example, ``prod``, as shown in the following example: .. code-block:: bash @@ -230,7 +226,6 @@ Using the installer script, you can install and activate zero-code instrumentati curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm -- - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/node.conf``. @@ -257,8 +252,6 @@ Using the installer script, you can install and activate zero-code instrumentati The ``systemd`` zero-code instrumentation method automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``. - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. - You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example: .. code-block:: bash @@ -309,8 +302,6 @@ Using the installer script, you can install and activate zero-code instrumentati curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm -- - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. - The system-wide zero-code instrumentation method automatically adds environment variables to ``/etc/splunk/zeroconfig/dotnet.conf``. To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment `` option. Replace ```` with the desired attribute value, for example, ``prod``, as shown in the following example: @@ -345,8 +336,6 @@ Using the installer script, you can install and activate zero-code instrumentati The ``systemd`` instrumentation automatically adds environment variables to ``/usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf``. - .. note:: If you wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance by specifying the ``--with-fluentd`` option. - To automatically define the optional ``deployment.environment`` resource attribute at installation time, run the installer script with the ``--deployment-environment `` option. Replace ```` with the desired attribute value, for example, ``prod``, as shown in the following example: .. code-block:: bash diff --git a/gdi/opentelemetry/collector-linux/collector-configuration-tutorial/collector-config-tutorial-start.rst b/gdi/opentelemetry/collector-linux/collector-configuration-tutorial/collector-config-tutorial-start.rst index b7cdea50e..388a24f7d 100644 --- a/gdi/opentelemetry/collector-linux/collector-configuration-tutorial/collector-config-tutorial-start.rst +++ b/gdi/opentelemetry/collector-linux/collector-configuration-tutorial/collector-config-tutorial-start.rst @@ -66,7 +66,6 @@ After you've installed the Collector, navigate to /etc/otel/collector to find th . |-- agent_config.yaml |-- config.d - |-- fluentd |-- gateway_config.yaml |-- splunk-otel-collector.conf |-- splunk-otel-collector.conf.example diff --git a/gdi/opentelemetry/collector-linux/deployments-linux-ansible.rst b/gdi/opentelemetry/collector-linux/deployments-linux-ansible.rst index 0e426b809..3088ac44c 100644 --- a/gdi/opentelemetry/collector-linux/deployments-linux-ansible.rst +++ b/gdi/opentelemetry/collector-linux/deployments-linux-ansible.rst @@ -96,14 +96,6 @@ The following table describes the variables that can be configured for this role - The amount of allocated memory in MiB. The default value is ``512``, or 500 x 2^20 bytes, of memory . * - ``splunk_ballast_size_mib`` - ``splunk_ballast_size_mib`` is deprecated starting on Collector version 0.97.0. If you're using it, see :ref:`how to update your configuration `. - * - ``install_fluentd`` - - The option to install or manage Fluentd and dependencies for log collection. The dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries or development tools. The default value is ``false``. - * - ``td_agent_version`` - - The version of td-agent (Fluentd package) that is installed. The default value is ``3.3.0`` for Debian jessie, ``3.7.1`` for Debian stretch, and ``4.3.0`` for other distros. - * - ``splunk_fluentd_config`` - - The path to the Fluentd configuration file on the remote host. The default location is ``/etc/otel/collector/fluentd/fluent.conf``. - * - ``splunk_fluentd_config_source`` - - The source path to a Fluentd configuration file on your control host that is uploaded and set in place of the value set in ``splunk_fluentd_config`` on remote hosts. Use this variable to submit a custom Fluentd configuration, for example, ``./custom_fluentd_config.conf``. The default value is ``""``, which means that nothing is copied and the configuration file set with ``splunk_otel_collector_config`` is used. .. _ansible-zero-config: diff --git a/gdi/opentelemetry/collector-linux/deployments-linux-chef.rst b/gdi/opentelemetry/collector-linux/deployments-linux-chef.rst index 3b2b65ee3..fc7a55844 100644 --- a/gdi/opentelemetry/collector-linux/deployments-linux-chef.rst +++ b/gdi/opentelemetry/collector-linux/deployments-linux-chef.rst @@ -112,18 +112,6 @@ For Linux, the cookbook accepts the attributes described in the following table: * - ``package_stage`` - The Collector package repository stage to use. Can be ``release``, ``beta``, or ``test``. - ``release`` - * - ``with_fluentd`` - - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools. - - ``false`` - * - ``fluentd_version`` - - Version of the td-agent (Fluentd) package to install - - ``3.7.1`` for Debian stretch and ``4.3.1`` for all other Linux distros - * - ``fluentd_config_source`` - - Source path to the Fluentd configuration file. This file is copied to the ``$fluentd_config_dest`` path on the node. See the :new-page:`source attribute ` of the file resource for the supported value types. The default source file is provided by the Collector package. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``/etc/otel/collector/fluentd/fluent.conf`` - * - ``fluentd_config_dest`` - - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``/etc/otel/collector/fluentd/fluent.conf`` .. _chef-zero-config: diff --git a/gdi/opentelemetry/collector-linux/deployments-linux-puppet.rst b/gdi/opentelemetry/collector-linux/deployments-linux-puppet.rst index 5a761e55e..320cc2b2a 100644 --- a/gdi/opentelemetry/collector-linux/deployments-linux-puppet.rst +++ b/gdi/opentelemetry/collector-linux/deployments-linux-puppet.rst @@ -101,18 +101,6 @@ The class accepts the parameters described in the following table: * - ``service_user and $service_group`` - Sets the user or group ownership for the Collector service. The user or group is created if they do not exist. - ``splunk-otel-collector`` - * - ``with_fluentd`` - - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools. - - ``false`` - * - ``fluentd_config_source`` - - Source path to the Fluentd configuration file. This file is copied to the ``$fluentd_config_dest`` path on the node. See the :new-page:`source attribute ` of the file resource for the supported value types. The default source file is provided by the Collector package. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``/etc/otel/collector/fluentd/fluent.conf`` - * - ``fluentd_config_dest`` - - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``/etc/otel/collector/fluentd/fluent.conf`` - * - ``manage_repo`` - - In cases where the Collector and Fluentd apt/yum repositories are managed externally, set this to ``false`` to deactivate management of the repositories by this module. If set to ``false``, the externally managed repositories should provide the ``splunk-otel-collector`` and ``td-agent`` packages. Also, the apt (``/etc/apt/sources.list.d/splunk-otel-collector.list`` and ``/etc/apt/sources.list.d/splunk-td-agent.list``) and yum (``/etc/yum.repos.d/splunk-otel-collector.repo`` and ``/etc/yum.repos.d/splunk-td-agent.repo``) repository definition files are deleted if they exist in order to avoid any conflicts. - - ``true`` .. _puppet-zero-config: diff --git a/gdi/opentelemetry/collector-linux/deployments-linux-salt.rst b/gdi/opentelemetry/collector-linux/deployments-linux-salt.rst index f07bb2b52..45379c0ae 100644 --- a/gdi/opentelemetry/collector-linux/deployments-linux-salt.rst +++ b/gdi/opentelemetry/collector-linux/deployments-linux-salt.rst @@ -110,21 +110,6 @@ For Linux, the formula accepts the attributes described in the following table: * - ``service_user`` and ``$service_group`` - Sets the user or group ownership for the Collector service. The user or group is created if they do not exist. - ``splunk-otel-collector`` - * - ``install_fluentd`` - - Whether to install or manage Fluentd and dependencies for log collection. On Linux, the dependencies include ``capng_c`` for activating Linux capabilities, ``fluent-plugin-systemd`` for systemd journal log collection, and the required libraries and development tools. - - ``false`` - * - ``td_agent_version`` - - Version of the td-agent (Fluentd) package to install - - ``4.3.0`` - * - ``splunk_fluentd_config`` - - The path to the Fluentd configuration file on the remote host. - - ``/etc/otel/collector/fluentd/fluent.conf`` - * - ``splunk_fluentd_config_source`` - - The source path to a Fluentd configuration file on your control host that is uploaded and set in place of the ``splunk_fluentd_config`` file on remote hosts. To use a custom Fluentd configuration file, add the configuration file into the Salt dir. For example, ``salt://templates/td_agent.conf``. - - ``""`` meaning that nothing is copied and the existing ``splunk_fluentd_config`` file is used. - * - ``fluentd_config_dest`` - - Destination path to the Fluentd configuration file on the node. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``/etc/otel/collector/fluentd/fluent.conf`` .. _salt-zero-config: diff --git a/gdi/opentelemetry/collector-linux/install-linux-manual.rst b/gdi/opentelemetry/collector-linux/install-linux-manual.rst index 6ecf3c72c..f7a7f1218 100644 --- a/gdi/opentelemetry/collector-linux/install-linux-manual.rst +++ b/gdi/opentelemetry/collector-linux/install-linux-manual.rst @@ -89,7 +89,6 @@ See also: * :ref:`linux-packages-post` * :ref:`linux-packages-auto` -* :ref:`linux-packages-fluentd` .. _linux-packages-rpm: @@ -164,7 +163,6 @@ See also: * :ref:`linux-packages-post` * :ref:`linux-packages-auto` -* :ref:`linux-packages-fluentd` .. _linux-packages: @@ -213,7 +211,6 @@ See also: * :ref:`linux-packages-post` * :ref:`linux-packages-auto` -* :ref:`linux-packages-fluentd` .. _linux-packages-post: @@ -305,35 +302,6 @@ The ``splunk-otel-auto-instrumentation`` deb/rpm package installs and supports c To learn more, see :ref:`linux-backend-auto-discovery`. -.. _linux-packages-fluentd: - -Install and configure Fluentd for log collection --------------------------------------------------------------- - -If you require log collection, perform the following steps to install Fluentd and forward collected log events to the Collector. This requires root privileges. - -#. Install, configure, and start the Collector as described in :ref:`linux-packages-repo`. The Collector's default configuration file listens for log events on ``127.0.0.1:8006`` and sends them to Splunk Observability Cloud. - -#. Install the ``td-agent`` package appropriate for the Linux distribution/version of the target system. Find the package in :new-page:`Fluentd installation `. - - * If necessary, install the ``capng_c`` plugin and dependencies to enable Linux capabilities, for example ``cap_dac_read_search`` and/or ``cap_dac_override``. This requires ``td-agent`` version 4.1 or higher. See :new-page:`Linux capabilities `. - - * If necessary, install the ``fluent-plugin-systemd`` plugin to collect log events from the systemd journal. See :new-page:`Fluent plugin systemd `. - -#. Configure Fluentd to collect log events and forward them to the Collector: - - * Option 1: Update the default config file at /etc/td-agent/td-agent.conf provided by the Fluentd package to collect the desired log events and forward them to ``127.0.0.1:8006``. - - * Option 2: The installed Collector package provides a custom Fluentd config file /etc/otel/collector/fluentd/fluent.conf to collect log events from many popular services and forwards them to ``127.0.0.1:8006``. To use these files, you need to override the default config file path for the Fluentd service. To do this, copy the systemd environment file from /etc/otel/collector/fluentd/splunk-otel-collector.conf to /etc/systemd/system/td-agent.service.d/splunk-otel-collector.conf. - -#. Ensure that the ``td-agent`` service user/group has permissions to access to the config file(s) from the previous step. - -#. Restart the Fluentd service to apply the changes by running ``systemctl restart td-agent``. - -#. View Fluentd service logs and errors in /var/log/td-agent/td-agent.log. - -See :new-page:`Fluentd configuration ` for general Fluentd configuration details. - .. _linux-docker: Docker diff --git a/gdi/opentelemetry/collector-linux/install-linux.rst b/gdi/opentelemetry/collector-linux/install-linux.rst index 96eeebf03..8cb093fd8 100644 --- a/gdi/opentelemetry/collector-linux/install-linux.rst +++ b/gdi/opentelemetry/collector-linux/install-linux.rst @@ -41,7 +41,6 @@ Included packages The installer script deploys and configures these elements: * The Splunk Distribution of the OpenTelemetry Collector for Linux -* Fluentd, using the td-agent. Turned off by default. See :ref:`fluentd-manual-config-linux` and :ref:`fluentd-receiver` for more information * JMX metric gatherer .. _linux-scripts: @@ -87,15 +86,15 @@ To configure proxy settings to install and run the OpenTelemetry Collector, see Use configured repos -------------------------------- -By default, apt/yum/zypper repo definition files are created to download the package and Fluentd deb/rpm packages from +By default, apt/yum/zypper repo definition files are created to download the package from :new-page:`https://splunk.jfrog.io/splunk ` and :new-page:`https://packages.treasuredata.com `, respectively. -To skip these steps and use configured repos on the target system that provide the ``splunk-otel-collector`` and ``td-agent`` deb/rpm packages, specify the ``--skip-collector-repo`` or ``--skip-fluentd-repo`` options. For example: +To skip these steps and use configured repos on the target system that provide the ``splunk-otel-collector`` packages, use the ``--skip-collector-repo`` option. For example: .. code-block:: bash curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ - sudo sh /tmp/splunk-otel-collector.sh --realm $SPLUNK_REALM --skip-collector-repo --skip-fluentd-repo \ + sudo sh /tmp/splunk-otel-collector.sh --realm $SPLUNK_REALM --skip-collector-repo \ -- $SPLUNK_ACCESS_TOKEN .. _configure-auto-instrumentation-linux: @@ -156,7 +155,7 @@ To use host bindings, run this command: Options of the installer script of the Collector for Linux ================================================================== -The Linux installer script supports the following options for the Collector, automatic discovery with zero-code instrumentation for back-end services, and Fluentd. +The Linux installer script supports the following options for the Collector, automatic discovery with zero-code instrumentation for back-end services. To display all the configuration options supported by the script, use the ``-h`` flag. @@ -282,29 +281,9 @@ Automatic discovery with zero-code instrumentation for back-end services - The ``splunk-otel-auto-instrumentation`` package version to install. Note: The minimum supported version for Java and Node.js zero-code instrumentation is 0.87.0, and the minimum supported version for .NET zero-code instrumentation is 0.99.0. - ``latest`` -Fluentd --------------------------------------------------------------------- - -.. list-table:: - :header-rows: 1 - :width: 100% - :widths: 30 40 30 - - * - Option - - Description - - Default value - * - ``--with[out]-fluentd`` - - Whether to install and configure fluentd to forward log events to the Collector. See :ref:`fluentd-manual-config-linux` for more information. - - ``--without-fluentd`` - * - ``--skip-fluentd-repo`` - - By default, a apt/yum repo definition file will be created to download the fluentd deb/rpm package from ``https://packages.treasuredata.com``. Use this option to skip the previous step and use a pre-configured repo on the target system that provides the ``td-agent`` deb/rpm package. - - - Next steps ================================== - - .. raw:: html
diff --git a/gdi/opentelemetry/collector-linux/linux-config-logs.rst b/gdi/opentelemetry/collector-linux/linux-config-logs.rst index 1c5602962..5ec7592ca 100644 --- a/gdi/opentelemetry/collector-linux/linux-config-logs.rst +++ b/gdi/opentelemetry/collector-linux/linux-config-logs.rst @@ -11,81 +11,5 @@ Collect logs with the Collector for Linux Use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. -Fluentd is turned off by default. If you already installed Fluentd on a host, re-install the Collector without Fluentd using the ``--without-fluentd`` option. - -.. _fluentd-manual-config-linux: - -Collect Linux logs with Fluentd -=========================================================================== - -If you want to collect logs for the target host with Fluentd, use the ``--with-fluentd`` option to also install Fluentd when installing the Collector. For example: - -.. code-block:: bash - - curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \ - sudo sh /tmp/splunk-otel-collector.sh --with-fluentd --realm $SPLUNK_REALM -- $SPLUNK_ACCESS_TOKEN - -When turned on, the Fluentd service is configured by default to collect and forward log events with the ``@SPLUNK`` label to the Collector, which then sends these events to the HEC ingest endpoint determined by the ``--realm `` option. For example, ``https://ingest..signalfx.com/v1/log``. - -The following Fluentd plugins are also installed: - -* ``capng_c`` for activating Linux capabilities. -* ``fluent-plugin-systemd`` for systemd journal log collection. - -Additionally, the following dependencies are installed as prerequisites for the Fluentd plugins: - -.. tabs:: - - .. tab:: Debian-based systems - - * build-essential - * libcap-ng0 - * libcap-ng-dev - * pkg-config - - .. tab:: RPM-based systems - - * Development Tools - * libcap-ng - * libcap-ng-devel - * pkgconfig - -You can specify the following parameters to configure the package to send log events to a custom Splunk HTTP Event Collector (HEC) endpoint URL: - -* ``--hec-url `` -* ``--hec-token `` - -HEC lets you send data and application events to a Splunk deployment over the HTTP and Secure HTTP (HTTPS) protocols. See :new-page:`Set up and use HTTP Event Collector in Splunk Web `. - -The main Fluentd configuration is installed to ``/etc/otel/collector/fluentd/fluent.conf``. Custom Fluentd source configuration files can be added to the ``/etc/otel/collector/fluentd/conf.d`` directory after installation. - -Note the following: - -* In this directory, all files with the .conf extension are automatically included by Fluentd. -* The td-agent user must have permissions to access the configuration files and the paths defined within. -* By default, Fluentd is configured to collect systemd journal log events from ``/var/log/journal``. - -After any configuration modification, run ``sudo systemctl restart td-agent`` to restart the td-agent service. - -If the td-agent package is upgraded after initial installation, you might need to set the Linux capabilities for the new version by performing the following steps for td-agent versions 4.1 or higher: - -#. Check for the activated capabilities: - - .. code-block:: bash - - sudo /opt/td-agent/bin/fluent-cap-ctl --get -f /opt/td-agent/bin/ruby - Capabilities in `` /opt/td-agent/bin/ruby`` , - Effective: dac_override, dac_read_search - Inheritable: dac_override, dac_read_search - Permitted: dac_override, dac_read_search - -#. If the output from the previous command does not include ``dac_override`` and ``dac_read_search`` as shown above, run the following commands: - - .. code-block:: bash - - sudo td-agent-gem install capng_c - sudo /opt/td-agent/bin/fluent-cap-ctl --add "dac_override,dac_read_search" -f /opt/td-agent/bin/ruby - sudo systemctl daemon-reload - sudo systemctl restart td-agent - +Do not use Fluentd to collect logs. If you already installed Fluentd on a host, re-install the Collector without Fluentd using the ``--without-fluentd`` option. diff --git a/gdi/opentelemetry/collector-linux/linux-uninstall.rst b/gdi/opentelemetry/collector-linux/linux-uninstall.rst index 4070268d1..c1e2457ec 100644 --- a/gdi/opentelemetry/collector-linux/linux-uninstall.rst +++ b/gdi/opentelemetry/collector-linux/linux-uninstall.rst @@ -9,9 +9,9 @@ Uninstall the Collector for Linux Follow these instructions to uninstall the Splunk Distribution of the OpenTelemetry Collector for Linux. -You can use commands to uninstall the Collector and Fluentd packages if you used the :ref:`installer script` or :ref:`Debian or RPM package ` to perform the installation. +You can use commands to uninstall the Collector packages if you used the :ref:`installer script` or :ref:`Debian or RPM package ` to perform the installation. -If you installed the Collector and Fluentd using other methods (such as Ansible, Puppet, and Heroku as described in :ref:`otel-install-linux`), follow uninstall instructions specific to the tool you used. +If you installed the Collector using other methods (such as Ansible, Puppet, and Heroku as described in :ref:`otel-install-linux`), follow uninstall instructions specific to the tool you used. .. _otel-linux-uninstall-details: @@ -29,25 +29,6 @@ While not an exhaustive list, here are key notes about some of the files that ar * On Debian-based systems, the following files are deleted. If you want to keep these files, be sure to back up the individual files or the entire ``/etc/otel/collector`` directory before you perform the uninstall. Files not in this list aren't deleted. * ``/etc/otel/collector/agent_config.yaml`` - * ``/etc/otel/collector/fluentd/README`` - * ``/etc/otel/collector/fluentd/conf.d/apache.conf`` - * ``/etc/otel/collector/fluentd/conf.d/cassandra.conf`` - * ``/etc/otel/collector/fluentd/conf.d/docker.conf`` - * ``/etc/otel/collector/fluentd/conf.d/etcd.conf`` - * ``/etc/otel/collector/fluentd/conf.d/jetty.conf`` - * ``/etc/otel/collector/fluentd/conf.d/journald.conf`` - * ``/etc/otel/collector/fluentd/conf.d/memcached.conf`` - * ``/etc/otel/collector/fluentd/conf.d/mongodb.conf`` - * ``/etc/otel/collector/fluentd/conf.d/mysql.conf`` - * ``/etc/otel/collector/fluentd/conf.d/nginx.conf`` - * ``/etc/otel/collector/fluentd/conf.d/postgresql.conf`` - * ``/etc/otel/collector/fluentd/conf.d/rabbitmq.conf`` - * ``/etc/otel/collector/fluentd/conf.d/redis.conf`` - * ``/etc/otel/collector/fluentd/conf.d/syslog.conf`` - * ``/etc/otel/collector/fluentd/conf.d/tomcat.conf`` - * ``/etc/otel/collector/fluentd/conf.d/zookeeper.conf`` - * ``/etc/otel/collector/fluentd/fluent.conf`` - * ``/etc/otel/collector/fluentd/splunk-otel-collector.conf`` * ``/etc/otel/collector/gateway_config.yaml`` * ``/etc/otel/collector/splunk-otel-collector.conf.example`` * ``/etc/otel/collector/splunk-support-bundle.sh`` @@ -55,25 +36,6 @@ While not an exhaustive list, here are key notes about some of the files that ar * On RPM-based systems, if you modified any of the following files, the modified files aren't deleted and are renamed with the .rpmsave extension. For example, the uninstall process renames a modified agent_config.yaml to agent_config.yaml.rpmsave. You can delete these .rpmsave files if you don't need them. Unmodified files in this list are deleted. Files not in this list aren't deleted. * ``/etc/otel/collector/agent_config.yaml`` - * ``/etc/otel/collector/fluentd/README`` - * ``/etc/otel/collector/fluentd/conf.d/apache.conf`` - * ``/etc/otel/collector/fluentd/conf.d/cassandra.conf`` - * ``/etc/otel/collector/fluentd/conf.d/docker.conf`` - * ``/etc/otel/collector/fluentd/conf.d/etcd.conf`` - * ``/etc/otel/collector/fluentd/conf.d/jetty.conf`` - * ``/etc/otel/collector/fluentd/conf.d/journald.conf`` - * ``/etc/otel/collector/fluentd/conf.d/memcached.conf`` - * ``/etc/otel/collector/fluentd/conf.d/mongodb.conf`` - * ``/etc/otel/collector/fluentd/conf.d/mysql.conf`` - * ``/etc/otel/collector/fluentd/conf.d/nginx.conf`` - * ``/etc/otel/collector/fluentd/conf.d/postgresql.conf`` - * ``/etc/otel/collector/fluentd/conf.d/rabbitmq.conf`` - * ``/etc/otel/collector/fluentd/conf.d/redis.conf`` - * ``/etc/otel/collector/fluentd/conf.d/syslog.conf`` - * ``/etc/otel/collector/fluentd/conf.d/tomcat.conf`` - * ``/etc/otel/collector/fluentd/conf.d/zookeeper.conf`` - * ``/etc/otel/collector/fluentd/fluent.conf`` - * ``/etc/otel/collector/fluentd/splunk-otel-collector.conf`` * ``/etc/otel/collector/gateway_config.yaml`` * ``/etc/otel/collector/splunk-otel-collector.conf.example`` * ``/etc/otel/collector/splunk-support-bundle.sh`` @@ -83,12 +45,12 @@ While not an exhaustive list, here are key notes about some of the files that ar .. _otel-linux-uninstall-otel-and-tdagent: .. _otel-linux-uninstall-both-otel-and-tdagent: -Uninstall the Collector and Fluentd on Linux +Uninstall the Collector on Linux ================================================================ .. note:: Before you perform the uninstall, be sure to understand its impact. See :ref:`otel-linux-uninstall-details`. -If you installed the Collector and Fluentd using the :ref:`installer script` or :ref:`Debian or RPM package `, you can uninstall both of these packages by running the following command: +If you installed the Collector using the :ref:`installer script` or :ref:`Debian or RPM package `, you can uninstall both of these packages by running the following command: .. code-block:: bash @@ -103,33 +65,27 @@ Note that this snippet includes a command that downloads the latest ``splunk-ote To verify the uninstall, see :ref:`otel-linux-verify-uninstall`. -If you don't want to uninstall :strong:`both` packages and just want to uninstall the Collector package :strong:`or` Fluentd package, see :ref:`otel-linux-uninstall-only-otel-or-tdagent`. +If you don't want to uninstall :strong:`both` packages and just want to uninstall the Collector package see :ref:`otel-linux-uninstall-only-otel-or-tdagent`. .. _otel-linux-uninstall-only-otel-or-tdagent: -Uninstall only the Collector or Fluentd on Linux +Uninstall only the Collector on Linux ================================================================ -The uninstall command described in :ref:`otel-linux-uninstall-otel-and-tdagent` uninstalls :strong:`both` the Collector and Fluentd packages. +The uninstall command described in :ref:`otel-linux-uninstall-otel-and-tdagent` uninstalls :strong:`both` the Collector packages. -If you want to uninstall only the Collector package :strong:`or` the Fluentd package, use the following command for your platform. +If you want to uninstall only the Collector package, use the following command for your platform. For Debian -------------------------------------------------------------------------------------------- .. note:: Before performing an uninstall, see :ref:`otel-linux-uninstall-details`. -* To uninstall the Collector package only, run the following command: +To uninstall the Collector package only, run the following command: - .. code-block:: bash - - sudo apt-get purge splunk-otel-collector - -* To uninstall the Fluentd package only, run the following command: - - .. code-block:: bash +.. code-block:: bash - sudo apt-get purge td-agent + sudo apt-get purge splunk-otel-collector For RPM -------------------------------------------------------------------------------------------- @@ -154,48 +110,24 @@ For RPM sudo zypper remove splunk-otel-collector -* To uninstall the Fluentd package only, run the command for the package manager on your system: - - .. code-block:: bash - - sudo yum remove td-agent - - or - - .. code-block:: bash - - sudo dnf remove td-agent - - or - - .. code-block:: bash - - sudo zypper remove td-agent - To verify the uninstall, see :ref:`otel-linux-verify-uninstall`. .. _otel-linux-verify-uninstall: -Verify the uninstall of the Collector and Fluentd on Linux +Verify the uninstall of the Collector on Linux ================================================================ -While you can verify the uninstall of the Collector and Fluentd packages by watching for success messages in your command-line interface after running an uninstall command, you can also verify the uninstall by running a command that checks on the status of the Collector and Fluentd services. If the package has been successfully uninstalled, the status reflects this. - -* To verify the uninstall of the Collector package, run this command: - - .. code-block:: bash +While you can verify the uninstall of the Collector packages by watching for success messages in your command-line interface after running an uninstall command, you can also verify the uninstall by running a command that checks on the status of the Collector services. If the package has been successfully uninstalled, the status reflects this. - sudo systemctl status splunk-otel-collector +To verify the uninstall of the Collector package, run this command: +.. code-block:: bash - The expected result is ``Unit splunk-otel-collector.service could not be found.`` + sudo systemctl status splunk-otel-collector -* To verify the uninstall of the Fluentd (td-agent) package, run this command: +The expected result is ``Unit splunk-otel-collector.service could not be found.`` - .. code-block:: bash - sudo systemctl status td-agent - The expected result is ``Unit td-agent.service could not be found.`` diff --git a/gdi/opentelemetry/collector-windows/deployments-windows-ansible.rst b/gdi/opentelemetry/collector-windows/deployments-windows-ansible.rst index 8d93472cc..1198077fd 100644 --- a/gdi/opentelemetry/collector-windows/deployments-windows-ansible.rst +++ b/gdi/opentelemetry/collector-windows/deployments-windows-ansible.rst @@ -109,14 +109,6 @@ The following table describes the variables that can be configured for this role - The amount of allocated memory in MiB. The default value is ``512``, or 500 x 2^20 bytes, of memory . * - ``splunk_ballast_size_mib`` - ``splunk_ballast_size_mib`` is deprecated starting on Collector version 0.97.0. If you're using it, see :ref:`how to update your configuration `. - * - ``install_fluentd`` - - The option to install or manage Fluentd and dependencies for log collection. The default value is ``false``. - * - ``td_agent_version`` - - The version of td-agent (Fluentd package) that is installed. - * - ``splunk_fluentd_config`` - - The path to the Fluentd configuration file on the remote host. The default is ``%SYSTEMDRIVE%\opt\td-agent\etc\td-agent\td-agent.conf``. - * - ``splunk_fluentd_config_source`` - - The source path to a Fluentd configuration file on your control host that is uploaded and set in place of the value set in ``splunk_fluentd_config`` on remote hosts. Use this variable to submit a custom Fluentd configuration, for example, ``./custom_fluentd_config.conf``. The default value is ``""``, which means that nothing is copied and the configuration file set with ``splunk_otel_collector_config`` is used. Next steps ================================== diff --git a/gdi/opentelemetry/collector-windows/deployments-windows-puppet.rst b/gdi/opentelemetry/collector-windows/deployments-windows-puppet.rst index 1c745fe2c..dd5662071 100644 --- a/gdi/opentelemetry/collector-windows/deployments-windows-puppet.rst +++ b/gdi/opentelemetry/collector-windows/deployments-windows-puppet.rst @@ -85,18 +85,10 @@ The class accepts the parameters described in the following table: * - ``collector_config_dest`` - Destination path of the Collector configuration file on the node. The ``SPLUNK_CONFIG`` environment variable is set with this value for the Collector service. - ``%PROGRAMDATA%\Splunk\OpenTelemetry Collector\agent_config.yaml`` - * - ``with_fluentd`` - - Whether to install or manage Fluentd and dependencies for log collection. - - ``false`` - * - ``fluentd_config_source`` - - Source path to the Fluentd configuration file. This file is copied to the ``$fluentd_config_dest`` path on the node. See the :new-page:`source attribute ` of the file resource for the supported value types. The default source file is provided by the Collector package. Only applicable if ``$with_fluentd`` is set to ``true``. - - ``%PROGRAMFILES\Splunk\OpenTelemetry Collector\fluentd\td-agent.conf`` Next steps ================================== - - .. raw:: html
diff --git a/gdi/opentelemetry/collector-windows/install-windows-msi.rst b/gdi/opentelemetry/collector-windows/install-windows-msi.rst index 720e125d7..1e06df7d7 100644 --- a/gdi/opentelemetry/collector-windows/install-windows-msi.rst +++ b/gdi/opentelemetry/collector-windows/install-windows-msi.rst @@ -171,56 +171,24 @@ Learn more about advanced configuration options (including Service Logging) usin * :ref:`otel-install-windows-manual` * :ref:`otel-windows-config` -.. _windows-manual-fluentd: - -Install Fluentd MSI for log collection -================================================== - -If you have a wish to collect logs for the target host, make sure Fluentd is installed and enabled in your Collector instance. - -.. note:: You need to be an Admin to configure log collection with Fluentd. - -Perform the following steps to install Fluentd and forward ``collected`` log events to the Collector: - -1. Install :new-page:`Fluentd MSI ` version 4.0 or higher. - -2. Configure Fluentd to collect log events and forward them to the Collector: - - - Option 1: Update the default config file provided by the Fluentd MSI at ``\opt\td-agent\etc\td-agent\td-agent.conf`` to collect the desired log events and forward them to ``127.0.0.1:8006``. - - - Option 2: The installed Collector package provides a custom Fluentd config file ``\Program Files\Splunk\OpenTelemetry Collector\fluentd\td-agent.conf`` to collect log events from the Windows Event Log ``\Program Files\Splunk\OpenTelemetry Collector\fluentd\conf.d\eventlog.conf`` and forwards them to ``127.0.0.1:8006``. - - To use these files, backup the ``\opt\td-agent\etc\td-agent``` directory, and copy the contents from ``\Program Files\Splunk\OpenTelemetry Collector\fluentd``` to ``\opt\td-agent\etc\td-agent```. - -3. To apply any changes made to the Fluentd config files, restart the system, or restart ``fluentdwinsvc`` . - - .. code-block:: PowerShell - - - Stop-Service fluentdwinsvc - - Start-Service fluentdwinsvc - -4. View the Fluentd service logs and errors in ``\opt\td-agent\td-agent.log``. - -Learn more about general Fluentd configuration details in the :new-page:`official Fluentd documentation `. Custom MSI URLs ================================================== -By default, the Collector MSI is downloaded from :new-page:`https://dl.signalfx.com ` and -the Fluentd MSI is downloaded from :new-page:`https://packages.treasuredata.com `. +By default, the Collector MSI is downloaded from :new-page:`https://dl.signalfx.com `. -To specify custom URLs for these downloads, replace ``COLLECTOR_MSI_URL`` and ``FLUENTD_MSI_URL`` with the URLs to the desired MSI packages to install: +To specify custom URLs for these downloads, replace ``COLLECTOR_MSI_URL``with the URLs to the desired MSI packages to install: .. code-block:: PowerShell - & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; collector_msi_url = ""; fluentd_msi_url = ""}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} + & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; collector_msi_url = ""; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} .. _windows-chocolatey: Install the Collector using a Chocolatey package ====================================================== -A :new-page:`Chocolatey package ` is available to download, install, and configure the Collector and Fluentd with the following PowerShell command: +A :new-page:`Chocolatey package ` is available to download, install, and configure the Collector with the following PowerShell command: .. code-block:: PowerShell diff --git a/gdi/opentelemetry/collector-windows/install-windows.rst b/gdi/opentelemetry/collector-windows/install-windows.rst index fd76c30da..dd9f5925f 100644 --- a/gdi/opentelemetry/collector-windows/install-windows.rst +++ b/gdi/opentelemetry/collector-windows/install-windows.rst @@ -55,7 +55,6 @@ Included packages The Windows installer script installs the following packages: * Dotnet autoinstrumentation, if enabled. See :ref:`get-started-dotnet-otel`. -* Fluentd, if enabled. See :ref:`fluentd-manual-config-windows`. * JMX metric gatherer. * For Docker environments only, Java JDK and JRE. @@ -64,10 +63,7 @@ The Windows installer script installs the following packages: Install the Collector for Windows using the installer script ================================================================ -The installer script is available for Windows 64-bit environments, and deploys and configures: - -* The Splunk Distribution of the OpenTelemetry Collector for Windows -* Fluentd through the ``td-agent``, which is deactivated by default +The installer script is available for Windows 64-bit environments, and deploys and configures the Splunk Distribution of the OpenTelemetry Collector for Windows. To install the package using the installer script, follow these steps: @@ -146,8 +142,6 @@ The Windows installer script supports the following options: * - ``hec_token`` - Set the HEC token if it's different than the specified Splunk access token. - - * - ``with_fluentd`` - - Whether to install and configure fluentd to forward log events to the collector. See :ref:`fluentd-manual-config-windows` for more information. - ``$false`` * - ``with_dotnet_instrumentation`` - Whether to install and configure .NET tracing to forward .NET application traces to the local collector. @@ -170,9 +164,6 @@ The Windows installer script supports the following options: * - ``collector_msi_url`` - When installing the Collector, instead of downloading the package, use this local path to a Splunk OpenTelemetry Collector MSI package. If specified, the ``-collector_version`` and ``-stage`` parameters are ignored. - ``https://dl.signalfx.com/splunk-otel-collector/`` |br| ``msi/release/splunk-otel-collector--amd64.msi`` - * - ``fluentd_msi_url`` - - Specify the URL to the Fluentd MSI package to install. - - ``https://packages.treasuredata.com/4/windows/td-agent-4.1.0-x64.msi`` * - ``msi_path`` - Specify a local path to a Splunk OpenTelemetry Collector MSI package to install instead of downloading the package. If specified, the ``-collector_version`` and ``-stage`` parameters will be ignored. - diff --git a/gdi/opentelemetry/collector-windows/windows-config-logs.rst b/gdi/opentelemetry/collector-windows/windows-config-logs.rst index 576aad892..eb3c9af57 100644 --- a/gdi/opentelemetry/collector-windows/windows-config-logs.rst +++ b/gdi/opentelemetry/collector-windows/windows-config-logs.rst @@ -10,50 +10,5 @@ Collect logs with the Collector for Windows Use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. -.. _fluentd-manual-config-windows: - -Collect Windows logs with Fluentd -=========================================================================== - -Fluentd is turned off by default. - -If you wish to collect logs for the target host with Fluentd, use the ``with_fluentd = 1`` option to install and enable Fluentd when installing the Collector. - -For example: - -.. code-block:: PowerShell - - & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; with_fluentd = 1}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} - -When activated, the Fluentd service is configured by default to collect and forward log events with the ``@SPLUNK`` label to the Collector, which then send these events to the HEC ingest endpoint determined by the ``realm = ""`` option. -For example, ``https://ingest..signalfx.com/v1/log``. - -To configure the package to send log events to a custom HTTP Event Collector (HEC) endpoint URL with a token different than ````, you can specify the following parameters for the installer script: - -* ``hec_url = ""`` -* ``hec_token = ""`` - -For example (replace the ```` values in the command for your configuration): - -.. code-block:: PowerShell - - & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; hec_url = ""; hec_token = ""}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} - -The installation creates the main Fluentd configuration file ``\opt\td-agent\etc\td-agent\td-agent.conf``, where ```` is the drive letter for the fluentd installation directory. - -You can add custom Fluentd source configuration files to the ``\opt\td-agent\etc\td-agent\conf.d`` -directory after installation. - -Note the following: - -* In this directory, Fluentd includes all files with the .conf extension. -* By default, fluentd collects from the Windows Event Log. See ``\opt\td-agent\etc\td-agent\conf.d\eventlog.conf`` for the default configuration. - -After any configuration modification, apply the changes by restarting the system or running the following PowerShell commands: - -.. code-block:: PowerShell - - Stop-Service fluentdwinsvc - Start-Service fluentdwinsvc diff --git a/gdi/opentelemetry/collector-windows/windows-uninstall.rst b/gdi/opentelemetry/collector-windows/windows-uninstall.rst index 52b1e52d9..649434a42 100644 --- a/gdi/opentelemetry/collector-windows/windows-uninstall.rst +++ b/gdi/opentelemetry/collector-windows/windows-uninstall.rst @@ -14,7 +14,7 @@ Follow these instructions to uninstall the Splunk Distribution of the OpenTeleme Uninstall using the Windows Control Panel ==================================================== -If you installed the Collector with the installer script, the Collector and td-agent (Fluentd) can be uninstalled from **Programs and Features** in the Windows Control Panel. The configuration files might persist in ``\ProgramData\Splunk\OpenTelemetry Collector`` and ``\opt\td-agent`` after uninstall. +If you installed the Collector with the installer script, the Collector can be uninstalled from **Programs and Features** in the Windows Control Panel. The configuration files might persist in ``\ProgramData\Splunk\OpenTelemetry Collector`` and ``\opt\td-agent`` after uninstall. .. _otel-windows-uninstall-powershell: diff --git a/gdi/opentelemetry/install-the-collector.rst b/gdi/opentelemetry/install-the-collector.rst index a0bd32b5f..727335bbe 100644 --- a/gdi/opentelemetry/install-the-collector.rst +++ b/gdi/opentelemetry/install-the-collector.rst @@ -185,24 +185,6 @@ To collect logs with the Splunk Distribution of the OpenTelemetry Collector: * In Kubernetes environments, native OpenTelemetry log collection is supported by default. See more at :ref:`kubernetes-config-logs`. * For Linux and Windows environments (physical hosts and virtual machines), use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. -.. note:: If you wish to collect logs for the target host, install and enable Fluentd in your Collector instance. - -.. raw:: html - - -

Collect logs using Fluentd ¶

- - -The Collector can capture logs using Fluentd, but this option is deactivated by default. To learn more, see :ref:`fluentd-receiver`. - -To activate Fluentd refer to: - -* :ref:`Configure Fluentd for log collection in Kubernetes ` -* :ref:`Configure Fluentd for log collection in Linux ` -* :ref:`Configure Fluentd for log collection in Windows ` - -Common sources such as filelog, journald, and Windows Event Viewer are included in the installation. - .. raw:: html diff --git a/gdi/opentelemetry/opentelemetry.rst b/gdi/opentelemetry/opentelemetry.rst index aa56b70ee..3723c344b 100644 --- a/gdi/opentelemetry/opentelemetry.rst +++ b/gdi/opentelemetry/opentelemetry.rst @@ -78,8 +78,7 @@ Also, the customizations in the Splunk distribution include these additional fea * Better defaults for Splunk products * Discovery mode for metric sources * Automatic discovery and configuration -* Fluentd for log capture, deactivated by default - + .. note:: Check out the :new-page:`Splunk Distribution of the OpenTelemetry Collector repo in GitHub ` for more details. .. raw:: html @@ -168,8 +167,6 @@ To collect logs with the Splunk Distribution of the OpenTelemetry Collector: * In Kubernetes environments, native OpenTelemetry log collection is supported by default. See more at :ref:`kubernetes-config-logs`. * For Linux and Windows environments (physical hosts and virtual machines), use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. -.. note:: If you wish to collect logs for the target host, install and enable Fluentd in your Collector instance. - .. _otel-intro-enterprise: .. raw:: html diff --git a/gdi/opentelemetry/other-configuration-sources.rst b/gdi/opentelemetry/other-configuration-sources.rst index d867815f8..bf0a87912 100644 --- a/gdi/opentelemetry/other-configuration-sources.rst +++ b/gdi/opentelemetry/other-configuration-sources.rst @@ -7,7 +7,7 @@ Other configuration sources (Alpha/Beta) .. meta:: :description: Configure these optional components to retrieve data from specific configuration sources. After retrieving the data, you can then insert the data into your Splunk Distribution of OpenTelemetry Collector configuration. -In addition to the Collector packages and Fluentd, the following components can be configured: +In addition to the Collector packages you can configure the following components: * :ref:`Environment variable (Alpha) ` * :ref:`etcd2 (Alpha) ` diff --git a/gdi/opentelemetry/sizing.rst b/gdi/opentelemetry/sizing.rst index f7a52b73d..c1659a51d 100644 --- a/gdi/opentelemetry/sizing.rst +++ b/gdi/opentelemetry/sizing.rst @@ -13,7 +13,7 @@ With a single CPU core, the Collector can ingest the following: * If handling traces, 15,000 spans per second. * If handling metrics, 20,000 data points per second. -* If handling logs, 10,000 log records per second, including Fluentd ``td-agent``, which forwards logs to the ``fluentforward`` receiver in the Collector. See more at :ref:`fluentd-receiver`. +* If handling logs, 10,000 log records per second. Sizing recommendations ========================================== @@ -59,7 +59,7 @@ Scaling recommendations To define and scale your architecture, analyze the behavior of your workload to understand the loads and format of each signal type, as well as the load's distribution in time. -For example, consider a scenario with hundreds of Prometheus endpoints to scrape, a terabyte of logs coming from fluentd instances every minute, and some application metrics and OTLP traces. +For example, consider a scenario with hundreds of Prometheus endpoints to scrape, a terabyte of logs ingested every minute, and some application metrics and OTLP traces. In this scenario: diff --git a/gdi/opentelemetry/splunk-collector-troubleshooting.rst b/gdi/opentelemetry/splunk-collector-troubleshooting.rst index 198a9fa85..5f65a9ebb 100644 --- a/gdi/opentelemetry/splunk-collector-troubleshooting.rst +++ b/gdi/opentelemetry/splunk-collector-troubleshooting.rst @@ -196,11 +196,6 @@ No response means the request was sent successfully. You can also pass ``-v`` to Error codes and messages ================================================================================== -You're getting a "pattern not matched" error message ------------------------------------------------------------- - -If you see an error message such as "pattern not matched", this message is from Fluentd, and means that the ```` was unable to match based on the log message. As a result, the log message is not collected. Check the Fluentd configuration and update as required. - You're receiving an HTTP error code ------------------------------------------------------------ diff --git a/gdi/opentelemetry/troubleshoot-logs.rst b/gdi/opentelemetry/troubleshoot-logs.rst index 7aebde865..e5e31b3c1 100644 --- a/gdi/opentelemetry/troubleshoot-logs.rst +++ b/gdi/opentelemetry/troubleshoot-logs.rst @@ -47,30 +47,9 @@ If using Windows, run the following command to check if the source is generating Get-Content myTestLog.log -.. _fluentd-collector-troubleshooting: - -Fluentd isn't configured correctly -========================================= - -Do the following to check the Fluentd configuration: - -#. Check that td-agent is running. On Linux, run ``systemctl status td-agent``. On Windows, run ``Get-Service td-agent``. -#. If you changed the configuration, restart Fluentd. On Linux, run ``systemctl restart td-agent``. On Windows, run ``Restart-Service -Name td-agent``. -#. Check fluentd.conf and conf.d/\*. ``@label @SPLUNK`` must be added to every source to activate log collection. -#. Manual configuration might be required to collect logs off the source. Add configuration files to in the conf.d directory as needed. -#. Activate debug logging in fluentd.conf (``log_level debug``), restart td-agent, and check that the source is generating logs. - -While every attempt is made to properly configure permissions, it is possible that td-agent does not have the permission required to collect logs. Debug logging should indicate this issue. - -It's possible that the ```` section configuration does not match the log events. - -If you see a message such as "2021-03-17 02:14:44 +0000 [debug]: #0 connect new socket", Fluentd is working as expected. You need to activate debug logging to see this message. - The Collector isn't configured properly ========================================= -.. note:: Fluentd is part of the Splunk Distribution of OpenTelemetry Collector, but deactivated by default for Linux and Windows. To activate it, use the ``--with-fluentd`` option when installing the Collector for Linux, or the ``with_fluentd = 1`` option when installing the Collector for Windows. - Do the following to check the Collector configuration: #. Go to ``http://localhost:55679/debug/tracez`` to check zPages for samples. You might need to configure the endpoint. @@ -81,15 +60,13 @@ Do the following to check the Collector configuration: Test the Collector by sending synthetic data ================================================================================== -You can manually generate logs. By default, Fluentd monitors journald and /var/log/syslog.log for events. +You can manually generate logs. .. code-block:: bash echo "2021-03-17 02:14:44 +0000 [debug]: test" >>/var/log/syslog.log echo "2021-03-17 02:14:44 +0000 [debug]: test" | systemd-cat -.. caution:: Fluentd requires properly structured syslog to pick up the log line. - .. _unwanted_profiling_logs: Unwanted profiling logs appearing in Splunk Observability Cloud From 97b731838180974ebb63cb23aececc56079d8fb4 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 10 Apr 2025 11:27:48 +0200 Subject: [PATCH 10/16] Fixes --- .../collector-windows/install-windows-msi.rst | 2 +- .../components/fluentd-receiver.rst | 29 +++++++------------ 2 files changed, 11 insertions(+), 20 deletions(-) diff --git a/gdi/opentelemetry/collector-windows/install-windows-msi.rst b/gdi/opentelemetry/collector-windows/install-windows-msi.rst index 1e06df7d7..1c29a2185 100644 --- a/gdi/opentelemetry/collector-windows/install-windows-msi.rst +++ b/gdi/opentelemetry/collector-windows/install-windows-msi.rst @@ -177,7 +177,7 @@ Custom MSI URLs By default, the Collector MSI is downloaded from :new-page:`https://dl.signalfx.com `. -To specify custom URLs for these downloads, replace ``COLLECTOR_MSI_URL``with the URLs to the desired MSI packages to install: +To specify custom URLs for these downloads, replace ``COLLECTOR_MSI_URL`` with the URL to the desired MSI packages to install: .. code-block:: PowerShell diff --git a/gdi/opentelemetry/components/fluentd-receiver.rst b/gdi/opentelemetry/components/fluentd-receiver.rst index 578e8100d..5fc356442 100644 --- a/gdi/opentelemetry/components/fluentd-receiver.rst +++ b/gdi/opentelemetry/components/fluentd-receiver.rst @@ -7,21 +7,15 @@ Fluent Forward receiver .. meta:: :description: The Fluent Forward receiver allows the Splunk Distribution of OpenTelemetry Collector to collect logs and events using the Fluent Forward protocol. +.. caution:: ``fluentd``` will be deprecated in October 2025. In Kubernetes environments use native OpenTelemetry log collection instead. In Linux and Windows platforms use the Universal Forwarder. See :ref:otel-config-logs`. + The Fluent Forward receiver allows the Splunk Distribution of the OpenTelemetry Collector to collect events using the bundled Fluentd application. The supported pipeline type is ``logs``. See :ref:`otel-data-processing` for more information. The receiver accepts data formatted as Fluent Forward events through a TCP connection. All three Fluent event types, message, forward, and packed forward, are supported, including compressed packed forward. -.. caution:: Fluentd is deactivated by default for Linux and Windows. To activate it, use the ``--with-fluentd`` option when installing the Collector for Linux, or the ``with_fluentd = 1`` option when installing the Collector for Windows. - Get started ====================== -.. note:: - - This component is included in the default configuration of the Splunk Distribution of the OpenTelemetry Collector when deploying in host monitoring (agent) mode. See :ref:`otel-deployment-mode` for more information. - - For details about the default configuration, see :ref:`otel-kubernetes-config`, :ref:`linux-config-ootb`, or :ref:`windows-config-ootb`. You can customize your configuration any time as explained in this document. - Follow these steps to configure and activate the component: 1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform: @@ -33,7 +27,7 @@ Follow these steps to configure and activate the component: 2. Configure the receiver as described in the next document. 3. Restart the Collector. -By default, the Splunk Distribution of the OpenTelemetry Collector includes the Fluent Forward receiver in the ``logs`` pipeline: +Next, add the Fluent Forward receiver in the ``logs`` pipeline: .. code-block:: yaml @@ -46,12 +40,6 @@ By default, the Splunk Distribution of the OpenTelemetry Collector includes the logs: receivers: [fluentforward] -For more information on how to install Fluentd when manually installing the Collector, see: - -* :ref:`fluentd-manual-config-linux` -* :ref:`fluentd-manual-config-windows` -* :ref:`windows-manual-fluentd` - Settings ====================== @@ -64,9 +52,12 @@ The following table shows the configuration options for the Fluent Forward recei Troubleshooting ====================== -For troubleshooting Fluentd, see: +.. raw:: html + +
-* :ref:`fluentd-collector-troubleshooting` -* :ref:`otel-linux-uninstall-both-otel-and-tdagent` +.. include:: /_includes/troubleshooting-components.rst + +.. raw:: html -.. caution:: If you wish to collect logs for the target host with Fluentd, make sure Fluentd is installed and turned on in your Collector instance. +
\ No newline at end of file From c782c47ad9e39d66173eebc7d9746d4d02c72194 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 10 Apr 2025 11:36:51 +0200 Subject: [PATCH 11/16] Fix? --- .../collector-windows/install-windows.rst | 132 +++++++++--------- gdi/opentelemetry/opentelemetry.rst | 2 +- 2 files changed, 66 insertions(+), 68 deletions(-) diff --git a/gdi/opentelemetry/collector-windows/install-windows.rst b/gdi/opentelemetry/collector-windows/install-windows.rst index dd9f5925f..261478359 100644 --- a/gdi/opentelemetry/collector-windows/install-windows.rst +++ b/gdi/opentelemetry/collector-windows/install-windows.rst @@ -105,77 +105,75 @@ Options of the installer script for Windows The Windows installer script supports the following options: .. list-table:: - :header-rows: 1 - :width: 100% - :widths: 30 40 30 - - * - Option - - Description - - Default value - * - ``access_token`` - - The token used to send metric data to Splunk. - - - * - ``realm`` - - The Splunk realm to use. The ingest, API, trace, and HEC endpoint URLs are automatically created using this value. To find your Splunk realm, see :ref:`Note about realms `. - - ``us0`` - * - ``memory`` - - Total memory in MIB to allocate to the Collector. Automatically calculates the ballast size. See :ref:`otel-sizing` for more information. - - ``512`` - * - ``mode`` - - Configure the Collectorservice to run in host monitoring (``agent``) or data forwarding (``gateway``). - - ``agent`` - * - ``network_interface`` - - The network interface the Collectorreceivers listen on. - - ``0.0.0.0`` - * - ``ingest_url`` - - Set the base ingest URL explicitly instead of the URL inferred from the specified realm. - - ``https://ingest.REALM.signalfx.com`` - * - ``api_url`` - - Set the base API URL explicitly instead of the URL inferred from the specified realm. - - ``https://api.REALM.signalfx.com`` - * - ``trace_url`` - - Set the trace endpoint URL explicitly instead of the endpoint inferred from the specified realm. - - ``https://ingest.REALM.signalfx.com/v2/trace`` - * - ``hec_url`` - - Set the HEC endpoint URL explicitly instead of the endpoint inferred from the specified realm. - - ``https://ingest.REALM.signalfx.com/v1/log`` - * - ``hec_token`` - - Set the HEC token if it's different than the specified Splunk access token. - - - - ``$false`` - * - ``with_dotnet_instrumentation`` - - Whether to install and configure .NET tracing to forward .NET application traces to the local collector. - - ``$false`` - * - ``deployment_env`` - - A system-wide environment tag used by .NET instrumentation. Sets the ``SIGNALFX_ENV`` environment variable. Ignored if ``-with_dotnet_instrumentation`` is set to ``false``. - - - * - ``bundle_dir`` - - The location of your Smart Agent bundle for monitor functionality. - - ``C:\Program Files\Splunk\OpenTelemetry Collector\agent-bundle`` - * - ``insecure`` - - If true then certificates aren't checked when downloading resources. - - ``$false`` - * - ``collector_version`` - - Specify a specific version of the Collector to install. - - Latest version available - * - ``stage`` - - The package stage to install from [``test``, ``beta``, ``release``]. - - ``release`` - * - ``collector_msi_url`` - - When installing the Collector, instead of downloading the package, use this local path to a Splunk OpenTelemetry Collector MSI package. If specified, the ``-collector_version`` and ``-stage`` parameters are ignored. - - ``https://dl.signalfx.com/splunk-otel-collector/`` |br| ``msi/release/splunk-otel-collector--amd64.msi`` - * - ``msi_path`` - - Specify a local path to a Splunk OpenTelemetry Collector MSI package to install instead of downloading the package. If specified, the ``-collector_version`` and ``-stage`` parameters will be ignored. - - - * - ``msi_public_properties`` - - Specify public MSI properties to be used when installing the Splunk OpenTelemetry Collector MSI package. - - + :header-rows: 1 + :width: 100% + :widths: 30 40 30 + + * - Option + - Description + - Default value + * - ``access_token`` + - The token used to send metric data to Splunk. + - + * - ``realm`` + - The Splunk realm to use. The ingest, API, trace, and HEC endpoint URLs are automatically created using this value. To find your Splunk realm see :ref:`Note about realms `. + - ``us0`` + * - ``memory`` + - Total memory in MIB to allocate to the Collector. Automatically calculates the ballast size. See :ref:`otel-sizing` for more information. + - ``512`` + * - ``mode`` + - Configure the Collectorservice to run in host monitoring (``agent``) or data forwarding (``gateway``). + - ``agent`` + * - ``network_interface`` + - The network interface the Collectorreceivers listen on. + - ``0.0.0.0`` + * - ``ingest_url`` + - Set the base ingest URL explicitly instead of the URL inferred from the specified realm. + - ``https://ingest.REALM.signalfx.com`` + * - ``api_url`` + - Set the base API URL explicitly instead of the URL inferred from the specified realm. + - ``https://api.REALM.signalfx.com`` + * - ``trace_url`` + - Set the trace endpoint URL explicitly instead of the endpoint inferred from the specified realm. + - ``https://ingest.REALM.signalfx.com/v2/trace`` + * - ``hec_url`` + - Set the HEC endpoint URL explicitly instead of the endpoint inferred from the specified realm. + - ``https://ingest.REALM.signalfx.com/v1/log`` + * - ``hec_token`` + - Set the HEC token if it's different than the specified Splunk access token. + - + - ``$false`` + * - ``with_dotnet_instrumentation`` + - Whether to install and configure .NET tracing to forward .NET application traces to the local collector. + - ``$false`` + * - ``deployment_env`` + - A system-wide environment tag used by .NET instrumentation. Sets the ``SIGNALFX_ENV`` environment variable. Ignored if ``-with_dotnet_instrumentation`` is set to ``false``. + - + * - ``bundle_dir`` + - The location of your Smart Agent bundle for monitor functionality. + - ``C:\Program Files\Splunk\OpenTelemetry Collector\agent-bundle`` + * - ``insecure`` + - If true then certificates aren't checked when downloading resources. + - ``$false`` + * - ``collector_version`` + - Specify a specific version of the Collector to install. + - Latest version available + * - ``stage`` + - The package stage to install from [``test``, ``beta``, ``release``]. + - ``release`` + * - ``collector_msi_url`` + - When installing the Collector, instead of downloading the package, use this local path to a Splunk OpenTelemetry Collector MSI package. If specified, the ``-collector_version`` and ``-stage`` parameters are ignored. + - ``https://dl.signalfx.com/splunk-otel-collector/`` |br| ``msi/release/splunk-otel-collector--amd64.msi`` + * - ``msi_path`` + - Specify a local path to a Splunk OpenTelemetry Collector MSI package to install instead of downloading the package. If specified, the ``-collector_version`` and ``-stage`` parameters will be ignored. + - + * - ``msi_public_properties`` + - Specify public MSI properties to be used when installing the Splunk OpenTelemetry Collector MSI package. + - Next steps ================================== - - .. raw:: html
diff --git a/gdi/opentelemetry/opentelemetry.rst b/gdi/opentelemetry/opentelemetry.rst index 3723c344b..79bcaa052 100644 --- a/gdi/opentelemetry/opentelemetry.rst +++ b/gdi/opentelemetry/opentelemetry.rst @@ -114,7 +114,7 @@ The Splunk Distribution of the OpenTelemetry Collector for Kubernetes ingests, m end Infrastructure -- "metrics, logs (native OTel)" --> receivers - B[Back-end services] -- "traces, metrics, logs (native OTel)" --> receivers + B[Back-end services] -- "traces, metrics, logs (native OTel only)" --> receivers C[Front-end experiences] -- "traces" --> S[Splunk Observability Cloud] receivers --> processors From a97c2539b153711583d1cd83f5ed26ee5d825093 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 10 Apr 2025 11:45:20 +0200 Subject: [PATCH 12/16] Fix? --- gdi/opentelemetry/collector-windows/install-windows.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/gdi/opentelemetry/collector-windows/install-windows.rst b/gdi/opentelemetry/collector-windows/install-windows.rst index 261478359..5cf5e8d67 100644 --- a/gdi/opentelemetry/collector-windows/install-windows.rst +++ b/gdi/opentelemetry/collector-windows/install-windows.rst @@ -32,8 +32,6 @@ Alternatively, you can also install the Collector for Windows: Prerequisites ========================== - - .. raw:: html
@@ -44,9 +42,6 @@ Prerequisites
- - - .. _windows-otel-packages: Included packages @@ -142,7 +137,6 @@ The Windows installer script supports the following options: * - ``hec_token`` - Set the HEC token if it's different than the specified Splunk access token. - - - ``$false`` * - ``with_dotnet_instrumentation`` - Whether to install and configure .NET tracing to forward .NET application traces to the local collector. - ``$false`` From da8f04a1dbcb24f3fe8f1fa67cd26f0545bea136 Mon Sep 17 00:00:00 2001 From: Brianne Hillmer Date: Wed, 23 Apr 2025 08:56:06 -0600 Subject: [PATCH 13/16] update aggregation limitations --- .../metrics-pipeline/metrics-pipeline-intro.rst | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/infrastructure/metrics-pipeline/metrics-pipeline-intro.rst b/infrastructure/metrics-pipeline/metrics-pipeline-intro.rst index 2c56b2970..417114534 100644 --- a/infrastructure/metrics-pipeline/metrics-pipeline-intro.rst +++ b/infrastructure/metrics-pipeline/metrics-pipeline-intro.rst @@ -167,20 +167,16 @@ MPM is not available for the following types of metrics: Aggregation rules limitations -------------------------------------------------------------------------------- -You can only create aggregation rules using your metrics' dimensions. Aggregation using custom properties or tags is not supported. For more information on each type of metadata, refer to :ref:`metrics-dimensions-mts`. +* You can only create aggregation rules using your metrics dimensions. Aggregation using custom properties or tags is not supported. For more information on each type of metadata, refer to :ref:`metrics-dimensions-mts`. +* New aggregation rules are applied to new MTS only. Existing MTS are only used as a reference to create the rule and display the projected outcome. Histogram metrics limitations -------------------------------------------------------------------------------- -You cannot archive or aggregate histogram metrics. By default, they are routed to the real-time tier, and you can drop them with rules as well. +You can't archive or aggregate histogram metrics. By default, they are routed to the real-time tier, and you can drop them with rules as well. .. _metrics-pipeline-intro-more: -Aggregation rules limitations --------------------------------------------------------------------------------- - -You can only create aggregation rules using your metrics' dimensions. Aggregation using custom properties or tags is not supported. For more information on each type of metadata, refer to :ref:`metrics-dimensions-mts`. - Learn more =============================================================================== From f58b1d6d3168779ae101ba61f6d96ac1fdef5177 Mon Sep 17 00:00:00 2001 From: Johannah Perez <75388387+johannahp-splunk@users.noreply.github.com> Date: Wed, 23 Apr 2025 14:52:50 -0400 Subject: [PATCH 14/16] Update apm/span-tags/metricsets.rst Co-authored-by: jcatera-splunk --- apm/span-tags/metricsets.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apm/span-tags/metricsets.rst b/apm/span-tags/metricsets.rst index fe2fec9da..8d637a59b 100644 --- a/apm/span-tags/metricsets.rst +++ b/apm/span-tags/metricsets.rst @@ -92,7 +92,7 @@ MMS are available for the APM components listed in the following table. Each MMS * ``sf_error`` - No -Monitoring MetricSets in APM are generated as histogram metrics. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate datapoints from multiple metric time series. Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. For histogram MMS, there is a single metric for each component. +Monitoring MetricSets in APM are generated as histogram metrics. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services and aggregate datapoints from multiple metric time series. Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. See more in :ref:`metric-types`. For histogram MMS, there is a single metric for each component. Previously, MMS were classified as either a counter or gauge metric type. The previous MMS included 6 metrics for each component. From 926a922593afe54b05b7a6860a2c7e3f5096b103 Mon Sep 17 00:00:00 2001 From: Max Bechtold Date: Wed, 23 Apr 2025 15:55:25 -0500 Subject: [PATCH 15/16] image --- _images/gdi/gdi-onboarding-diagram.png | Bin 0 -> 53829 bytes gdi/get-data-in/get-data-in.rst | 7 ++++++- 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 _images/gdi/gdi-onboarding-diagram.png diff --git a/_images/gdi/gdi-onboarding-diagram.png b/_images/gdi/gdi-onboarding-diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..627a2320f6bd1f95319d88a39f6d3ae62a55a56a GIT binary patch literal 53829 zcmdRW^O>i?I{4y+7kBI+0NJ0cW%d7tu&|qUX z)9$OfUY!daX-BBG8|8XFRNV1R_I_RZ*MMbA>9OZLm$zpsEB49{vS(TK{n!km^FI1R zzk8JXyb#r_Y4~$^e_4Yt$PA}>y-cw@!ed3JLUj>+w1tjp$V21~BV`l4sc?qme}gdn zffGm@@Hoz=Ys;Sf4jbGOHAqR}1=CL1)IX}I%=?#>(w_toX!7)~Z?dbl7e?W2^$n^j zx8t(?+psdGZj{H5q~)T*WLqB zf#|3Fll5%>`N^vUcxkkwgzkU4%0W{)Cn4J8sgJ2_$S~2U=s4fVIX`bZ(k)r_V?W}{ zWp{IEKE3m}F!A2I;Zg#x50@?>!`v_EO9?Pb*+m`q zeqTY|pD{KC6(_z~5u5aWA=PoMA6>ybD}0E5uexwq(_*yKv5HyUHc>YVnV=_!!q7o^ z?4gZL_LYxURNp-Uj-VJr)C5@jf{!OQr?1ffpm7D_2mAZ%ylv*IlG?^4S1ZY~vJKs* z2@iuLGc=-p`9I<_YgmL9UQN|twJhA1l9c5_XW#CI!T`ru+?830Dc(oe5-ncipR^ZDOcdb;|+H76*LTH=G|M zK)!72x87_eEOkK7z85z&<=^@Prxu-lkN)_{$SZ@&1zWDY)(tU= zjV}Bp*ftS*O|qSOD|NcEQM%N4@=*c&k?TlV3f4LJQ)8}wKH4?8A>(3y>Vy7ltLo)? z(rO-{F}Y&+Mf)dM7zqHT3b;2AU6O{%%$v9Apbe=mzsU0x@BwF!2&X{6g2|4C)d_;R zcgu@8W6p0ySY!t`=hFR52Pvu)^Qm+#XYBo+*d>g0sN43^?tf(+Vu<@|*M5~iE)J%Y z#69MfI5tP?rd>ozmKzG%NNr2+uWDjEFHi)PXb@p*qu*T7VKolzH^%Am;aW!0_c{x5 zf)i~4V~c!1pPs6rNLOZIeg&=?Vcm=2B8Wubt7tKU_L)>FDy^W}#n9niydcZEga@Qy z%{bE|Sh4at_Jp~TNsiT|?jPP5> zN@UlF!G2>UR$zjiDMXXO)NsKhV{GStujhoCLz@a88>M|XR7~t61C=LP5{U0KT>d4e z+Rlm8M><<#36_EkJPGmu9nNGuXp|wKr|38)9L#J?g~Gr4MrW2uGbHXe@+fpS<}~mp`w;he6e&YweSm7e;-<#lBTcGxn@EvPq^@D-Iq>=Z}aYysS_zi88rqfkM0evrJ}a| zpb6}lLSN3_6}T!>*C%aZ8{|)*SIQh2J_@@W|s%wQ(o7SM4pV&d< z^LpD{C#<`*N5RLLgv{atTYApitG^c@iIhTVY5xKUA{s0CLTp{6|ZIT+} zUY&5|uIR-@Ch58tNd!z^t5RXC^9_o1^$6u(YksNvE+pB7Mto#1(;F=Ye5o|{2A^~Y|JW}VlQGvvD*7!p3Y-e*^F+C5WMGa+2+nyXAx zH*^0uRYLNhQraO;8LENJ7f&OPu~RUzwrml>`|kI=T;!ttoU(p*(T2He2=TL0BUE0Y z9-o*8rku&mnR_bf7aL2)!)ClXsZ2~6vA6|q3PSwFkncUm_IC4~+Z=3Y%ya2d;X5?) zfxi>1wxA&J!5MqBOW5!I3xSjce($xkx1~+?eu0j>eD`QkUFJ2@tlOu5Z=iGR%-6w~ z>Y>Ndz5K)B1*Cvs>_It&#Ss>CdqaatGR4)JxK)p0|A+@)m(_B`FH&EthL$Nij3OeV zDJ2W2;?QaaYLp^r@m}CqLckfn_{p^?Kh%l~GODgJ|KxVZ5}xwLxqvT zmN9-0o$2P~qpA@VUAtUXql@xvlaf#9=5R!t@EMSVL~9-pg)Tzqw+4kO&OPrw6J%8r zwy&vY+C`GOn0Q%yG`qC$a?6yk`wUJC_Uv3LVLRDYYB0+h8~mRaI;G@zMZVS6b#*S) z$F%S%>?`RVG$xK6(y&c-K@3)2G3w++hziA&>5Vz;fIe-KgJ_xDC82wLqAKzgA`&0} z?E8~7x36#X!`-koJ*}H}Q9z9$N7&LR9%j-j{r);ui6O6jt(-T5t>MqD?(FYiXqhK; zxBxKtB;Bjjf;Rv2G}FRsrJEcRW3Tpid`fZ?!`R7q4A_$@olE%?Mo~Krs=uCH)c-F~ z+k_hs?UVy--ad0=gVz$fWrw>hS29!(PbHTVlNVr1Q&X5$;d}UeV@N=PY%3^i0%z0c z8EEr^pMJqKSwfBMijkc*fM=cJj-|dbZOd5ji)hyt!BdU#af89CU=HywDQsPcdKv7X z^14?(iak0aXMC|>?a+NI)szrtNUvAJ8AogGkAF6V9x0+Gj(JV<>sSW9m1g;HGAZV5 zvl)T*yrIgM7_332yz=suQUz!P>-~LPKXkn(l7`=<>zQ(trxh2QnA&6+n3X`S*?UZCqG}uA}xcTsMf~E)HGBcidg@1^kXJ;OEZm* zKW?P-S?_~3m$X9lT$^T@+Ni5vnru`y))yJZ=6c#kTr4%tbNyN{w<9ZDoc-SdhKhGn z;ZIHBH-i5$0vt=2KEvh^2L;PiqyMw6e*5RCkZv$~El}hmXrPhqUr+R)Xva}Y@}pbm zr+4EIDy`K2jcGY<(ZH+jnOgSnc=)EjuGMi+rA*?-e_#ejHri5Rr)kluMyBQS-_>eD z;9>jQWWcaay~E!=1>WN~e#Arz{%wsRa90Q^bI zJ|*)m%zrbV=^%3~&rl;6K>xX7;NR2iMpJqg?e^k_$eZ z>W$XbZcBqh7F<(8o(VK6` zFgE%Pw(dRkPuEUFl17rhtXO@CJWqKcoL$*wE-o-qRk6R>JRPRIrZ#|{;YBTDtHCdA zbCeb8mC}*Q?AIb3-Rjbzyx?RzT4HPUAlM!nvY_TQ-#DXqFL!`__I%;UJj3-1X}pEu zFG0ik(%aC^`t=b-7Ya>a!d1-aYg&0fF`FMI?&}uO?$h_WmuAqZ#;?HrBZWTv+_@29 zdeTilyeaS1KUeQtk{}FO&?|%0j=VMjEW6C{Cw|8z?6Dp6(NVPV7sunkLp;-G!N)ir zBc>{XI0%)?S99{bGnb+!x(B!B?DQY^u3kdrkOah3Ni(rZs)IsAz0AbD6h`rHdW0#) zL)U{|;?{8D%-EAujd?W}&l#e1BRwAPr5?JeT!q1*%C-IN(3>pNau98x(J%gi8Qyth z-qX7ajEOO6Z6HC-`XXP|R9dv!i-kD7DL9;(7JUSFb)|3i)FcT8XYUIn$0ro3eVqG| zNRPu%DeLUETj5dO_IL<=Y9F$K+EsrNJS?*sSU>f&DYJ8l;n6Q){36Pt-DE>rh~nl- zm*D&rB;!J-V;6?A<-IsyYt9*W`?Y-{U41CQ82q5X7|}_0T7G5~kd}MCYRuFHWAzJ= zeec*{ys!9+dbM(zHxkwbN%t!nd%OQ>wO{1vAyyFw15EBEq^xVtS2F+w3B-#3=dqz@ zjP^|lldBLr0WToQNToo4=o|;duU#DtWQY?dggYE!r$Fgf*q;(3rO{{FaplsqC3@M? z=mAn=W(|Gf$Rl{D3Qj&(P$b?Z54$$AGxCAbWi?}94Iyr2o(VOX_m; zXj!+N$b(oue51HU3zg(m=d)W#ssbjD3j5w^spvj@Hu8G9-J?T{!)|}O6L9->rNfWh zI(OE66P10uF$lF+=W@d=K6u#^wWabh{}3GA6y-$o{6N`%9m_$9dVw~1GcylRuZd@3 zFX+aqQd>xJYLqNdlwb&%u%4lqj{oX^oLs(uSRgMzs(Dn~C1EL#_f+W@4cf}4GE`c3 z!Q?%Rj^9uMo}j z5+{=g>r{jH6|y|YbR>+DqkFRje1;NaI`Im<8{-wKABu4$clD@YIOrg|s^ai&G2pu; ziQkq0FU0@?dc2i{sk9P#yDiN;0U8p(gv*P#g%Kjksl!LVXI60$ZkKl)}poI%}0o1Vqzg>VX|VCq5&ZxtTwiC|)8o`oFo z?BX%Caf&IyJzBo9G)zDz&$3G%BnNsQ>d$a%Ba7Ve45t7%d!b9O59hgJe4iWl(lg6F zZj?ojO|o~B=Dtp`tu3;lmLj&Bpw%j;&XLsua^yRv@HS=xM$0Hnybt?46F3F5R;iW; zT)0Iyx1)J#R?mxY<`G-c{R#(}(R)I^;3zK9D&NRs^++uo zkDwDjz!9v;7tDM1WQ$})56K_C#N%we;K-^@Q)axd+bLvdcjY2Xa7M&HTWzsMJ3v_f z%$()=i|SS|L{uUG`xMtDc4DBnw7tnW_$J^Z3xwj=syfV`GvERD3-TV# z_tNMxB%$pJZtWaLWpESW4$m}H8qQ05_v_4V+fyGts|yT0MwJ)N8j_>`l-Jz3GnpG8 zehfYnKB__UGZyH}`MhuT6cYM1k0FIq)ctvFoIDn_v;NOn&K1~ck)pBfLK*AsLaU;& zhrhYsQiy!E3E`y0ZQq{{)*kY{tt^Ygfe`6^Iw={&Z#lZ8A7-O}J$-Y!@xdry2G#i{R4G;>lS*A#LpDAY zKY;-`ir=#RS7IM&a?*Y<<%j270p=nP&FAq7<1F^*R{HOQ)I?x2-{(VoT@Nn+=$~1s z&Eici0aTelRwjFU+as=u1)v%4X(;#29yN8NF~nSVidN&Mg~AgdH4zgqelF}}sxd-~ zf2zuu+W)6Z(!oMh6n$MqZbxGKj8pi2#K)^8VB(Y&>CF~lvK%J!R+xC6)w)@`L8401 znq<^V=_(qUA0Fgf4T zWEWNZZEbjG>qL0%65F6qgT|+*3Evz6V8^?Z=eow}?062yQE~S``=B`Ar*uA>nqBnE zpL~?8a94070nJD6R_9a{_A362IKJ^Bmb0N`1&MU!r1j$O$_>(fY&-WHrg?poIs)eH zWo&%-J`U?bjI_ow^CC)MwwbF12!HEV7C}bL&s2;T0A7XAQ&ZeU6VXZ{=0h}iM<-8+ zEfE)QZ2R!51ia0I z3Km+~MDvs-xr)z3t=S~}Sdui&&;}A4G`5z%16k$msd{)jbT}l`HC> zI;@MQ%%)$&4MMZ*bYfnDx4 z4Wf`x!O5oT40bh+oT7EVb=vI>tYtK82 zoJ{=08@EMLJ3Be#t`(51)Y*@Noob6n&t=SqW=XYP*mz%qj;ca9810kz^mb@a1a+H= z9Eab|=<22COL89Ehy*pYUVRpMUAI4ugE~@h|G#C?{L3hLMti%c%02;bnQo&<9kg>X zZP=w+SV#c-WM!-$C5w^l%IFoP==jB)whfmW3MRMoi}qn1)g-pxCJL9D=@%Z{vsx~+ zU}_3EHu}oiXR524emb>c@7eENU!+MCV~+|b{)+F~5aZl-PBG6HRZ+cf9szMLrLch> z89U`kb$tB`+@+HRxzG=duAqEJ5sRf7LR%4Oufs)4>Qv}4gu#N4jIaJjJ^V)9v~a*E z8*Gox&GaLKS|9edg=~^bbZ4Y!JE!~cZ>;hMDw7fKM)rC+oi2#AYE8$MTm;-;#qoa{ zN5Q$a&B*?g-7TgW@DJuR4ab3F`Sk&}ng28qF1wb~WzGqBH)i%9DBA}|k>gIAt?&N> zIs4%99suz6|4?10;bVOr$9PT0PD41;2C;g&20nAtcB#s#_ziL!wu_n1R%?m3dadyh zV7H1;to$lgonLk5#+IZoeA|SxzhU@03Ak#-6>^kR0)%4q+Ol^vFI*`_LGGCIn>}i1x>+pk$nbFWz=pc*D5i-R_wNoQsW|YEiB1`vm(B=*hHw!vdH+WE z^k^6(y}dGV&@A2UBqxgWXvy2J!FCV!Uinn!!(mW>C$CSc?4*#^UMQHwLy_><;!DKP zBgaJwn`2xz3-u=-5B$BKf`8%mEu2jBU|$xmsHEpMOxHbayqD@B92?>y_lG_K+E0~>TYiqk8L5+>A zP?5Ee-6M@E35t);A~+oa_^5p%b6!v$Xs$=1y|st&E;l6o*d}x z{&BAbW53%y>}_Dyh7!?nlyy9HM`4K62tL}*p{m~(;CN{;9c@hxHic4#W0E!jGe3G{H2Zqg>NWMhwS0zab>}gU_?vgGOFV{n6Y8jFClk7 z<$Wd9qKLB_NG$5^HTw?L{<9c$!)TmQ2g?7 z%B4j$T~C4FAm4vxbn=$nXG7rVFVr$3Zc2~FMO&vrcrD$0yy2c!f{P$Y8B8r=wE}p8 zza{XZHBshN;?v2uC?!e4f;H%dDl|{QhKlsrYT%mWu?k&P+HNmrZ-XG!V`oR4arIISdxKGxoAfgJDkQ zp7O22X96}V--L&opGxF4Krq?2M*Dp)DJ%-nnMlr*r=i0=a1kzt%<#)Cw@OKJd;Dur{(XN!#{bD-<@bBf?{9gfc~n@tV+R9-O00o!dn6w82j#K+Fw52p4i-aoG@z0M2I;n0)9{e&-tua8W6r>ZC-~QpU;E zld<&t{N~H3q@s3*H_W~#4wQB@M#H(3aVEaapDuSuT$E&TmBZ_m^G$1XAlcLUiAr*w zikB!TY&>SOq420Shj)*WPx`=LY)m1C2hLdWZ7|3~vu_7(_z~C8Qn>AiY|rO6Cb&YZ?i>IF_ z@knUB#P1Chjb{9r*$$SC1`jx4nIt7al>nmu@^Iept%}KWXuQZJh>{W{| z#WGFd3YC&iB1P{UJ@w&fBQxvG(c?yhrca*nC*4`u`AT^`ny;8O)FB;8Y=;yZ%&+RJ zkh5tV43dWZD3|yO8SK4QmrpB&`3ZQcYJ1;Q)7pYySs^5zF;OZ?g3^_= zDRE1%XK{p!cZ)We5Dj}P-5uE?Eu|y>Z1kY0lV5zzK-2g{?K5g_Dmnu0dK*m|15uIE zsf_aoCo`_+>yvcJao^=u2{S_mMy0~PnwS^Ug-;0|k)knx>$pNKGFSg`$b7*&{SW=F^$OiwA(NZw4 zfJyLD7I`L0KcRBG!vO`MzDbsC&sJ_#P|X`*Y+B2YijjCJHP+xtakeJ?l}4cuYCzAv zfrB@C3Xn>GiMvyc=G=<+VL5{zl<&CQ9h-M+mG*1=w322!XqP_kW?PCZ+^$b!Q4ZHn zDLQth98RuUlV<%Y42u9{?V3v(0suF!s+Njv6NN)Y-c23Y{uwP+x1)VS7Jpm)5-J4VauZ%KP4CO$kL^j29vv#eY+{qIF*XEk3C4*QkuOg&NG&Kr{K@7^ z+97p{AT!z@7`1b090Od{>NQ0*}u&BaMh`Q*N^OL+Zy*Gho`oxBjg#4Kw%_xww( znhNWtP|6VatWSeQ9O^U?2sW+U_F~1#D3S@D^|6U)o&$-=ZDnnLtGo#!P9WcU-m}QC zT)3lWX_2Ei)>8d3fcly5O)$a47j8qjPnAsBXo$!>3tuzVQynJoPto+nMw+#UmVb2J z=s(1KsDniLwUQt7A?ZG|#1Jpp-)@#iom~iREMA@{Vh{LLjyh6O6}Yy{HoDa>W;gpj z_;lJntk1gZ_U!3NU$hmSDw)Wn5JeQN%`03DU6Tx#-GsT-n8(BI0P{KmwUKiIPr88kcX%?K!prhoccEU_AYEtpie>S^QPWS}tVXJ@3zLg)3vnVKW zPnu2c$%-u?Xs(Xtc`fD=W-|3)C^=6WL+<8WG_n{m)mf2gh6H|OUsyLy(f4Wj#vNp& z2b>?`{K$o9r((ubAKp{!vP-0Jsj6mfh)& zxbRZ+Q0#*%D`v==WB0t_L&Lu^SJa^21WJ>J~%S-4FK>+#uUPKZK4+b)b=Nd6ABu>7N7!w3~u8xZF)k9=w6d z&9sYdrVmf~b|o;~>WMDq%C~}IS#I) z$NyDq%s+mihZ0KX|D)1zE-+VaZ32jEw_l=^WSSObKyTu!P!aPf0%!(yKUNnmD!J2; zo|eDq)xM$3Br{RS_>lklNklkcvdTr_uf@hs__!>l<=3Y>h^;c3Ug1k6WV~K4k1pCW!a}L zelN-T&buo4hl-0Ycgspj#{@;$gdsbBc^MXF#2ux?j8~7tFDp6lyq2_{Sq2sP4RNnD zz}2Ec#g+fax#zzn@qUn)QjNK|zW)y_EwNafv3{U%wlt;@!{0~>ehkJ&!X@I!Ze>5- zd%~xqKfr0!cTYyhpt4f#MRUU+*vD0cDS0RUrTs3$+dT|V(fl){`a2B{7kTW48Na-+ z$XNMbZ4(ALz~Q+^-*D-Fu)GdBJkhqiF#rEh-Hflv-tk2rOICcF3<73~EGFNSCF6~E zX@BoorU5hb zf%gASORbsiZ~KGv=q{7`4cWI&hPt{8I6qNhmy!jvJE((yuBigQfIML{0I|{tvD688 z8k}IRK{D%~NWHcZ5MbqII%_`z>Kos388y>f&E(vVM4L`pyCu&XwXZ%JcOZ+rg(v>% z%06yhC^5oip{W8EAsB_D7S>%~T?;$eZ!iye1N}yQMz4_~<85={GJa;X2m%S*c6!>f z`2u%Sme>tNT&)^zHThJQC_xo+0`jCg9lU?12X|wmeZv%AdE9b*P5z@u%z;SE{c+eS z1kDm@gXzd@depRMRnf6SsD9;(b*3Lcj(QL?I`IrH7UyW6qgJ{4(~ahDmN=8RQ}x(i zfEO|86`?i`b=IYole*BlBso^?c{Xbs9M9+-Rd4#QUCyV>1};_Yxwo&bGzH|1Dy|kF zjg5-%6me4l?B7tGHpw)}BC@)j^wdF&g(^*O_369z2I@h$UMXl zwe}|Gj9R_!86_zBB6&D<)V`YezG1~SF7P@Seo}CU{06}mk2l^9*Z}JfEjYc`^GD>W zHRVTE3Ff+p=r^y{Wp-A^mm!W$t^l}nFq0c_Tcyvw)jmDi-46jE$}hSuzrsLc-5K3> z0&p_ySQ!7w5|Ahi1tQ;GYcd9p?C?Lc%DFx;Ed9FjWYDchTQKa+H0$UncgIQVsHU*D zvGmRa%{@sn0rP;2dui8*wIYn0sS{2*bm##AXEiyQI#8NUII#U5OuN=`2~l*83}gEO z>cwnlia?Lw4JS)fsxDe?=KY$wt%e0H-$R9{c5X7U7Svj0K%|sHg)a*IxcrBwB66~h zj`sJ6c`2=K%g2mj4!(VO36cJ`xQl!1Y@8?QXlN{TzPB&VSM$p%fc1p=R?0Aoj=pAP zgyC*?mj=^Hv))SceYU6LC&y_K+=GSF{D%9?J5SKP*~04ugG-_Fi7L-xrJ^$Y*Tank!Yu!xT=rC!^~mcv*|^BsUQ%UGlIn4m8FAy;;}T zolIB3_p{I; zy?k0X#a5-4mnG*oKokwT>B8VUu$@es>kFZGzy?mm0c`b!&CWrUOH^Y!@gp*d7K68L z3uUA*R6*@{t_*%LYVzv{AbaMMr1aXO`=#&NwBGNr7W4_#tYHY{#JFhTawr#?BvTHe znN^NTiYAr!XS1UDQy7iLi>a;H%qBmwT9@+d;-$3Q@|O>N?AV$Z+yqkT^8{D2cN0o@ zMnHXC-)wrGYzRE_$69Hq@m@O$SZ1%oBvJD^zw&_yzossRYyL>K982l2XKoW)2GCI- z;DNaJkbji!ylf$Z!xfyZ5Xb`|wDKw$KHv55mXYThZ=G|fFWJRCdINFEduFk1ZD>9y z04?nJ9X~Cx<;YTQoZ~3t?;UDFW}w^cf2dp>uMxS$6TLil5r~5VucPFo3vi~U5Aaqa z#E7L@M_j$InTK3_qv8=l+A&{yno1RW&B;=ACWsaVye_g53xD?^o=&TME|lUO zfe9U)+&IFa-(^j{lI#7KdeA44 zHGN!i*Ch}nS8>sTiR?gH#GH9=;iYlnUFJB^M$s@Lo5mKLU3W+M$-YTYs~V6JZhAja2%)X~|`!ufka>zO?V4 zG(qe|x1?W$#Td=B;|@dRcnamh^N!cqdrF0gLK>o_DJ!8{d^Z|)n)H;@Pf5;e%Z4~F z-Uf@^uAk7cUX#Oi2icnQq{6icP9as$y?VEO_MEBvuY#%9oa=?-7b7|4FYe)lOrgax zBc6KJiNO+qKzJea2EnABhJ85gitCbkyW?AjY6oSYx(Xnmb#`?p@74RRMjyJL>{pxj z3?%1wOm6ATX2AI)PyflE9oEzHS`jg~_o4Un&}CKO*vax+^n0TaY|)PkMeSYC*}{Y? z-upD4s2$dV+f?59nwwVNL#NiO>;+neW6|GEz8mc~dWyg3JX;5DN%lS?RPtI|dF;mm za&Qa+wnY?=&_e8W^I1@+LQ&gScy}_-60K1@A9$Kwo=1VW&KZ3rZ6^EI203xvcJTT| zQFubdF&(ok4Wqvwgypp70lXQ6P~i4j}X6OS4gse)rO()>!Qgz$!an^rd8-G!_EA^`Av+a#|fQHT?e5=V3< z%>M&pL&g3W`cxO&ij3DtV-8&%-o=6?^85O|Pc8zIBWaceenp+3~b6wtSWA z#N+0ff}WG31<8$D-utIP?SPnS&hgr!Z-#%oB5IsWf#9#GU!0&xfm42``DdFe(Q==f zFh|$c#{wxO3J~OZ-%V@~O0Z)Y=o1uC)N6m^gl=$9riQ$yXq6WryXesk2O5T*+35%6^!^FP_*z-7ScMVQyxT zwfAVqP7bsu*+WL42&Ld9eT-4PtouOmVH|wYpDkKtI{H+x5Z3zTF)_klQV;#gR3;&gB_I=)iaxuCmFSEf{zz3D`(R1E@F(t+h3IQkZ%y*s+Ee8&qCv8_b$;_ z1Am2x79wdNm`T2jpG;L%ocmB9W{ftfHW%0_hM2U%*%hr>QU%b}B^b(ye96wnl`md( zj8m%BR%8EK)5g4oPCs^bL&QY0eFau%(Eh_Mul9QLA@WL2^C5DO<0`sZ+))6b$XCue zRUKyhZMB}_Akt0UJ2wXEYCpj{OOxbqC<=co>P4z++UXDCt-i~@F#`u$P!`QlO|R$2 zt()l47*6G7e{$2G9m$hOBvTuln%}6fe%j!o;F=tpm!<@vjZ^e`0ir^6V+{*tiO1Db zq>|@S-+mnDelk0wT4=AgnAJTY^FI5+hB|`a(zpG(ns{&OGX)9NAob9*qeR(8HijE= z6|*N>L253dVx&IzyIQr`1iUFF$4DKr=;Uw1qElf7sMYN{}UZ%X5MK2l3WL!MbEOslCoON=47A?<~1I`Oui#Y~2ZBT3g$fpVyZ zVWxHl!AI%l1R?y2sbEa!SU*q6&1CEPs&kzrYr?3D;K(=y!U!f8BO5B==_ZHyrgO1i z3us(9`~aNsd~VK^uLO&7V%v1T;L3y~Sd`D1(7V@TZ6&}k-!nt(GcX(U;dA-SyYwppx%;SJ&kXnlhO^v8lVpDrAwTh>i1taP~Q6pia z5k!`n{4B;5@`G9l)+?_>aCPK!XX(J#xlk>}wo6tzUJkVIunSnoCg8CN=oI6K#l{c* zgaznf8c49E#$ZaYN|v?ScuFok_ANlDJ%|7On_~H$p1jVvE~({HEc_ z5Xx<^dr2zB0Vg+9o#VqHdf-fFDjni#AFHJt(MMV#!j~6soqxsggwS)`4q z^2AdH8%rw`4cka(p@^Ct;8J%)i}ZGA6XtPZs+m&;>**JXz%Wtvxb>&fjMSbVXdsxsqJzMx65DKsv|$=C4li4m7{$Ea#VUJUn8$X+<&0EQ#^RVXiZP-)a^Ws zLB%e=NFm7ehVYEr2LvVbgaO&SEUgzv-?o7>%a0K|CBZq>>V!zvoy7 z$_#QK=C5-r|0=*7dTKw}@No42W%UypVm9ZAeg3(yUu41HUe-SMRz*-WdJTT9QJz3> zq*JaXk$9aBOO&=TO08jDLIg)=JzUC;rr|0jY|9 zqerrQP^R$|p!ek9y+lClf$`6in&14GsTMMuV20)Nx2a53az%28J!ezj99!-d6>YwK zV5-S>PrTW1i)(Cazt7MU4S$PQex6_T;=@UCu850d-)0Ap(iM}{cV)sXP54KtwwUK@zYjw7pfso_O^{rf(Zr=^^$HAxUIeKQc;db3KOCxIY0owm-BKq zKkuB2q~(yt*Dg8t^yc%^1ScPDi)4V4J?Zp9u2=;Lz>~cNS(F)3lcO@tOgE^9y7jEJ zEx5`Sq?b^9FAueUJ-!3KS)%8LmXjTY9W!T_hyWnpOjUP{PqsGVl{Clhh^6bmd>vi7 zUFT8cO7&m`D&Fj$4#;umnV)qfZ@VGMaP2qCadRk_#MCFe6bYJmlLW@C`tYM1*(-B_xW`8W_47_WHM~cieeK{8^scn2uafD7IltQ*U8A?L z5nwr-ot4@nR!!@s=O0NCPl?Ns(_7n;jkiz!&b)yqw!(91Pz*mdD@ia$j$5E>gELi` zmy=9f*ZWygJ`eK%nsHs)pdrfv#lBqWN$5(Ce}4;rWx9;-K8utyi1^BS5DtE&9DEIPX4+&KN<^X54rdP zWBzw>8txJ^+2EN$CFR0)f06HhBmu>CbHTH6+P^(t+xX``&+lsz#TfAGA<*|&hX1=B z3a?-N4bB|0yC5}t{)bP5O2B8|y3qII&nEw(A6dBJgyQJZ6SkFq5tTBgt-mSI%FF6E z|M!R7O{b{+0Bu?=fAP3EIuix#OnXc(Gs^g{$((WKJg49v9AN)i)fnKtoUFZ3;37|N zyr5XsX}EkgJFdmvad}u6Q|td9lM+d-h1;@MkUxwrmR;h>v7(E2o{@BM2!LEQ z4Z8lM!~Zb&TT%qM)2ptA76YGtsSwIBWdO>}Py+Cc0O(|FclTeEuqhL^C#N(GcCC<2 z7QeX2U7u!3Wyr*BqqX(y_D(SB`Z-y+`Ga1{s34?I3n|~FQ!EPR}x6lNHtOfX{R9};W zZP~k;69(Yz>;#nuQ17SO^cA%)^6+y+^2i%{aRO5gzceX&Kc66R>;zy*cqRHD+Rf1E zlza`yJ(vuCXE@cl`|4~4s_hS11_2ZmuzE6m4i5%`1QwPG`HJ&N=L@+u-81XANa*j= zMO$k%+7dHbPd*FOtLLI2Rlb=g9Zk4!6r^zbk){tSXZD!mCDwXtOf-kH*Zo0>z-{2H zFa>;vLt;Fe?&@LJ((00@+36eul3ZO`Vo1dz^=(X|l)Xir$Ik;=Wf||PfQnt!fw{`I zkh-&OKF87FUnL#^XHb>a0+WNIj@zH_tzko5+WBMd0fZy|jKh+c~+{!4TtpIQg%_sdN% z5m`#9QBwvBNiliU7?o8BAA-n0>CT@L4h*f?UsD#LLS*}KPAxQ3r2Go!Q#fq1jZ!mb ziUw`NSBgb6B2cXcA0|^}>{VcMO`Bfp8kbb3_3}}#K$9K7&Z5jFU(nbT?1W+7N4SHZ*Couu80AdJ;) zwC^a^UIwM3zaoONWBL18&XI1#X5sXOY|x3!^FhQ~F+ zjbKE7vu|5ItI*nbcDBb%KlR5BL=D1NjvF;4S*iUnXl}1aJ_)fMGzH(#AH|o&&nnyK zZEW?>Id_|);Pj;!J@Xoh;V|X&gzjKdDHdI|d=nkvL{d8+;}D~Gm@M5&^6pyrDEAq~ zVfSHemT$&etw#GIhS!PBTkYh$0)Equin-mA2BO?j*5+#3M>+o(r;LFX&w=c5KI(5r zy48cY$>4uyP{HSrNai-{m4z0pMu39AYxNi{Sj~qUGUeE%D}0hNTb9CP8QOZFfK&iO z@r$l6vNndjYZG_VT_`@t+A0N)l7{(+cS-SNNPMVPmhc{K{(rQ+WmKC@7yercMT$cy zu0@LkEAH-Iq`137(LfacFI z4uiVSN_Hj)#bqIEDTGzlv~OINZI~`I1QL!PbZoN9es1UOBF^}Fd)n`jT|X4GDV-eF zTvh?$wZW4Y$RhPFn=!r%vYS+f_O%Z^g_Mty`YH|*YJ?F#-1P!fccvqA35qd#t+TIX z+-Ij51u_TV_tXS#>dRmj*pIGWn9;(s$@)?fZ~5_@LUC}#pmPG-AcYStMh*rJBhxa9i;G!)h-0N**<&j4gYZ*pd{5>Y(GUD(eV4WY$e+uAmz;6aiPnyBD-pN-TwN#pX&&K6A>+4A4H#?~EhQy4mJ0Vx$n_*>l z9fHN0k@I;~5p`>)wh3}yZBqO+niLLZEVFw?2AT!%B>AV>ZOwjLg;jFsF)djCXgM=7 zMZOYn`_O_NFo$>S;z7wR zj?A*qde)lXH%$bON_gX=s^9cHOWx#gthgQSu!o_1CsyBof*Dka-g5)rAron-MeiKz+LTt7a z;F1L_R2T3_wB=))jHg%=@_A*}(+n)h#)Z(ADCXNc>^p#GMu4{oe+D|Jn2D|ILX4*s z&W1RG9%?iJnHEkGy+dW?7!>61WDn zwES|y4&w@%uv?V`i==>=!T=x1&;Tl2P|&wYC!f&GN>V>lHK@?QZW7RUELJ=3{Tl}{ z#&jW22J_?|zfL<`(_4lXR05-tUGCgjSw|^MNHUeb!))XX!5u$ajWPPo z15h6wSY?8A`N;M1(fJlE4E8Oa31ee`2yE zq{s-UT>ORXU5z~!IHuCl{3NeP(YD}uZ9)kwwO$!0dv~US>QQYMV!}JqoD#l8L0cn= zsj$BfUu!D84k|gP0(^rtJFT2*QBbhEqQ$1r<6z2QKuDsv&vc9&Pc{09xccCwcq93q zJ)jRpuZJU+=_sm&>KN!?S{P-<7GQrGQye>=#62O^AIa(|0YAfYpF7e97rM@82{5tM zS01qKbZIWjpHEa#wbWIGzhNe8L)9Cm-TO?4aZLQ6Kmn^=l&6%9pVHRWBo*hoF2fIJ zld^g>mn{_3Vtlm>9C==4KBdzT3BM<;j&+=%jTK`xKaZukXJ!24zGg8PfD`<2`EQ8| zw69ikl31s4fm>V=DVpiDaI23vbJ6XyBsw29Puvt%TRUSzN~=hGivjD(k#P!oqL3fr zDrnykNl&InB6g#t-u(`;22iZrXQ;vz#RPP1y|J^1C4toErY<_h$bM_t_@htxG37>P zh~uX%0H8~>3pxbWcz9`l3nUUeQP`h!Oq}6SkFoTFmv+NPkJbB0puhoCKgc>1MG$d^ zy|zme6lV2ks)r?mF~#wlH5q}}?(2{_SV)Wm`)(qiv`RhYuEVd^S+R|$@`1j}Fg*Yn zIZhavg5o+pe;@(HQiLZh=v&E{g6IKH0G(-So85SkP#rV~eA(eQnzJ(1uY9xNgIaAQ zf{5q=EtEs<3fCFs9Sxv$bo4ZL>608o7duxL+H|B?GIiMfn%n`gj6II0PoC30POY@f zZZL7{EeID`lIj_`$4M>3nP-k?SDt4g<@{z}j#U?$$Yf~Cb>Zx%0s(tHUhQ+uX$>Z# z3SI$I{QO>NU@%Rc71pJhA8Z5s9`VtVdYe38_I)9TpHD&~n>0W`0~A$y5v9a;J`?Yt zD~}~-2?m+Wjs8$1!w_|ojVwyh1mh;Q$-a$RN@rcKx#tT6+po_O?;Oi;EI&}fvom%* zq@s8#F1eE~041Mc&M#1Q9R}mTll6lt?soU~sc$sSyc;y5aF`*W_9V2C(<=?UkNmhG@3eneElM_Bcy5kun4Z=Gt?0 zveDj~CzwK~)5{m(zWbN0v8|kxM%GJ!t#{-*Xb!h;k&qszp04f|QM_MAOu3C8-aw+? ze~?oe&sni(P0v7xeFA|Wcy`)(eTs0aQ78kFUxRB!>YMVnw-z!?A1SdgZ|Q*%_wbBs zKbQkSVoX=M)iUbGOX{87aqrZ$aCOxzPf*X@*DU-AojP)5;+=_17$-7!*bU8Y0623!W@SK%&(z{{fU;9Hv6Vr~K6!DJy_ zpob&uTNJMpk{b=_e4F)Y87f1-Gtm{jHKZQ6@U72)vM`zk8fCfv_`5mE(brY}LU&Z*1(IbTjhg?y z{-(lKuVXp(R^P{{Osb*)j|M!*K4nAhT6p{0nu}1>qayQrK zj2AW+3LepbHL?cFk{Lg^WDcnRr|b@ovPbkGa9!EEBwt}A3HvA$juvRQzXv( z@sJ?`JlBNF5Y!~u^dKLiq*O7H3XjCGnBq`^!*T}K&4vvt+q@I0$Z3dtP4=my4TWYs zGtE1uqu4+7q6E+`@QdFP)(msBo6`?^OIeaG2atPL#ufshtd~UJ`59Fw zJza47F@gtcijl80g+^bhci;JQZ$)HTdI+Zm&`Lv+%1t7LHird=SbOc%NZ05hfhCHh zL)%1Vzs^JP3d!q24@xS&Lf9Aq7yLaO!_~-ReG!TYY-3!+@fI0h-j|>=h!gq_QfI#f zPm|DNZ6vn@jDD|j>MN-{;G6J=PJ8Xe8@R;yee(NICPhdq$I#Syli($E`KEepPd&$2O|id3mQb)&pa-r`9b;-XTS{R+8h zgA*1-P?r|Y=Jm$w$dZreQ|yC@h?Fg7i2TBJxb3rX*vY5aLIQ9CS_(x8EVdrt3N1FMgq8guT7MV-xza*2u;ZD0#L6M z+pPCIdfkTynp^y6`z3c=f==c*p}9iJ+uq<@TT4Z#>=J~h#>r7M!Y~>4%=5yHQ+?|u zR3|Vt5s9nFx%KsA{o!QQHK}$>;!?6zs#};3GKM~OF&uxIcq0|9b2&%C?fxv0buia6 zA`w#{i90^t97L>*C!Z?rA^`>5bErcs!YUZo7^ALgAhI)iCsYdN!E`AjOLF^>g-h>s z-0H8r3@cRJ9F?gbdq0{XZFGUIE^^O>cTAC%Y6DaKEkDR5oqN!I!tv>$PQ=cv3)_2* zVvh>&ZT%v&X*dbsAzv|!aqR6atE9-U!!6(#PJT7~oe04TcvPi=J$vf7q)RzZr<#F( zGb4;&H3OaCKZojRdVzaSa*AA#VJQz%w*$@5S6r4KGaZ5>j-=_r@k$CTDt4^c7h zFl=VF%l^PN#Gn$Y!HsXIhz&@bM8cW^;|*j{>(5?KRulj!D*a!U*s2weofgMzU?ZuG z7X_tvi>~uwIXgr!gW!e0@xM}$*SnBDasN;$sP2qr>2sUEoH(2Fi_y{Vgnj>@>ZfV6 z*Sm+hAb_AR=g;3=qNtyhuYBqN2vur{u4_{3K_WcF#k5C&IfUV}-kNul`(tRc`YmeG_=e9p3_=A7Du5oo=1`FNWw{KJxzxzu$ ztS$6Bc2ExuEsw*@;QWg-+27u+?6lAE1Ld7bI`BAX=~e#gBUOlcf_GkE2}9xs(hUA< z5BQJcYk!c~z1tk$`7|_@PkcK;`iprPSJ(esC;yQDC^14LCtWXS@QT}RymcZsKEt5D z40P?BzD^#-_;W+(51Re!b$>8Fm5ZdW>-E~X-p|6=h`?=D?cl~1?5C3{rS?I*{T<++ z?*7>gAPPFA>AG|9x6k^9L{_CpmGFcG!_(@K{$F2zF<)0|7pHaV*BiA{zW?W)e|=h) z2Hf(h?G{GlLX8ZAfi&X|*B=yH1>+A66|7fyV zUcmP$6oYC<{({#3#-qyt@1iQTs5bXM7p(Zl(Mu!efb&1mX=A{84QXC6>se#An}y{wd?tK7pX?onJb>kb639qMOD($MNhm5bj#0?2en@fg?xgifI)%A_uG{V5 zv~Vs0O&g@Oj8rh^ec}7~r0EXjB#q2dDnKZ=GSMl~@uLRz-Mhdh?gkcVpy6M87$zh* zf0SCkzO)@=BFDP=UY#UTW5jwp>jp^k6l{5pO{4i#qcKfpD!Aqv;JbFc>8ml5CU_zO zmtcFoN!C)V+QRFEsTUnU_@VKJ_Ek+y^B$7~4>&naG!Y^`UhAmtyvYC3)9ccp_F38NB;v9_IYz4l{01IaSYP0fH zs>Q~9&`Yn$fvLj0Cbn_^nz2r?PTW}ML}P7 zB&L?6ot2LN`a^dq{H2Y@zVy@h_T23_#kn$u?z!K(&W8Q#9;(}Cca78%HIA*;iq&vw22Q0i2VAbsjd9xteUP)6F+x>;}^wr^Q&uxG~UF9*H zvZpDnN%zo4z+B6l{%~4#b zPR5NYL${}etA3k;l;b;4$=(bZ|0b#^n?}eo+5Y3Tgx|vX!F|sKI{RUxdF@?9ppU0` zc1}eJ`omYkL)r<7Owzhgk4bvVC55)zON@ysn$LygmZHay4#$Us$Elb&qSEXM9r#_j z9{i}bUf`GCR&2rLdj(|eJ8S4oWbSV%cb^m<7xoH-lvpELdO}+TJ5JHlk{XrAQP?1X zg`E}06Ti=yq>yx46L?jEz7(QmP#!#r{SScozc1f_iu51_>rl0yQC z(b{b|k;xJFh=F9$O;?5Q!GIm$V+5IG##O*g^8J8VFz{~M__jIU+s9?$P{%q`AN}}M z7OfG9f!_^XTDbFv-V{`-VI10n*D|wp`&C(Eo4PwS1Vc#Nm{E9Ut!T#Gufs_ZpJGA~ z1t{cdro2-WN`6G1Kcp}=8htnU^%^SExF^J~bztHO zOel9vym^?5?l_8QNcuoRmL_mh7|!jdTjic6l99>*E?{|FpiB04fYU{3;8@W3o#>70Hj<9#SOH%yKpXL%skS)UB zN^$w(l}OURj!4bCo`!Pc7-gc!`lOz+M06cpD>=LDkOcanKR2GoLZc(^MZlmEL$$z} zOKaPLWU5Zx3*P^rKww*VzAkRemnd{Uab}pGcbO>RmM80QR}8lvyHf$Lij^z)QS;7H z=Q`Ov{nVBpACQAhucPsyx;@awiVQO~9H{&>L?A)=*Fq`iJd2bAr7p(k@ zh|sEv+}3^OiHM7U-8c~sQI9us!UJ3}16jW22-^iOR3VB@Rdv2^dm?5dn{n3NH8)pphC3oD_I^@Yx@Ug`io_D~e|1q`$p@T{ioZT@x7ZH? zv)zA{w{5I8(U8tAJ}&7ze1u@$P)Gx=oCY!q#FnF}y9}d{0P!(6jF0aZ$YDB@#CYk*~z!Ez%&L5d_nM=XF_S4^1C~2Z`4)9 zJeK~}QNt&qK70#`-!OVS;{ju)Sd>RI8}T%&#R2bVD$O=VR9rK$9+hxHP@a(*saLaN zE*POE!h3B^e@H?X`S2sH&$^V+uDN%`vEPx!1t@H+6XV^tEEsCPhc(_QY2zm!LtR{o zO>s^%KjfsjEdke=MSzm`u04*9T=p{8ZXSr6CdzP9lY-Muw9@{p)}bT&r(y?c&3N79 zXU~)^ks_BePSu$Wt}~H58_)u#+NhXP3Z}Ma$Y5%jeBRq!}}RmZ;TQ zsO8?lqhg;28dQkNi5Q8_<#gKx9B&wFB|2N zjuX%`quEuGxSLjf#-0LgCG)!YI`W;|vtds2G9SjbN4yh&k;{L6_!E5|r>cy;*D}p> z^A$~)NB7r`e4=wxqmV;kW8YlGlEuxrOEUs>u=E&Xm&lP+pN@BXA z@^Mc^oF0b9A?VVgRe3Y?zJ+_p?hd3f*B{>or+nDT*~q;aFTl(%Q~VHz;Q&ZE*K;cz zQ+oEA+0(cuT)@`d=QdneXNj7~cQVVJ2EkvS$W%|CmN?@)FWW%;H>^8 zg&m=dj_m@hFRqMig4oxmP1!>RX}nFjpM4+i>UkzLuatq8%WS+^!VKtP?-5Hl@yl8q(#J-YxIj z^m!n*<(tyHH3y$`de{;}$5()jl;1Dh!-$73<4;V7W_JYFK?oG@>6w)eM;^|8<+u9< ztDzrJ&TB8P770vtv|yVVf5O=di1capf~N%@4R4Alw+xZB3EcSJHxV)^I*tMJt)p*| zv&)$8bnC!CcF0-KBi8G7f|#=h!R@tBll2B@il<|ArSO~b$5_=Xk?|6N){A%PHvj1V z;R_F#kkCb%Hop+m6a_~d|xxH!Z~2|8SIl--T8<;Xs_Rd$Xt~PJzj5* zvcNiZpQTEJ{!&7&y3rUA@XWG<;>WJmJkam4ag0yQ_vTf)Oa2i}jsG6AF9x*`#u2Km zT>1j_oc{2GpYMw2w!dL(-|gYUFzW-XTbwqn4LiWxoK17{Gbe6+;8X&DrCW3!lPm;5 zdiznMPX9nhA#&*y-!4CubfK&w8$^sxg}X-)x5fNwX1lYf_G)>l6+A^Kwb#_S7t+B3 zQx%p6$9StwZm?+(lh9n}miAozLZU zP^Kz%lw0`e7ETV3_Bt|POWnj@_U=LO1bd>Hc00KKY&$lJHnGnz(mmIc0vP8@J zuXjt;Jm_HhwmgwS#~(3;fl|wN4MI3dp$O0gNWnm|_?2~CnPVrj)1JAZOODXoP{7Px z1t$7=)LY2H^riZUtH@vEgOl2249O=g)~n_Z%t8r@r)spVcr)k-1^)ayhu5ZizC&hK z^LAk=>3_E8@GRU$BFCq_lPPTcuk5%86~W8|H~9_)%{DPW`C6hs$G_trJW_^^5T;vR zdHK=tKeDxXT$gZZL6MIC0)%-S7Yvx4z3a?h!}GrwBZmO4iq!V@pZE_}sHcWwh4~MQ z7k@i2{R2mAq`Q!s2`G&J>!j`YhblC@_5%JVh7WLeHK))4!v8#}P5P%GyQ0s~_&>x$ z8_tyU*G{GX^W^krIF|T-2{%abL)C2_JWU??-1RR`7+Hbigp{8#Y%j8}xAX}DQ5#fR z+YR?Nx&H2b==Wo#^860Pf@%QAnM0zF8rkceZH*xJLIKm99`RO9j2$Ip+fmk8J+*7 zxo9}B0Im<3F+nt?_clX`nE&Eox$?!QbyobgRvBdtO=R|M1W|=NFm+$9H^h96tVV0u z^oaGscF7p7abIMR#oeye-yoe^U6p0-?~{p){pK(RUYtpwsLNuiX%BRC#t{v;L~6O`zI1NoN@u)N=h2AQbQgRbDx z+2aA}{oBW5(G>WhKEb95Qa60s2X8GNgJiRQx+c5Yrg7KGD^6r296Z2&5}c;t$qgZb z?*){xjAlyOqzBrW9BGFgz2hDyoE^u!4+VdrSHw!}g^C$5PqJy^%z089z+oa-u$SFS&EOG|u#N%4E>f=O_Gyxok8=g*M0icg^r3bF$5)Ngqe*@K_C0=1Vig29wD(%FMleIv{O$tp3~xEp@`= z4FDj=tB=da9?Ou@CP%!V)gC_-8O<1zbjukeVU|YbUXfwo5)jx!prp0^RkR$uQOJ5E zpFlq!TW7fvd{WX*L5& z2-;Z54>y9zsF`g^8#XVIM~R#x7*cx&)Cm<*w5s=e9yC-s*3WYY zYfoywbiFMPwZA`PMfJCB?eKl3hXRWCKT=wDgI1PhhLanuBCZG>n|@9eJ_S@=N`9cy zpfT_!n20n1P(M>D`sR1j_hRtpL&;JerjMu(+s~%aE)yAywR1Kk5sE#OO;qxyAp?>-hes;-*z}xwG zkHVC!x)6_F;9b#0a}&zrh!i9Kr}yX(F9P-gXZhido*P-b�-Qw}*8vDd=s9fPqhPG8-vfBanx6H4Yiv_+252d7W9vN(IgpxT&NUj^ptUa+yyYmy=Y`R7d9Aar zhO|=7m2JJ_YUGiVs5$#T%Dyx&!7nUn<*9T}i=quiJsqLA@#(|CgB$JpL;Iru05xiy zuk?P+F+Xcoeg$|+G^bH91nUFQ+INhn7T-|@2PwxlaE{*-0$yi1M;37wD(bI|NQHC? zsFR?Qhk(&&sF~s^Zdyfrbp0qiA-QF2ic9O==lY#n5$fcct<;`%UVUv zNj<6Cl}5R3Df*F%|1qoFl3l$=ZNG0|84i|n$)gbw4Q06y3uQo;2-S(%azHjh_6kR( zwacKUtm#m!0T-9ENqwrNexXPm)^SmD5Q=Qv7Rquq*{>wiCD5@Lrc49!Y58X^4<*;K(u{J|vzp`pDxOYQlK9Np7KdpPArp}cKl=8E0YZOa zU(nDt#m1U;9a%JO>P`_Vb zHw(-Br~0M{F(~ck%Zn^Y)cEA-Aj0Q4mb>KH%{*_ijPus}>dsN*GegMa0m6=da_-*?@b6euIb5eLKY4PjI2xi`-XJJ}vk)nzWXljd_*kU?4}}tJ8;u zOuA6LCTA-#FVMQ{rt*LL(Tkid=Ij@;gHCZ-@TX$@u1m=l$1V|9ojC)4qp#_d@eSg! zVGTbE%gnRizBvK4#!?Yrptz@3p0MVq?sNA2a)q(jm;fW8l7YY2lh)v*cV z0CrqCavgJV##>x1zGDC`9(J$S&CPrlk-Kt0$Xg0uGW&zI>A$l#F0-Dj$yM+GIi?M_7Zf> ziSmCL#cKJP)z@N!8~QqsF@z4uzg|26r6&U4b}*IA(^h~Udr!^nJ*R}C?QlYqzoq4r z(8xK*(bR9I@pu7U7_?Ct)Iy`Q`RYQOmr+HmsQl5>#xgA6@9*s#gHai-#-(M zpe(94>Z^wQGwBr+MEJ$|xB&|4F0mxaC;GY>gc3I}oZp+Ch<;16EA-P}R^BVIc0;%i zpU)Avz&9qKT0`>tckAB7I|Kc+5qm>VmJLsSio(aj0mchabPB`aEp~d7=kUg?uzJ!% zJjncIGA;cRTI#>8{NwX5LbHfIyQ&xa&VOF_Ph%HFa-bSlHPprb^V|H>g7~RS5Ph)T zcE9~U?}tAa#YP|gkYV>zq3{0w$^Y&) z74bPw>`o3|dGPIEBPC9 z7E~!;0L(ui>sRkBj?Hz2bUaHN>%jTpVp0P>?Z80!h^Re`O#JY{Mq;@ay8M~`4kbM z)$-(gsU!HbU0*u(OszGqD#O|o!?iVag(Qz$U9_w_Hk+q!$2g$1%Qn>`a!4-*Glw_bJpn_xwg#Yx1{~`SWz6Sy&^SHl^@?71M?Q7 z_1w=qqs;9phCi^i?)g(=SiFT@Zt~0MTDfTY7O+ReE{{DueozUvd^K^B^P~M3FjmFZ z`RMF!T{R%ZAu6Ox`e@PrncX~cEv=6eAvT9H(6{cJ_@@oV zImt_G=c0>Y;bS71Pd#rlC^UZy%fu1Vi;&FlL)33KNoV3bnL1&BMg>iQ6c$uK=KBDT zU|D>7VaR*_&!GTMh-->lUt4%%@i^s@E~x=S&1lH}`UM^cbbOwbT&{{QK8?90lqi&^ zwB9nsd^6BMG@>_SrS+MpGDa8~}8L}ih?3p zD!bhSooOD9vB2+o{hm@4g&2df{e)N6x-B7+$d;(8jPH@R=FsQR%P#wxi#~okaa|}Z ze@VH}a+`bI8|Je0@Op%FAq8TgzxV4ETR+p)!aSns^ik`%mqP0l^D;_lcCp4&G&@B} z0C$XsJj3S*VEM3SL+9+yyLzrd;>H@5MK3}OA2+-+ren>d2k4HUWuSXct*o%l!lMMy zpa%sJq6bR?$$%CWRKD_NBY%c%BdNb^G#*PJiBSmlcd~I&#Z~oFBYgTI$nBd;V ztna2KvCI|ZOvY_gzDUJq4iRB4nJxk`V3PFYbPKRAr&4U|(+_?haw0`uBHiJ76NQvB zgA_Io3Bku}FXXK#*Yx2k*k%;zqdnwiIIx{bSwn@g7jpZKUUIh}MiF7SP4}Z@rdx-B zs!ICWef7nr;mA7{fNuW~wkmDM{nCy*e}Sj~roLqPyWua0f{!@KoZ;%_q?ILDb2{y% ztn2EbCCM-xWTqSUN_6l@W#HEw#M}(ja@(@u6nEE~7s%tRR~;@|?Z2m|*_Rt1=aN6< zCkwqn-vVSd0XS+u*VwF6S#7%dtcnHZ-YHx$%OG!sq9Yd3@3a_wmi3Zyq+(l8Yu5z( z;9x!ABi{WK4B!>Ey&LGB*e%@dUUFir+ z0sxgy9!ftk`HM^{UfWkBlz}f*moWqi`uH*ENLnG(l_Y@)xP7ZdTl5}I3xL_w?`^!U z)Q6=48ORw=h4C{i#$;=vLR`MhU((Zq2-k{cd_1HAtyVpF5>c`|#ys&n=_$y`+vMD> zKiG$_}w4JOK|IJPUx9j&qpBk#zM;r;wX>Sfa5R_jFJbm6io(ge@?k%jn)l8+JH1p=#& zR?Q@}XPw96jzpl1j;H;5GGhTxTR~;ADRAGNIPjTcI!|RlX0Z<;?Io1Fz&~q$>ctgc zah{fBz_PWlBbE<$N7s`EF0E~baTQx3SC7f~zEmYfQ|wF`=VkdJxiSdqh`R)RTiyV^ zK7O$m^@w0}_0)H~!?zV!<0z9EF003z4a*fbX4yy;VrxHnGiUqefg?QMw8Vdx%Wn-_ z=Xf;Us`=F2izSW{EvX(zB*TyhVQH)p?!C7AUISp=XdoZBh4_)at-5t@D|MOl9tDcE zN`7MwsV=>$74{@P-gqeXym35avHozcQ;cXHKJSPzChRWTGTxk`HRFJGXII)8o54`BS$pUCSCAj&NfrT7>q~|0daR zJt_R^d?5gnC^DJAL9NPAWFg-={=z31AE&9_2dwiRmm)G{iMi|6Z9vw@VKA^s^=-|O zT-l7#a@EJ;kBY4_TH~9<3xwoFZ7-|yGBsXbQ}G5c`LyAs@R**^U`ZI-!U*;oDl3Ou z2%L{z+=iLn{D!{L>%G&s`1ueKds#6>(mqDmGkI#mEy{srmKziZa@5L zOE<+#b+2$f_2hDZ%$erUwZf=E5s~U5z`$JXeKuXD!cx^OI51_QdLD~xP?n;-mNLK6 ztV1EiFY*y2#1VTMa~5WunB^@bWr4hxmJz(D?(;!ALP%98-HWYAP&RaRp zksH_$q9!1^(z4K)*B+3LJQ^ROFVUVm%gx1@d<)Pq-EY{V++Wv3+3YhOYGAAIR>5L# zn#jT#RwTE1E|pHRbl;)A4mzZY2`OGuHgD50j5i`P-kT@Bky%qDDBV#lyd%c0M)C_P z)=+Qigtnd_4Rox1;f{DZzzh`BEtP6EpG#xutu?n1opHb8ef3j)ZHg7DK>-SfK_nPQ*2bIwR_NlHyv6Q++_n z$>J+KA$h#tC(1M)EN}6lYHZMv=pgkx(NJ@j-zFN{%Dz)+QRe6L<@BN+RE4ps-2Q_* z!{ZLa-M<);GZVaR%)RH44`0|qBJGLncIl7fJ<3X_CTZhhZgQR|5u=BHmdxrgw*5B8 zYna+!NB2cmQgY|NJPnayLNUgq3c(c&O<5B!)~nS?HPu&^8PvH#aQ!^Zx|}(90(pEU z%F@}e?~O{g*?Dorvne_6`f4wsJDvxHa#%GzR~bx0RPUM4r4CWkrN4>``%N1k7G>Te zqq2vug#tU!5P3XNf-atwRIwI12$n;orj|!ewm5uUvxTRO>BUKDD#;LCj=yTB@$|E3 z45WT!R_7!C!4AtCU&W4lV|m6juw%YQ`QsC-Z#>&!PMfORD)m$SlJz;AHwmt?$jGMF zm6AYeodQF>a+2OwsN$S6He^@znHdF9Y)zo%-Ioi{Y0T86^E(-!Qa11cyw#2wc*J@{ za8B{wyEdx#Hm_o$Q`0!nF%?G#!m~UR&FtC+R%xA`>Hk>dKg=q?zHN45lu7?l-K6qk z=@~=e5a4Pi;b{vW)Dd*s;}>Ic-G4j6VE<@7Wlw@X&HGWlm>$|J=-Ks9Kh6HZEi zsK6Uz=JsQf=oIHyY7<%=vaab~!)WT|{f{`TYWP^Bl1aBSUDL~sap`f+NwZOJd%VHa z@z6n$0JtBpt~HgT01KV&XAorUd5}qC6G=xPvl#uPpTN}e${foY%NUYTs$z4P@P%SQc_S1}ds{Lm)(X=W}*#Rq1m5E(r$%4f7Yaysk~1T95Lu zlw$ArQ{VXels88w+8tyclu}W?%Q?wZ;C>mB258x`_qn)=@-}Ru<(%1amU3)!OU8gH z%>^sI#lGzysDDn!1mf;F+(AhvTW0VlKV3NS zd_4=rdYbm!CeN!!W}0DJm;PbjYqGrAZL91e&c?Gun(zu6K$qA3uE>*7*=0w*Df{Fj zJmQj1<_@Sye4dLPOBnT-`3~|uQT`Sa zW34rQC+?A|0TF{;Ib;5|*LC>4S75J)jAQy->hUUnw2#MfDZ^ItJPey5tZ-#Z`LUZ7Wfc>qu#txFURwi()Q48085*}t z6-5e1|^Yi3`+nG zhx2}Tg{PAJ*E^y3Xs~5{QBs4TBC!>1bb907r*3x$!}I*zu7c;Lt1&7lKK!!sWf*0b z8awAXOJ6K?R1MPL=fG~~k^e&B@-Y*(IhY)7@*Fsw5X_cMe0*6(2Xrlm)HgZWrJKFA z#@WjLXvpRoz)WM7ec; zUjHIL}nQos-zB74EKB^I$(hRBir4jQz#_0SnC zTNN4qq&?UnXsnKTZLU8Lcu1^&8>`M%9;I?l9_moJ@hrL_GILy7+Mno>+paRND(%Oq8-Q`DW~?&X%)=>vx3mvH67S zhN6qPQbA~+^@zl4|B>?TLJ_K@mozFTE3v>C#>zfuQ@SuvPQ|oWGF0CYY&eDc$l_QO z9B!FB@_sC5Guf=idwXs@BuCM`uAt_85)c^cUGe|(N*d~j0O!sI^j86B!y~==Gwre zk3~*nD6>ivMRna>uD%y=nN7|HW6ojM73454>5_y82-`EYP!Tdj*|+*y^Wd_nfyKjJ zT&mh9XGPJc;xG-xkrHpCxb{OPDlDkiNVgih-R?46l1UG!ERtW$=^?OA{57AlR8ANp z&(h1(ayKi<9||w;EwWK)diTj1b+jIMG`l2mi5-g`;PVL$+C{K`ONs|+EDt8;fW9{@ zY|Jbm(RtE*GPb>?{a%qq;}#jD@HpFwNS>6^MwB#LPRn-1b2DW4w$8qR=B=<-8`r2_ zp}!2t_t3mLP9jc1gqG@gwFjtCGL@IuxH{Q)sytDnRL>Tiw$pC>@U2_B4J<Io2R1&_H?fu5^%wx4>yd~ zzw#NaA*)-J^+v$W;jl+Kn>79}N#zHDUJHOzv)*5$QZX)FM%zYHoD^cLTkCYw zEhdy$;tqh^Ddt5+z?2^%V?S=@X>>UvASkS{fF?Va+uFaU>146O+F~Yy$K0n9*opgs z2tzmVn~AK6?2spOwJZ66od#b2=<|{jUqItb*tro^T3PuSx`D#v48n2PHNn*3X(?CMI!=EVtn$PC_M>@jes22U-zfl zwcu~+8?Yzbb-F=JtoXj*Gmwh)vI$6@CU7f72z)%0bjn>#ijaU`I@vm+^VDdG{VEu~ z$WAx#aS*T$^+xfd<#fVrN6yDsAU{7iyl^d-7`kN__BeTms<%lKDb)e8d3BgJRpSW5 z-6}&&gB`R`w|X14!k^3UIl_17gN&KA`lxyAYt~yMKN?`)U`46DB;K{2ySc74(dh7b zm%+ZiTB&wK$UxpJWP98qs zV=sF=Tr!Q~JkX=*=eXu@&FCrm8d5!o-FkbD zjy%_Jj(z7Y@srqI@l5%IFHJVmDIkhV)tfySK?s1(h(z5Pa+P{1e?wI&4Da1wjyRH!&ndFuU7LTKE-|Y36I&{dttPBmP zl3t-`t7@P~-?bO-6C&RWI_-`6IAWC)Ap_iEvYcM3;@L7zK6OYIi2#pCZt$Zw9HiO> z3==k|WD&D|@@i9Uizv?|X9l*}rRqn$rIPib5U$db1OpQm<%yH8XPcO?iTY(4Y|40J zuTHLBLIA1`?9pBkW8USenePy1lC+3k@Dm8nz5U@WYsb`6{6Rm2Ano@V?=C1{u)hHN z!=R*?mE-QR0X&WXQ?}b^^IzepEQgRbb*2HHbNs_>!Ns^04jOMa&%cpq2w;MaEiKKt zBQF#FD?Z`GBV(NQLs|An|Eyv)A}$3Q{kSdONr9_u?Pk*0nFw>!eDwHpXHC>|EGn1; zee>JRB4^5;|4B>=_=t8x1;!NERo&=0nFxXxcS&0y%I?us>F8ZDPC9x}dV}*{(h=m3 z%B5zlLHS>a3#WB?KVZYDTpfkYcmAbvQT$Q4-q4J9{I`6Z)}`M;`16@A`bpV;TmPRQ z-+O^mxjy0th`UW zf8^4spRd&qSDPy>Y7R0R&Fcv2C;4o7yn$%+z_^_n;lWGNR4+)P$h( z(WH=x94CWS0!-;aqXf53O6b)oM$OiIi@}upsc^WN&b8dDs;T~s5oX?z2LmH-M$P}4 z<^8GGfQ@Giah_RIaBP-E-y|DP*Yx6d;2SUrXl15m=apwG!Q(1pxb%~&pe$GVCcSVnqnd#kg7xoF}iu1*&!qQ)kt3`7pX)ihwI#<-P z4rfol-d#yhbzWrU74>%7EI?60JPk3viE1K<*Yw69hD~AG5sjJZ9p0x<=Li+#I(0R- zc9e2?Q2>eOYjNIA-Ym5^QUji;#&XqTD&xa#k3v2x8odo_tO^|d&Sul`N!z+e&V#)n ztiyEnS^FbehDU2*9#jx(%j66mC6mBoIgibOu8$_7s8AzUw3x}y3Kp01ldZa9$)H?2 zzL46vY%5>&+6r1Rh7n__)#*16{p#nxL=h|m#N_tNbuE8d`z6NOn~xhGzK`|>)Q0ND%D zH!w&dR8@fNBljv*TkNSy+K1@q1(8mG$62{>Xm2487hc>x%j_D@<@Ec;uUaUruH&zx zrWXQLRK($`?5CMw%p@JpG{+f!Z_@whJd!z_0XGXPGbO?TCJ!z%&G^rW-!voVEg!K=D;D9Oy%-y zICa?R&9}MeZxR+Wvc!#g3;DrRO~$XTN8V`nZJ0e zM!|9_c^eavAHF{Qu@|SRo-dgMGs1k~+wy!PVGzE_mMvwWjp^v7duwb=YMn;s1ZO@M z-~ituBQ{cc14sGM&ow%Mg(T}csaR;Wr+XFXUM$rriYw3Kg4j%wp~MV}nz7{it?AC+ z_rHeWHq-GN-kHBlIbT~K@<((JtJqvTmdDyCn)_HWwfu~HcES>&?P~3rV`AoFdJ_>2 zHng_NI*lWsuFn9koO^T(E=OtKulw>bD--tvzBv*2fr6b}x6F?uA7P~%7)N_dIDk}6 zlVo6uuPV?5?!u9fec+TQCX(NQN5TC5>r}bOns{!wPJ-v24(xXQ;c*SMQ>MiEM(7AD zMrdm;4KWr3i4KnJdi|0^SJ5%*H+bP*?YSoZ5DS!v!DD$sQ?J+F-ol5!@%D6%nbx^} zCI13Vw&zd~6riVsWp1dF?b?}fK1iR54gh(@#GW%!W@GX^BAwMv-?&;OJbSfnmgtMf z=-T~a&GvSBL=7-@eFac;rsNW{mQW;!m1~2kPZ4;HK81(Ahp6j4oe}ZB3vnf(KVft! zf&eNxdeI?Cdfl)d28leGU*aaBX7)wZ-x`@$rIRF~>*u)ZFpIi!6r<-1Z?KkC-|qMG zRt)H6QYVn4mw73pKZAmDQ&E8E!=A(6fBDvuRkE~Xk6{-Jk0ro|S!Neo>(Hx|uFQsY zf=*`br@?p+0K{ZekDvxk!nLl(0d}Kf>|YjWHimo-#~@e1txTDoJ-KN~N!|h8SukE=C<2#xc5m)j*gT$6{`E2B?Zw(WYBRzL?tQuDs`a zwh34b-jHH#D)*?k{O}bl{%jtmG9b2YP9!E&wPFByK6DdUu}^DasH+^i*(=L{{-LGY z9~bY~5&?CsxWUFsk%V-Cot0*r=%A!fNK5ftv0N^TThBRRwBTP(Vp z6}VuaOfa0p2)9>V;;&i+-NeJ6HHMiGRGWiEMUayanO(TT(VN!-n5o$7>;nomCfc%e zy9azj=ud9EsYdbUTYZ^|u1;{`J~@h1K&bPmQ0;2mPo$U~p})CkkZ3%N3^>!jk;FsZH)8yz>)?0{xMQspOL@k; zq^R2~^1WzI#6D&~hghREBNr9#JoR=E4vML%79oxYob7(QES`{J!;6O-RFL~g(l9+J zt;bIp3c*D-4s4L_l$P9WW%Y@5?ziXg;e8}Q>@+?HMzyX}Yhk!H1$BXkN!zFV$zixd z{9$mjHZ&ZZ1xR*OAk9k45A8IivKd`Mu9rMzef7Pk*1|PYf&>NNAZ8^m8FSHh@?fcU z&wg%6y#PahF&M0Qh7N7X4hq>faZtgGRvFko2(?zsw0Qz9sxN z=i!oPpSHxp8$OMWlu>qT2sdVdIwC1>Py~gNo>+43o0P*{%5$Da-Updcnffq+;3eY_ zrigRk&t#*!GZM>s$di0;JBSDNI1OGY@8b%Kl7cy-(hEp{LC-~4#{fO+-ze1#x&>27 z?9%L-K=CIBSo6hp7vEf3nsEkk=-20B!(gTOoP809*fJ5`VnJ%{r!fYz^AZo6GWSqP zm`M4L2na{!4A4ZI0T<9!qMi3Zi84&`lsau!t+ ze;%H1){h!~34qm>v@Qo7cx$`j;CyUQcq0;K;Lv>GWncW*UoPoIrqA^TWnPPR3cU25 zJx$6_ksl*iHqtGly2%z?sUSEJVo)UN?N9x6JD@&ex#auKV9c1%IMG@!CR{lMlscuP zmN;BHIo&F^@lt4p0nH*q{9Bc+Bgy#mRpRX-8~yui8s)Ys@zw4cah}pr&v?30tA`pa zQFw{Z3x2)e@inO!VJ*wpu0jMz-fVk|<}GXT*qp4hJ?o82E2Xc!tU}3=dVy+`T3627 zrS5w)!iZjE4 z!RemuY2PsgQS`FKLU^GAiCz99;U6?!lG4A79B=rM1;qK~$vK9k3d48_32I;;aF+lo0d-;*L{_Kslv;xo5;YdAeWo4y zvaq9Xq_Tp`@pe@Fsyl;i6_xoEjuz$W#^IKU_Frh0n&H7VZAbdII|hNBpjTen4Mt*4 zr&&K3tP|M8%U+Jw3|-pW>c6Ow|IHRa)o8t;xHP5aMzU&rAbIKR8MN~nu6Iv~cPI?= zSkcfkru8nx6saXj9hv2W0!&C*O`!*lr={DQAxWy);mD6=nhz~?^R zgT7lrYV(P^%jBt{{Rurd!8cSdc}>)emeD(*g(Ir-&~NN9 zgQtziHk;s7h>43jV)>V8p!xY@QIb6ozJ$z>Z+NM{;e>B-f`h2uEWLI>C+rg}!3cPQy8Xs8w}!`j}o4q zVC9I*ShtsdsH;`>f3X(m%~1BWCLgy0`&Yg7YilcXd%3pwaEAx`tj^_2>pAAx`&cr% zYkY_UgMelj=0nfBZOHvR6R;LLwv|0s@3BPukWjBYsmJw+4?Fv}d{xA(7QO}~xb{d@ zkdNmH=c`rs<=9E$e5EoC{8bd^Lq}5(&x*a_x8)S0@x*VA;OZnvXAh!R^sxomh!IZfO1rZJ8$M?-iSu&a2w_KtP4X4;bN>! zAw%!e-U*=L#qW_4(ZDcsoSs5ksobG7uo%8Fy&HJ^bJ}szdn!=n%$ksJhuBagLX=RY!b{y7+j}K;@@DkLSS0_Rx4`*8x z+=`O4CN}`rb4#F;)pa;^M`D8<#+pduUr4;CtCLGyY~Mo+J@M#P)7`4LyY&+-J~m?h z%lbED(G@y1z`M-{{FP>}%787H4%5pDq{xMriBaw_;P%oPA`o4s71WB4pXu_l{FK6n z-98Po7r?_u0_unkM8`Y!fb9tgbLmpgR&u>$UHQKfa_R&D{S|yHrD`t^Z>_MGa%#`0 z+QMl5wq!HIn8F4nF=5?85^3;n_z8@yb0qDLQ z9K2vm%USATETmyZua5Vq)hEUAg)7tIH^w5Epdd}o?#v-1Cv^kkaNhIWojEU0p@Q9X zv?_T2zR+Egc1t9U5&y+wO`S={P&q`T7%;g+(3HPzsO=SFCg3mAt*nG-F01x zB>pY95$qkIrM_HB7P4{CMH>rv8Ng21M3Nk^Pbpx=coD z*=)A37OX!_*o6>}eImhaN17bHq;2>^7hXGq57G28e%4cqez9^-awhJl?)kQ9ya5di z)1&C?$V)8YS$M*p#mZnw5n;}6A;Tm`E~ZtPInK|fv$y3UG$@zGA^oJGt_ggp6P_t8 z@6>_3+sF&vhY5hCDpruy%Y5+u*G8)I`zsiR_6J(5$tL^H9Xz5`NhJ#74?% zEMl-kz6si3ot8^)?Sy>zDWX(`Qi@(IGk&_!2#R&HX`vAE06lnd?n-()KJorQsks%8?Q^>Pfi51`L$g$cEOR&4I_gU3`PgMyX9HX^09m()=&BFL~tWmTxZbx-{eXb{MiR-ewCvi#N_HaoV8XhDL{Rhnb{Xp3~{?W;JQ3nC0h)~?)DniaDYj&fRvKEbbVM(4)=iLuxMg-Yh z#lgJZ^0)QXbRBvk<}%`$i>qsKG||O-d7yrvJYwb@Q6p}a{SQx{fLQx@we)D$MY?RoCveq{+mTBvBn?+*xC9* zSgAQ46%+aPr&*7IRw^S%w$N8}llh3fkuV#pb(qvaXa6zpLbUzaF(;&WK$)UlDXDA1 z?e`7wjQ_Tc%-fGQoX@>R*KU7l&ZEcZu)$5Y=Y#Md`r75x#az*A?4WwJqT<81e{gD| zRfC?jJcO__My8KfQ>LcNnXdbZO%v3}sV&58+2hje@400e)Qk%*KOWv0&n(=+QPgnA z;(05DP=0rzq9;Hcx<`hzbv;w01qC=QRAxQ%Uzo*RNMImrCBh2aoCl>Mpo*@5Q?An? zBrzr|Rlb7>vXT8!0qj}~aj_N^ce#C<`Xgtr?%INsqnE{}!9Hc`X`rO2_SRP9Lw*>g zxxtxvMwu|TsC1U4B>RQhl*wdA^!$_hi~=j#nlY-P+ri2#4B6T4(Na$3z_F(;j0YH4T?DlA(ta7wKw3PI-LG=1dAk@+)US0qQN7}DqVq0pjqIls*QnAAO^r2L$W zO@T7@j+DYro|QN7S=_$!2gWz&;g8cP^IK$%k)?LE^M`A~8S%=3w}~4kboR7Ks2L6~ zakravuU!ZfKk~3!vz&bO&J{rB8)NUww-*DHYHI(<$VEWDFtirN<8_j^>08I_(%$5` zr#UGiP1>saM+ynOfPc_gqlRZdv?r*smkjk%(vdOH`3J_6w+)z)8?(#SY}(PABh*%i zG;&zH?JUNyYO_2wPGwDg3CDZ&4B2bYkIxQw#HIFBMWbV@0d{2t6xruh?HoKQ--tni zpQ*5+`nx)xwm!<0zpFEvhBA(EqB+d&1Dc;_xaScOYoK9UvIdg{u=0{Om(dILH4O;n z!Mvo!&i$WZ?(9rkIcp2uxVGFZSDI(N$v-qwQ?5VEe?@ST4dj z!QZ&+<|PnJ^BlRfPQ8`Bzz z=y@$s9g`Zd(D0iVb60qrcCrw#bXqDuZFmj1iATHySW4R8{TaepA2~t8bf;kjE1L_{ z!72)cI??MyAT`PQvMl7#cF3Ba{H&W<|9;?1zFvxtk}G8Tr#Z*5E)5NrM`AJpHh*@Oxtg$q_ zO&TZZ(>V1pmxMw_zi}%yyvY>{Mr#bhh9tpw2|e=tHQb~nvbwZwzpF+G_A%+0j=-2l zG;tSXa-EW=w>R@2JY3Q?w^M?*q_`_9IkB1-g8k6{RS-K_%IpaE{XY|Ey9jmpJ+(pn zu2Y^BkJX;9i!zJe=Y2j6>QHjQ+!zxuK0jG?^j~?jvDvz6d?f%y(-Lvs!Mw$@|It)meNB-Db;gtfPjSTA!WIk-yF);)Vhge zvF7bUuUDl#USx_oTw1*UKF54|aLy_f^k};bZl%z+_SY$-n9k{KYMF0&=i@_V-2ra* zdO>J%tdw1e+94u~zTyc1%+fGuwo#1A?5>TD*a#>fAPx@Y zYS0Lj&Ve_lpQ)Zm!x$v@(Pve6R`Ri>Vxd~GTK$EuUtXZH1m0}65{q{wniPL$W0-1L z{n2H%QyEKm7o54}N^{6<&FC0tg}m3R8GmWEN-BDeh;%ADusSs(M7`m+@a1?cnf8dpD?#mEx_ z6%{m2r;I;5=?7W!d9DPdm^5^Ygj)Vwc;*&N;E)`T*=U6-{w(!$vF+?CebZ^FQ%73Z zSL^#B71aa(OYkqEzd>~PfT%cNG}zwctv&!97~6EjUP~<8!x#4Ln0S)G6V=1w@# zd_n^AvPR{_2Abt&jGw>}vYqRNl!N#7Zt$03nS_LEicb=kE3=uxK4|y#J1esyzjaS zOZG1N`XP5pIY7W{Yk19}yfE=gV(|EdVfW)Qv%JT0oxLL~0!XK4>l zaT@A?0k|#VY)Df(5%nC%hCxyr40&@~`?yAYime#TXdqt7*o?l(q_VOK0uGdqrver2 zl&k|S3n+WNR_-^hW}NllnOcm%*SgiduqD}*Tl2fT{drW8+M2Nw*SM?5^yw#QA=t6< z@0g6{EZlvUKJIyk{KzuO0oI!n4;eDiIP1BmlZxO*Cp{2lK+F3%P7iPX7CYeR9xPce zy9=p<-^MPf$_Dg+7lP01Tg$2wLBo_j1Rh3ptbB~N@Z(S7ADf$z;P_?8q(R@>n!-1p z+u}t$?V;QTA9b?$RnwvqDoV5To4cI9c#`*W(ks-EU16wfZvTWaCh|3pbA(S zanzx$A3tU1s`rl}Sr;jE5Q>@md$B)EZ8o6kCerKTN&ovC`j1s}8~@>9ZjRjN`Sh#S zI4dt>nVn_$Etb8jkq(~;VSj9Iz!2@ff?n+Z1ij3_oqDcX{+)kP!-+qH{>vk&#TouU zn_qC_t7RGe$?PT&kh|&*X=wk61FEsn;lFD6^Wk#H>G2XX{~q)I*kxQiIKw|!tA;1x zYO4Uag2JxPY=Rx|6p4x2(AK+l{oV z-Q0}tPp%;scs4hbu0~X5MS!WdZPb%MH(QPoxy|-_sd0K!@c%C}ALqSl+(@dy1=&Ey zaxA77(9iUgU23;LnC;q3m_0fc7PZrk1^^#nq z&1@K@8b4n46D?eM0H-1lT%R>ds_LJ4HrsT#r5V@qubLFyy8)D}x<-JZ&gY~YxjJCd zxBF*&5RA#%#8(GG@HuYCsPjMk>)CV~ZX$#n zC)(gv#EJMZ&48dDY!*5dVT-QH*2`Hly864v%wnWU(-+g#1jIU|x%XJUylbe+PdTEh7o(S)F(CZ9s zF>Nw(?qLmE2=N#!DIi@snsM}u-}R6{w46yePJ_pY>ciZXedq8ljWI%6|BMLE=ZtFB z26RKsq`XEnC&g(d_LaY!%>+Z}UxfA1FGdDSe>Plu_FlPMoI7`ZCurZ2`;*S8_N;o@ z-t#H3wc@K&*^mrZ7fp6ea&KWOBNtVfDDe!$JZ{ z3&Qd$lISJukMZV;<*KRO9{A`Ars~v4gT-mQEb|QAoz$^qKepJ+W}y20>DTeD|E?fR zMot&GNj>*3qVVpPcb!1EbXP_2;4Jjz0s9)ZhiMnq$LU$O`7HGKw^0N4t(4eT`tgUC zp;8+^QLTkSD~?BDEC3wK{om^F?qD85N0D%E1S%t=^RA3C77|I-nk| z@6F$fOdh^>{#G`1Ve{VqUei1K_pW?lJIr8>IIDl{Q;Ij_iW)^Z8Rn>0liH-})?y+A z#5cb%r8&2;dkS+0XvBe9cIZ#ir)mzjtZ1Vc^)imkWu^d=XLKw*8kao{rEh>aj{k~w z3zpZvAYL~1UHsNTW_>bNUtJK`$G3am>$C?lO0YHsu`Fda5_3tMlRRpPEJS40tPM|a z7Mtrj4Bz@^iR#}VlF|3X{R+Uy{X&4MbBX#J)Pv8x{`SIvqLSih*&C) z5r)}YDK_bvU9HF;N|>DNd1#2+(G??K}T3_v2|_ocQ2R- zhR1z-Hqd%y_ie^!eKgN-C&4#$>2ECZ*C9;rolq*R3o_=3_-AP2n;%o)&b;ob&5EmC z1XINA)ZN{*S23f0#2*u4dCF<>>VUsw#dWZ;ap?cX3HMJ~_a76&VGC~c&+zHrV8Rsy z;^&vU+M^o(2_>#`))BheDSry0r%GV@-vq?{!^Q7yaxA@RauZtHZLMQJgMGbG*}JAz|&xLtcsv&w%}rl#S|*7T{f)A)E) z0c7XsLqw|;smc8@sur)~n1(T$)7ioy@i@2Y0%BD!p>i+7HOpf_Tk_fFiy2gjiu#%k zqI|0MIM4u>Ks!}Gm>(=8a@e{pymz5FtxTP)wV#rfBjhw&&7@`4YHY5bl&^}zb`$fX1mIp-(d+5PEODV{txKYix!g}qh$Or46fIbOBe3x+CX+Hd)0H8NL^e! zvqRAueIlKweSA@oM5QyMd8!*OT{IB7dxtGrxl%fqC>0Fg<7DZ8Gd0eQ z+nXZV%4zXSITn(9<*IDiWM1Rh9F$aZak7s{?JP!FV#5#LiP=h?`l1BJzrQ54jZ)n} zu(Cz(%|zeoV^39Dv3MGe-Z$krcdVMz`htF^%R15_4rE1oFISF!ad-g~0^Mdaa*cL~ z<)YbqGIr@**)Avcy3}dLDP@I`Z>wa>gsfz+bR*296^>{GeEkB;;_yf*cvkL#h3m4jig$a`(X2NZ+qYE2bD zZaQfKC*;JZ#fqB!n|O4Hzm5T3ia!+nsmSB5QYpB|4s#}Xgi182={^JAik|1n@$Z`< zn!F$MK23=Fv?N(VW-i}jquliJ0WUK3=Y?J1Uhb!TlRwp=pb{yiTTXpeudCsE3Fhg~UHUB~i?+r&Y(4&BGFN=2AwW%fl#(LYq z=2M;v`#Oy@&3LAt6asgC;4QIPj_>rA9gU~v?&%LY5P+PVzLcM^)!6iOQcv)G0l5eqjd2sJ>7yJS+W2~(K9gES!r&UefwZT47uW@?SxS^@Lgj1i&v4I3qU&hgT^-Z- zeWjFhuSoG^$1m5_n!|136kdRs30s#-V)>n2C9n;LJo!y6R15q?p_<0U4yL?rBVe(h{sJRHOFyeh)I{F1@^!Yki>Gg8H~&0~#p zO2QvAAyp+F7t}+xV#6ukBXPXGDVhS&YuN9(C&t%e+$w0%@It5Iubc{A`afzGy38Ql zMCQ@m_W<(ScvP_*cz1P8Qy56!x^WSB68e+jZJ~RTTud=1x}VHWK$=LR7P21KY>$$j z=EbRiQ}7E75`Gj0e^IU_J$$&s1=IQ}7O1$YOB zV%*7C2^B*hLzHOmv;MjMC;Nt2t}Bvnn(`DPJ)~!DPcJdg)0X_vDZ@SCsQoCKk}3K0 zBFKfqCF4FuoHWo-qi!s#Ug#>rtjhN`AenBQ~QCPYxO+}>vGgnDUth0qa3er#CY~0f8O`4n9|CZ`K<8_Zm*K%qiD`&&i zrO}3;qK$X>E!{fEWHriHalc)C;^i;?gk@dBj24f3*;^OE3y8lboz9KI*=MHnNHIl( zfql41`f4&glTmv6rV@KS0v3Y`pTM4v*DB zl<)jnmOocAm2pc)ID!2PsdYjo6tUW z)Yr6WcULhcgYsDCzq}RiZTxo)-mo=QRpG$#6t$xq>wN*tdaZyln%;o<5n_gQLB(=s zVqSSXA2ei;{`7-kdfLNQBllPWc5}ue+vFdQvldOGfc7?6Y;?su|WlWN?`K08XI zl!@A@zZ5ygwow0MF%MjcjQT~A$22x_QAz6^BzvFg2(4WQ`89NbA^+S8?Rw^O-6YuG z!yGn2?UAqD6q7nW?-!c7F0STghU{&(i}&W*URLMfd_2f>I2mM8p5si*mWlYORUJ|B z8$;a|BiLt*ZDfpz4&>CbjYBQ2bR=R<=1|b)Z`u` z)@evgsnrNyAC^HBsi%wzV6xt4yL$l#NFLi4Ew0^{JGHfZFZua(=UIGR+PJsC^`m$1 z8DR7PQKwe03O_AW^UjXT+0w-Bn}M8ChtX{DZPBXhu?nwu>gclChYZxb%x{93-zc*_ zuEdM+j$1+bX5-m)_ZhS&TI(6p%E|!8IgfYMIzhKrp^M5mD9Y=T{RkmkP=5jn!5w4rB={G>>Bm)6A040_j|H_ z%!aZ9;zhvFOLk$3KvhRu%)qmYO5-LFV8Q}oLL1sw0Zf?LyO5Tm!-PQ0P)H0eQ1OqU zK1r^Dr8nMnckMDimZc_mFk7_dCkpDijMug4eQAvOg0dv#^5`Z|{2C{bc8=|3Ssknp zVDcylYPqv?TP9Fo>DsGA%$LQ<(~ZN(>!~oXLV%;TIiAc4Fw<2oYTs8Jpe0! zYP~O=)0TCk_X9h2G5$K81h38gfUYxSHjnbJx?2Ber|;_l^tX^k4z&w(K#5odIpCVX zXQASd4&IRy1*A1+6q4=}Cmt!&awt$TDxsIN0Pg!><%L~)lWK1bR%4DA72E|R( zSI8D$dE{YoRrPJOR047#o1fU0l`Q&GS&2YZOYQy4{@ksjryE4Kc5}Erg*|j$P>@bt z+frTmpx}eqQq}{fTgRe<+MLE7S(cdGk1b*kgUNKX<#pI~r4LXHHM6?TxNRJImdRPu z@9=Md4}pruu6GVKi_)NJf-&}yd(mJqfvSjjNv9u&T^t{Ek^&7TZ8=YK9K*IC5TTpqWYje2(=lIzi!9Xuhx1J)16-P15<%R+l>6#XioWg5ldg%|*-{+LthNPl&*eTw@y z%tnSSN7{;?RO3#fsd3pF{$a73=_PG)6Uh+lS7r+gj8Y}d91-svMH@v z_C_?#WNYzdcFf7Bw6i7roMJ^522@aWo|v511+Nt!ecvupN2}_GRoMaV1EJXOCdY(H zY+C0BvLeXbhRg9)QY;3=*6)#vQ62C0-$kd&One;qXF(lAd z{|5J$8VDFDmOj=dAr8^C{M2HGzgWeQXK7%>ep$0iQkNlSlHER6Q#bwJ3aw_*-fIPoar8_+n$a#Ca9nScduj(KAcmF6T z1f0KQuh#PejuF)*gp^+8IYrXAs!!-=*LPnn=m)Mw!r##kuF@dESASH4B1g)}D~u~W z0aqhap~ipVLxMkm$|q*~@l^sSfWuBR{hjJpq18AeuIiR$NFC#=1)1LH$^i!?0Z*@X hCoW5e<3_-xmq}sc*_YNw2G?+Z^0LY=N}lU~`9F1L32FcU literal 0 HcmV?d00001 diff --git a/gdi/get-data-in/get-data-in.rst b/gdi/get-data-in/get-data-in.rst index 55de84bad..5795924e8 100644 --- a/gdi/get-data-in/get-data-in.rst +++ b/gdi/get-data-in/get-data-in.rst @@ -24,7 +24,12 @@ Use Splunk Observability Cloud to achieve full-stack observability of all your d - :ref:`Splunk Log Observer Connect ` - :ref:`Splunk Synthetic Monitoring ` - Splunk Synthetic Monitoring does not have a data import component -This guide provides 4 chapters that guide you through the process of setting up each component of Splunk Observability Cloud. +This guide provides 4 chapters that guide you through the process of setting up each component of Splunk Observability Cloud. The following diagram shows the step-by-step process of setting up each Splunk Observability Cloud component: + +.. image:: /_images/gdi/gdi-onboarding-diagram.png + :width: 100% + :alt: The step-by-step process for setting up each Splunk Observability Cloud component. + .. raw:: html From acc780129cb0bd2427d62161bacbddf7e6e70563 Mon Sep 17 00:00:00 2001 From: Max Bechtold Date: Wed, 23 Apr 2025 16:03:50 -0500 Subject: [PATCH 16/16] update --- gdi/get-data-in/get-data-in.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gdi/get-data-in/get-data-in.rst b/gdi/get-data-in/get-data-in.rst index 5795924e8..8f7725cd3 100644 --- a/gdi/get-data-in/get-data-in.rst +++ b/gdi/get-data-in/get-data-in.rst @@ -27,7 +27,7 @@ Use Splunk Observability Cloud to achieve full-stack observability of all your d This guide provides 4 chapters that guide you through the process of setting up each component of Splunk Observability Cloud. The following diagram shows the step-by-step process of setting up each Splunk Observability Cloud component: .. image:: /_images/gdi/gdi-onboarding-diagram.png - :width: 100% + :width: 80% :alt: The step-by-step process for setting up each Splunk Observability Cloud component.