diff --git a/_images/gdi/aws-ts-awsmanaged1.png b/_images/gdi/aws-ts-awsmanaged1.png new file mode 100644 index 000000000..a7406adfe Binary files /dev/null and b/_images/gdi/aws-ts-awsmanaged1.png differ diff --git a/_images/gdi/aws-ts-awsmanaged2.png b/_images/gdi/aws-ts-awsmanaged2.png new file mode 100644 index 000000000..b234f204b Binary files /dev/null and b/_images/gdi/aws-ts-awsmanaged2.png differ diff --git a/_images/gdi/aws-ts-awsmanaged3.png b/_images/gdi/aws-ts-awsmanaged3.png new file mode 100644 index 000000000..b333b1bc7 Binary files /dev/null and b/_images/gdi/aws-ts-awsmanaged3.png differ diff --git a/_images/gdi/aws-ts-awsmanaged4.png b/_images/gdi/aws-ts-awsmanaged4.png new file mode 100644 index 000000000..7f36c39fc Binary files /dev/null and b/_images/gdi/aws-ts-awsmanaged4.png differ diff --git a/gdi/get-data-in/connect/aws/aws-troubleshooting.rst b/gdi/get-data-in/connect/aws/aws-troubleshooting.rst index eb3e584a4..e2283b6a6 100644 --- a/gdi/get-data-in/connect/aws/aws-troubleshooting.rst +++ b/gdi/get-data-in/connect/aws/aws-troubleshooting.rst @@ -7,7 +7,12 @@ Troubleshoot your AWS integration .. meta:: :description: Resolve AWS policy, permission conflicts, and other issues in Splunk Observability Cloud. -If you experience difficulties when connecting Splunk Observability Cloud to your Amazon Web Services (AWS) account or using the platform, read on to troubleshoot common issues. See :ref:`aws-ts-metric-streams` for issues specific to Metric Streams. +If you experience difficulties when connecting Splunk Observability Cloud to your Amazon Web Services (AWS) account or using the platform, read on to troubleshoot common issues. + +See also the following docs: + +* :ref:`aws-ts-metric-streams` for issues specific to Splunk-managed Metric Streams. +* :ref:`aws-ts-ms-aws` for issues specific to AWS-managed Metric Streams. If issues persist, you can also contact :ref:`support`. diff --git a/gdi/get-data-in/connect/aws/aws-ts-metric-streams.rst b/gdi/get-data-in/connect/aws/aws-ts-metric-streams.rst index 87495a7b2..b2f682e67 100644 --- a/gdi/get-data-in/connect/aws/aws-ts-metric-streams.rst +++ b/gdi/get-data-in/connect/aws/aws-ts-metric-streams.rst @@ -1,7 +1,7 @@ .. _aws-ts-metric-streams: ****************************************************** -Troubleshoot AWS Metric Streams +Troubleshoot Splunk-managed Metric Streams ****************************************************** .. meta:: @@ -9,9 +9,7 @@ Troubleshoot AWS Metric Streams See the following topics when experiencing AWS Metric Streams related issues. -.. note:: - - See also :ref:`aws-troubleshooting`. +.. note:: See also :ref:`aws-troubleshooting` and :ref:`aws-ts-ms-aws`. I've enabled Splunk-managed Metric Streams in my AWS integration but I do not see any metrics streaming in ========================================================================================================== diff --git a/gdi/get-data-in/connect/aws/aws-ts-ms-aws.rst b/gdi/get-data-in/connect/aws/aws-ts-ms-aws.rst new file mode 100644 index 000000000..74f7c1508 --- /dev/null +++ b/gdi/get-data-in/connect/aws/aws-ts-ms-aws.rst @@ -0,0 +1,113 @@ +.. _aws-ts-ms-aws: + +****************************************************** +Troubleshoot AWS-managed Metric Streams +****************************************************** + +.. meta:: + :description: Troubleshoot AWS-managed Metric Streams related issues. + +See the following topics when experiencing AWS-managed Metric Streams related issues. + +.. note:: See also :ref:`aws-troubleshooting` and :ref:`aws-ts-metric-streams`. + +I've enabled AWS-managed Metric Streams in my AWS integration but I do not see any metrics streaming in +========================================================================================================== + +Log in to your AWS console and review the following: + +Check the Metric Stream integration settings +---------------------------------------------------------------------- + +In your CloudWatch console check the following: + +* The :strong:`Errors` chart must show zero errors. + +* The destination and output format must look similar to what is visible on the screenshot below. Note that your destination URL might use a different realm, for example https://ingest.us0.signalfx.com/v1/cloudwatch_metric_stream. + +.. image:: /_images/gdi/aws-ts-awsmanaged1.png + :width: 70% + :alt: Metric Stream settings + +Check the associated Firehose stream configuration +---------------------------------------------------------------------- + +Go to your AWS console and check your Firehose stream configuration: + +.. image:: /_images/gdi/aws-ts-awsmanaged2.png + :width: 70% + :alt: Firehose configuration + +Check the associated Firehose stream metrics +---------------------------------------------------------------------- + +Make sure that: + +* :strong:`Incoming records` must have non-zero values. + + * Note that if there's a low volume of traffic going through the stream, the value of ``IncomingRecords (per second average)`` can be significantly lower than ``RecordsPerSecondLimit``, and therefore it might look like there are no incoming records. + +* :strong:`HTTP endpoint delivery success` must be 100%. + +.. image:: /_images/gdi/aws-ts-awsmanaged3.png + :width: 70% + :alt: Firehose stream metrics + +If there are any errors visible on the HTTP endpoint delivery success chart, update the Firehose stream destination settings to ensure that the HTTP endpoint is correct and use a new access token with INGEST scope. + + * Note that your destination URL might use a different realm, for example https://ingest.us0.signalfx.com/v1/cloudwatch_metric_stream. + +.. image:: /_images/gdi/aws-ts-awsmanaged4.png + :width: 70% + :alt: Destination settings + +Review roles and permissions +----------------------------------- + +Check the following: + +* Review the AWS role used by the Metric Streams. + +* Make sure all required permissions are included, and that the CloudWatch metric stream region and the region listed in the AWS policy are the same. + + * To find the AWS role used by the Metric Stream, go to the Stream overview page under the :strong:`Service Role to write to Amazon Data Firehose` label. + +This is a sample AWS policy: + +.. code-block:: none + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "firehose:PutRecord", + "firehose:PutRecordBatch" + ], + "Resource": [ + "arn:aws:firehose:eu-west-2:906383545488:deliverystream/PUT-HTP-7pH7O" + ] + } + ] + } + +These are the trust relationships: + +.. code-block:: none + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "streams.metrics.cloudwatch.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] + } + + + diff --git a/gdi/get-data-in/connect/aws/get-awstoc.rst b/gdi/get-data-in/connect/aws/get-awstoc.rst index 9f2f37638..f4835d02a 100644 --- a/gdi/get-data-in/connect/aws/get-awstoc.rst +++ b/gdi/get-data-in/connect/aws/get-awstoc.rst @@ -23,12 +23,13 @@ Connect AWS to Splunk Observability Cloud Send AWS logs to Splunk Platform Next steps Troubleshoot your AWS integration - Troubleshoot Metric Streams + Troubleshoot Splunk-managed Metric Streams + Troubleshoot AWS-managed Metric Streams aws-recommended-stats.rst GetMetricStatistics API deprecation notice aws-tutorial/about-aws-tutorial.rst -You have several data ingestion and connection methods when it comes to monitoring your Amazon Web Services (AWS) data in Splunk Observability Cloud. +Splunk Observability Cloud offers you several data ingestion and connection methods to monitor your Amazon Web Services (AWS) data. .. note:: If you want to send AWS data to the Splunk platform, use the Splunk add-on. Learn more at :new-page:`Splunk Add-on for AWS `. diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.rst index 9e09a44d4..a18e41292 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.rst @@ -2,7 +2,7 @@ .. _kubernetes-config-logs: ********************************************************************************* -Collect logs and events for the Collector for Kubernetes +Collect logs and events with the Collector for Kubernetes ********************************************************************************* .. meta:: diff --git a/gdi/opentelemetry/collector-linux/linux-config-logs.rst b/gdi/opentelemetry/collector-linux/linux-config-logs.rst index cb87cac59..1c5602962 100644 --- a/gdi/opentelemetry/collector-linux/linux-config-logs.rst +++ b/gdi/opentelemetry/collector-linux/linux-config-logs.rst @@ -2,7 +2,7 @@ .. _linux-config-logs: *************************************************************** -Collect logs for the Collector for Linux +Collect logs with the Collector for Linux *************************************************************** .. meta:: diff --git a/gdi/opentelemetry/collector-windows/collector-windows-intro.rst b/gdi/opentelemetry/collector-windows/collector-windows-intro.rst index 50e1b873c..eadb93fef 100644 --- a/gdi/opentelemetry/collector-windows/collector-windows-intro.rst +++ b/gdi/opentelemetry/collector-windows/collector-windows-intro.rst @@ -17,6 +17,7 @@ Get started with the Collector for Windows Install the Collector for Windows (manually) windows-config-ootb.rst windows-config.rst + windows-config-logs.rst metrics-ootb-windows.rst windows-upgrade.rst windows-uninstall.rst @@ -31,8 +32,9 @@ To install the Splunk Distribution of the OpenTelemetry Collector for Windows, f See the default settings and configuration options at: * :ref:`windows-config-ootb` -* :ref:`otel-windows-config` * By default, you'll obtain these :ref:`metrics ` +* :ref:`otel-windows-config` +* :ref:`windows-config-logs` .. include:: /_includes/gdi/collector-common-options.rst diff --git a/gdi/opentelemetry/collector-windows/install-windows.rst b/gdi/opentelemetry/collector-windows/install-windows.rst index 05b4a90bb..98d27162c 100644 --- a/gdi/opentelemetry/collector-windows/install-windows.rst +++ b/gdi/opentelemetry/collector-windows/install-windows.rst @@ -167,55 +167,6 @@ The Windows installer script supports the following options: - Specify public MSI properties to be used when installing the Splunk OpenTelemetry Collector MSI package. - -.. _windows-config-logs: - -Collect logs for the Collector for Windows -==================================================================== - -Use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. - -.. _fluentd-manual-config-windows: - -Collect Windows logs with Fluentd ---------------------------------------- - -If you wish to collect logs for the target host with Fluentd, use the ``with_fluentd = 1`` option to install and enable Fluentd when installing the Collector. For example: - -.. code-block:: PowerShell - - & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; with_fluentd = 1}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} - -When activated, the Fluentd service is configured by default to collect and forward log events with the ``@SPLUNK`` label to the Collector, which then send these events to the HEC ingest endpoint determined by the ``realm = ""`` option. -For example, ``https://ingest..signalfx.com/v1/log``. - -To configure the package to send log events to a custom HTTP Event Collector (HEC) endpoint URL with a token different than ````, you can specify the following parameters for the installer script: - -* ``hec_url = ""`` -* ``hec_token = ""`` - -For example (replace the ```` values in the command for your configuration): - -.. code-block:: PowerShell - - & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; hec_url = ""; hec_token = ""}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} - -The installation creates the main fluentd configuration file ``\opt\td-agent\etc\td-agent\td-agent.conf``, where ```` is the drive letter for the fluentd installation directory. - -You can add custom fluentd source configuration files to the ``\opt\td-agent\etc\td-agent\conf.d`` -directory after installation. - -Note the following: - -* In this directory, fluentd includes all files with the .conf extension. -* By default, fluentd collects from the Windows Event Log. See ``\opt\td-agent\etc\td-agent\conf.d\eventlog.conf`` for the default configuration. - -After any configuration modification, apply the changes by restarting the system or running the following PowerShell commands: - -.. code-block:: PowerShell - - Stop-Service fluentdwinsvc - Start-Service fluentdwinsvc - Next steps ================================== diff --git a/gdi/opentelemetry/collector-windows/windows-config-logs.rst b/gdi/opentelemetry/collector-windows/windows-config-logs.rst new file mode 100644 index 000000000..576aad892 --- /dev/null +++ b/gdi/opentelemetry/collector-windows/windows-config-logs.rst @@ -0,0 +1,59 @@ +.. _windows-config-logs: + +*************************************************************** +Collect logs with the Collector for Windows +*************************************************************** + +.. meta:: + + :description: Describes how to collect logs for the Splunk Distribution of OpenTelemetry Collector for Linux. + +Use the Universal Forwarder to send logs to the Splunk platform. See more at :ref:`collector-with-the-uf`. + +.. _fluentd-manual-config-windows: + +Collect Windows logs with Fluentd +=========================================================================== + +Fluentd is turned off by default. + +If you wish to collect logs for the target host with Fluentd, use the ``with_fluentd = 1`` option to install and enable Fluentd when installing the Collector. + +For example: + +.. code-block:: PowerShell + + & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; with_fluentd = 1}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} + +When activated, the Fluentd service is configured by default to collect and forward log events with the ``@SPLUNK`` label to the Collector, which then send these events to the HEC ingest endpoint determined by the ``realm = ""`` option. +For example, ``https://ingest..signalfx.com/v1/log``. + +To configure the package to send log events to a custom HTTP Event Collector (HEC) endpoint URL with a token different than ````, you can specify the following parameters for the installer script: + +* ``hec_url = ""`` +* ``hec_token = ""`` + +For example (replace the ```` values in the command for your configuration): + +.. code-block:: PowerShell + + & {Set-ExecutionPolicy Bypass -Scope Process -Force; $script = ((New-Object System.Net.WebClient).DownloadString('https://dl.signalfx.com/splunk-otel-collector.ps1')); $params = @{access_token = ""; realm = ""; hec_url = ""; hec_token = ""}; Invoke-Command -ScriptBlock ([scriptblock]::Create(". {$script} $(&{$args} @params)"))} + +The installation creates the main Fluentd configuration file ``\opt\td-agent\etc\td-agent\td-agent.conf``, where ```` is the drive letter for the fluentd installation directory. + +You can add custom Fluentd source configuration files to the ``\opt\td-agent\etc\td-agent\conf.d`` +directory after installation. + +Note the following: + +* In this directory, Fluentd includes all files with the .conf extension. +* By default, fluentd collects from the Windows Event Log. See ``\opt\td-agent\etc\td-agent\conf.d\eventlog.conf`` for the default configuration. + +After any configuration modification, apply the changes by restarting the system or running the following PowerShell commands: + +.. code-block:: PowerShell + + Stop-Service fluentdwinsvc + Start-Service fluentdwinsvc + + diff --git a/gdi/opentelemetry/troubleshoot-logs.rst b/gdi/opentelemetry/troubleshoot-logs.rst index 3782764a3..d5e214982 100644 --- a/gdi/opentelemetry/troubleshoot-logs.rst +++ b/gdi/opentelemetry/troubleshoot-logs.rst @@ -9,6 +9,14 @@ Troubleshoot log collection This document describes common issues related to log collection with the Collector. +.. note:: + + To collect logs see: + + * :ref:`kubernetes-config-logs` + * :ref:`linux-config-logs` + * :ref:`windows-config-logs` + To troubleshoot the health and performance of the Collector see the :new-page:`OpenTelemetry Project troublehooting docs `. It includes information about troubleshooting tools and debugging. My source isn't generating logs @@ -69,9 +77,7 @@ You can manually generate logs. By default, Fluentd monitors journald and /var/l echo "2021-03-17 02:14:44 +0000 [debug]: test" >>/var/log/syslog.log echo "2021-03-17 02:14:44 +0000 [debug]: test" | systemd-cat -.. note:: - - Properly structured syslog is required for Fluentd to properly pick up the log line. +.. caution:: Fluentd requires properly structured syslog to pick up the log line. .. _unwanted_profiling_logs: diff --git a/gdi/requirements.rst b/gdi/requirements.rst index 1a267d384..c8d960e26 100644 --- a/gdi/requirements.rst +++ b/gdi/requirements.rst @@ -17,8 +17,16 @@ The Splunk Distribution of OpenTelemetry Collector supports the following operat Linux ------------------------------------------------------------ +.. raw:: html + +
+ .. include:: /_includes/requirements/collector-linux.rst +.. raw:: html + +
+ See :ref:`get-started-linux` for more information. Windows