Skip to content
This repository was archived by the owner on Sep 2, 2025. It is now read-only.

Commit 5ddbd9d

Browse files
authored
Merge pull request #2167 from splunk/bhillmer-O11YDOCS-5885-tag-spotlight-improvements
[O11YDOCS-5885] tag spotlight improvements
2 parents 203783a + 7e8da09 commit 5ddbd9d

File tree

4 files changed

+32
-34
lines changed

4 files changed

+32
-34
lines changed
-604 KB
Binary file not shown.
659 KB
Loading

apm/apm-scenarios/troubleshoot-tag-spotlight.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Scenario: Deepu finds the root cause of an error using Tag Spotlight
66
.. meta::
77
:description: This Splunk APM scenario describes how to use APM Tag Spotlight to find root cause errors.
88

9-
Deepu receives a notification from Kai, the site reliability engineer. The notification says that the high root cause error rate with the ``/PaymentService/Charge`` endpoint is preventing customers from shopping on the Buttercup Games website. The notification also includes a link to the endpoint on the Splunk APM service map.
9+
Deepu receives a notification from Kai, the site reliability engineer. The notification says that the high root-cause error rate with the ``/PaymentService/Charge`` endpoint is preventing customers from shopping on the Buttercup Games website. The notification also includes a link to the endpoint on the Splunk APM service map.
1010

1111
To troubleshoot the issue, Deepu takes the following steps:
1212

apm/span-tags/tag-spotlight.rst

Lines changed: 31 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,59 @@
11
.. _apm-tag-spotlight:
22

3-
*****************************************************************************
43
Analyze service performance with Tag Spotlight
54
*****************************************************************************
65

76
.. meta::
87
:description: Learn how to view metrics for every indexed span tag for a service in a single window in Splunk Observability Cloud.
98

10-
Use Tag Spotlight to analyze the performance of your services in Splunk Observability Cloud to discover trends that contribute to high latency or error rates with indexed span tags. You can break down every service by each indexed span tag to view its metrics. When you select span tag values or a time range, you can view relevant traces to learn more about outlying incidents.
9+
Use Tag Spotlight to analyze the performance of your services in Splunk Observability Cloud to discover trends that contribute to high latency or error rates with span tags you have indexed. See :ref:`apm-index-span-tags` for more steps to index span tags.
1110

12-
For every service, Tag Spotlight provides request, error, and duration (RED) metrics time-series charts for every value of each indexed span tag within the specified time range in the APM navigation menu. Request and error charts display the total number of requests, errors, and root cause errors, and duration charts display p50, p90, and p99 latency. These values are based on Troubleshooting MetricSets (TMS), which Splunk APM generates for every indexed span tag.
11+
For every service, Tag Spotlight provides time-series charts for request, error, and duration (RED) metrics for every value of each indexed span tag. Request and error charts display the total number of requests, errors, and root cause errors, and duration charts display p50, p90, and p99 latency. These values are based on Troubleshooting MetricSets (TMS), which Splunk APM generates for every indexed span tag. When you select span tag values or a time range, you can view relevant traces to learn more about outlying incidents.
1312

14-
The following image shows the requests and errors of an example ``frontend`` service broken down by the values of indexed span tags. The graph is powered by Troubleshooting MetricSets.
13+
The following image shows the a chart for requests and errors and a chart for duration for an example ``frontend`` service. Under the charts there are bar charts with RED metrics for each indexed span tag.
1514

16-
.. image:: /_images/apm/span-tags/tag-spotlight-01.png
17-
:alt: This image shows the requests and errors of an example frontend service broken down by indexed span tag.
15+
.. image:: /_images/apm/span-tags/tag-spotlight-fy25pi2.png
16+
:alt: This image shows requests and errors and latency charts as well as requests and errors broken down by indexed span tag for an example frontend service.
1817

1918
To learn how to index span tags to generate Troubleshooting MetricSets, see :ref:`apm-index-span-tags`. For a general overview of MetricSets in APM, see :ref:`apm-metricsets`.
2019

21-
View service performance by indexed span tags with Tag Spotlight
20+
View service performance by indexed span tags
2221
================================================================
23-
24-
Access Tag Spotlight from the :strong:`Tag Spotlight` panel of the APM landing page. You can also use the search in the top toolbar to search for Tag Spotlight and select the navigation result to navigate to Tag Spotlight.
25-
26-
.. image:: /_images/apm/span-tags/tag-spotlight-02.gif
27-
:width: 99%
28-
:alt: This animation shows the user searching for Tag Spotlight.
29-
30-
You can break down performance for each tag by either requests and errors or latency.
31-
32-
Follow these steps to navigate to Tag Spotlight for a specific service:
33-
34-
#. From the Splunk Observability Cloud landing page, select :guilabel:`APM`.
35-
#. In the list of services on the APM landing page, select a service you want to drill into. Selecting the service opens the Troubleshooting tab for that service.
36-
#. Scroll to the :strong:`Tag Spotlight` card in the sidebar to view a preview of top tags with errors and high latency. Select the card to open Tag Spotlight.
37-
#. View the distribution of all indexed span tags. The RED metrics time-series chart displays requests, errors, root cause errors, or latency for the specified time range.
38-
#. Adjust the time range to view more or less data in the RED metrics chart. The default time range is for the last 15 minutes, and the data resolution is 10 seconds.
39-
40-
Use the :guilabel:`Service` picker in the top bar of Tag Spotlight to view metrics for a different service.
22+
To view service performance broken down by your indexed span tags, follow these steps:
23+
24+
#. Go to Tag Spotlight. You can access Tag Spotlight in several places:
25+
* Select the :guilabel:`Tag Spotlight` panel on the APM landing page or in the service map.
26+
* Select the :guilabel:`Tag Spotlight` tab in the service view for your service.
27+
* Use the search in the top toolbar to search for Tag Spotlight and select the navigation result to go to Tag Spotlight.
28+
#. Add time range, environment, workflow, service, operation, and tag filters as need to refine the data in your Tag Spotlight view. The default time range is for the last 15 minutes, and the data resolution is 10 seconds.
29+
#. Use the request and errors and latency time-series charts at the top to see a distribution of your RED metrics.
30+
#. Use the bar charts under the request and errors and latency time-series charts to view RED metrics for each indexed span tag.
31+
#. View the distribution of all indexed span tags. The tag bar charts display either request and error distributions or latency distribution. Use the :guilabel:`Cards display` menu to select the data you want to display in the bars.
32+
#. Select the menu on the top left of the bar chart section to select which metrics to display in each tag panel. You can also use this menu to select whether to display tags with no values.
4133

4234
Explore the distribution of span tags and values to find trends
4335
----------------------------------------------------------------------
44-
There are a number of ways to drill down into indexed span tags for a selected service so you can understand the source of an incident:
36+
37+
To understand the source of an incident use 1 of these options to drill down into indexed span tags for a selected service:
4538

4639
- Use the filter bar to filter the requests shown in Tag Spotlight by environment, service, and operation. You can also create a custom filter based on indexed or unindexed tags. The filter context is preserved even when you navigate away from the page.
4740

48-
- Hover over a span tag value for any indexed span tag to view the performance of the span tag value relative to all other indexed span tags and values. You can also select a span tag value to add a filter to filter the RED metrics chart to spans containing that value.
41+
- Select the menu on the top left of the bar chart section to select which metrics to display for any indexed span tag.
42+
43+
- Select a span tag value then select :guilabel:`Add to filter` to filter the RED metrics charts to spans containing that value.
4944

50-
- Hover over the RED metrics chart to view a summary of metrics for the specified time. The resolution of data for the RED metrics chart is 10 seconds.
45+
- Hover over the RED metrics charts to view a summary of metrics for the specified time. The resolution of data for the RED metrics chart is 10 seconds.
5146

52-
- Select within the RED metrics chart to load representative traces for the time you selected.
47+
- Select within the RED metrics charts to load example traces for the time you selected.
5348

54-
- Select a span tag card header to add the top 5 span tag values in the card to the RED metrics chart.
49+
- Select a span tag card header to add the top 5 span tag values in the card to the RED metrics charts.
5550

56-
- Control what appears in the RED metrics chart by showing or hiding metrics in the chart's legend. For example, you can select the eye icon beside :guilabel:`Requests` to view only errors. When the slider is set to :guilabel:`Latency`, select the eye icon to hide the p50 latency metric and compare only p90 and p99 values.
51+
- Control what appears in the RED metrics chart by showing or hiding metrics in the chart's legend. For example, you can select the eye icon beside :guilabel:`Requests` to view only errors.
5752

58-
Scenario: Find the root cause of an incident with Tag Spotlight
53+
Learn more
5954
================================================================
6055

61-
To view a detailed example of using Tag Spotlight, see :ref:`troubleshoot-tag-spotlight`.
56+
To view a detailed examples using Tag Spotlight, see:
57+
58+
* :ref:`troubleshoot-tag-spotlight`
59+
* :ref:`troubleshoot-business-workflows`

0 commit comments

Comments
 (0)