You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 2, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: apm/span-tags/cmms.rst
+38-20Lines changed: 38 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,29 +7,36 @@ Generate a Monitoring MetricSet with a custom dimension
7
7
.. meta::
8
8
:description: Learn how to generate a Monitoring MetricSet with a custom dimension.
9
9
10
-
Generate Monitoring MetricSets (MMS) with custom dimensions to monitor your services in real time. Splunk APM provides a default set of MMS, but you can configure additional MMS using indexed span tags to create custom dimensions. You can use your MMS to create custom charts, dashboards, and detectors to keep track of what matters most to your team. See :ref:`charts-dashboards-alerts`.
10
+
Generate Monitoring MetricSets (MMS) with custom dimensions to monitor your services in real time. Splunk APM provides a default set of MMS, but you can configure additional MMS using indexed span tags to create custom dimensions. Use your custom MMS to create custom charts, dashboards, and detectors to track what matters most to your team. See :ref:`charts-dashboards-alerts`.
11
11
12
-
To learn about the default MMS in Splunk APM, see :ref:`monitoring-metricsets`.
12
+
What are MetricSets?
13
+
=========================
14
+
15
+
MetricSets are metric time series (MTS) you can use to track the performance of specific metrics over time. To help you analyze your service performance, APM provides 2 categories of MetricSets: Troubleshooting MetricSets (TMS) and Monitoring MetricSets (MMS). Use Troubleshooting MetricSets for high-cardinality troubleshooting. Use Monitoring MetricSets for real-time monitoring and alerting.
16
+
17
+
APM provides a collection of MetricSets by default. You can also index additional span tags or processes to generate additional TMS and, as part of the indexing process, you can opt to create custom dimensionalized Monitoring MetricSets (MMS). See :ref:`apm-MetricSets` for an overview of the types of MetricSets in APM and a list of the MetricSets available by default.
18
+
19
+
Use this topic to learn how to index a span tag or process to generate Monitoring MetricSets. See :ref:`apm-index-span-tags` to learn how to generate Troubleshooting MetricSets.
13
20
14
21
Prerequisites
15
22
==============
16
23
17
24
* To create a custom MMS, you must have an admin role.
18
25
19
-
* To create a custom MMS, you need to first index the span tag or process that you want to base your MMS on. See :ref:`apm-index-span-tags`.
26
+
* To create a custom MMS, you need to index the span tag or process that you want to base your MMS on. See :ref:`apm-index-span-tags`.
20
27
21
-
* If you are unfamiliar with MetricSets, see::ref:`apm-metricsets`.
28
+
* If you are unfamiliar with MetricSets, see :ref:`apm-metricsets`.
22
29
23
30
* (Optional) Review :ref:`metrics-dimensions-mts` for an overview of the concepts underpinning MetricSets.
24
31
25
-
How custom Monitoring MetricSets add value to your organization
32
+
Use custom Monitoring MetricSets to monitor important metrics
APM generates a collection of MMS by default; see :ref:`monitoring-metricsets`. You can create additional MMS with custom dimensions to filter and aggregate the default generated metrics by a specific indexed span tag or process such as ``version`` or ``cloud.provider``.
35
+
In addition to the MMS APM generates by default, you can create additional MMS with custom dimensions to filter and aggregate the default generated metrics by a specific indexed span tag or process such as ``version`` or ``cloud.provider``.
29
36
30
-
You can create custom MMS at the service level and the endpoint or span level. When you create a custom dimension for a service-level MMS, APM generates MMS that include the service-level metrics with your chosen indexed span tag or process as a custom dimension. If you add endpoint-level metrics as well, APM generates MMS that include span-level metrics, with your chosen indexed span tag or process as a custom dimension.
37
+
You can create custom MMS at the service level and the endpoint or span level. When you create a custom dimension for a service-level MMS, APM generates an MMS that includes the service-level metrics with your chosen indexed span tag or process as a custom dimension. If you add endpoint-level metrics, APM generates MMS that include span-level metrics, with your chosen indexed span tag or process as a custom dimension.
31
38
32
-
When you create an MMS with a custom dimension, you can use this custom dimension to create charts, dashboards, and alerts. See :ref:`charts-dashboards-alerts`.
39
+
When you create an MMS with a custom dimension, you can use the custom dimension to create charts, dashboards, and alerts. See :ref:`charts-dashboards-alerts`.
33
40
34
41
To learn more about a specific scenario for custom MMS: :ref:`custom-metricset`.
35
42
@@ -46,15 +53,15 @@ Before you generate an MMS, first consider:
46
53
Select how you want to configure your Monitoring MetricSet
You can generate a Monitoring MetricSet in three ways:
56
+
You can generate a Monitoring MetricSet in 3 ways:
50
57
51
58
* service-level MMS only
52
59
* endpoint-level MMS for specific endpoints within a given service
53
60
* orient both service-level MMS and endpoint-level MMS for all endpoints within a service
54
61
55
-
Select the best option for your needs to allocate your organization's cardinality entitlement efficiently. See :ref:`reduce-cardinality` to learn more.
62
+
Select the best option for your needs to allocate your organization's cardinality entitlement efficiently. See :ref:`Configure your MMS to be more efficient<reduce-cardinality>` to learn more.
56
63
57
-
You can also filter by specific tag values to further reduce the cardinality of your MMS. All custom MMS include service-level metrics. In addition, you can generate endpoint-level metrics.
64
+
You can also filter by specific tag values to further reduce the cardinality of your MMS.
58
65
59
66
60
67
.. list-table::
@@ -66,7 +73,7 @@ You can also filter by specific tag values to further reduce the cardinality of
66
73
* - service-level MMS and endpoint-level MMS for all endpoints within a service
67
74
- This option creates a custom dimensionalized MMS for each selected service, as well as an MMS for each endpoint in each selected service. This is the highest cardinality option.
68
75
* - service-level MMS and endpoint-level MMS for specific endpoints
69
-
- This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled :guilabel:`Endpoint Filters, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you're interested in. This helps you to minimize unnecessary cardinality.`.
76
+
- This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you're interested in. This helps you to minimize unnecessary cardinality`.
70
77
* - Service-level MMS only
71
78
- This option creates an MMS for each selected service and no endpoint-level MMS. This is the lowest cardinality option.
72
79
@@ -86,7 +93,7 @@ Follow these steps to create a Monitoring MetricSet.
86
93
a. In Splunk APM, select :guilabel:`APM Configuration` and select :guilabel:`APM MetricSets` from the menu. The APM MetricSets page opens.
87
94
b. From anywhere in Splunk Observability Cloud, select :guilabel:`Settings` in the navigation bar and select :guilabel:`APM MetricSets` under :guilabel:`Data Configuration`.
88
95
89
-
2. On the :guilabel:`APM MetricSets` page, you have two options:
96
+
2. On the :guilabel:`APM MetricSets` page, you have 2 options:
90
97
91
98
* If you have already indexed the span tag or process, it appears in the list of MetricSets and is already generating Troubleshooting MetricSets. Select the edit icon for that span tag to open the :guilabel:`Edit MetricSet` dialog box and add a Monitoring MetricSet to your configuration using the following steps.
92
99
* If you haven't already indexed the tag or process, select :strong:`New MetricSet`. Enter the :strong:`Name` of a span tag or process you want to index. Then continue with the following steps.
@@ -102,7 +109,7 @@ Follow these steps to create a Monitoring MetricSet.
102
109
103
110
5. Select how you want to add tag or process data to your Monitoring MetricSet from the dropdown. See :ref:`mms-conf` for more details.
104
111
a. :guilabel:`Service and all endpoint MMS:` Create an MMS for each selected service, as well as an MMS for each endpoint in each selected service.
105
-
b. :guilabel:`Service and specific endpoint MMS:` Create an MMS for each selected service and an MMS for specific endpoints you select. To add :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to generate MMS for specific the endpoints you're interested in.
112
+
b. :guilabel:`Service and specific endpoint MMS:` Create an MMS for each selected service and an MMS for specific endpoints you select. To add :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you're interested in.
106
113
c. :guilabel:`Service MMS only:` Create an MMS for each selected service and no endpoint-level MMS.
107
114
108
115
6. (Optional) In the field labeled :guilabel:`Filter by tag values`, add tag filters to narrow the scope of your MMS to include only data associated with the tag values you enter. This can help further reduce cardinality in your MetricSets.
@@ -111,13 +118,13 @@ Follow these steps to create a Monitoring MetricSet.
111
118
112
119
8. Your new metric set shows under :strong:`Pending MetricSets` section of the table with an :guilabel:`Analyzing` status while the cardinality check runs. You might have to wait a few moments for the check to run.
113
120
114
-
.. caution:: Your pending MetricSet expires after one hour. If your MetricSet times out before you enable it, run the analysis again.
121
+
.. caution:: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
115
122
116
-
9. After the cardinality check has run, review the cardinality and enable your new MetricSet.
123
+
9. After the cardinality check has run, review the cardinality and activate your new MetricSet.
117
124
118
-
* If the cardinality impact of indexing this new MetricSet is acceptable, select the check mark under :guilabel:`Actions` to enable your Troubleshooting MetricSet.
119
-
* If the check times out before you enable the MetricSet, run the analysis again.
120
-
* If the check shows as failed, your cardinality is too high to create the new MetricSet. You can edit the MetricSet to adjust the configuration to reduce cardinality or you can select the x to delete the pending Metricset.
125
+
* If the cardinality impact of indexing this new MetricSet is within your subscription limit you see a green check mark and "Within Entitlement." Select the check mark under :guilabel:`Actions` to activate your Monitoring MetricSet.
126
+
* If the check times out before you activate the MetricSet, run the analysis again.
127
+
* If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See :ref:`cardinality-check-tms`. For steps to configure your MetricSet to be more efficient or other troubleshooting solutions, see :ref:`troubleshoot-mms`.
121
128
122
129
The following example shows a cardinality check.
123
130
@@ -127,6 +134,17 @@ Follow these steps to create a Monitoring MetricSet.
127
134
128
135
.. note:: Splunk APM replaces dots with underscores in dimension names for MMS time series.
129
136
137
+
.. _cardinality-check-mms:
138
+
139
+
Cardinality contribution of indexed span tags and processes
When you index a new span tag or process to generate Monitoring MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span tag or process. This gives you control of what you index and helps you to account for organization subscription limits.
143
+
144
+
If you try to index a span tag or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes by modifying or removing indexed span tags or processes.
145
+
146
+
To see your TMS subscription limit, go to :guilabel:`Settings` then :guilabel:`Subscription Usage`. Depending on your organization subscription, you might need to go to :guilabel:`Settings` then :guilabel:`Billing and Usage`. Select the :guilabel:`APM` tab and then select the :guilabel:`Troubleshooting MetricSets` panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see :ref:`apm-billing-usage-index`.
147
+
130
148
.. _charts-dashboards-alerts:
131
149
132
150
Set up charts, dashboards, and alerts for custom dimensions
@@ -155,7 +173,7 @@ Metrics and dimensions of Monitoring MetricSets
155
173
156
174
Each MMS has a set of metrics and dimensions for spans and traces you can use to monitor and alert on service performance.
157
175
158
-
To prevent overcounting metrics in aggregations, the built-in dashboards and charts in Splunk APM automatically exclude custom dimensionalized MMS.
176
+
To prevent over counting metrics in aggregations, the built-in dashboards and charts in Splunk APM automatically exclude custom dimensionalized MMS.
159
177
Custom dimensionalized MMS have a marker dimension, ``sf_dimensionalized: true``, to include custom dimensionalized MMS.
160
178
161
179
When you create your dashboards and charts, you can exclude custom dimensionalized MMS by adding a filter on ``!sf_dimensionalized: true``.
0 commit comments