Merge pull request #2098 from splunk/bhillmer-O11YDOCS-6037-inferred-service-MMS

[O11YDOCS-6037] inferred service mms

Author: bhillmer

apm/apm-spans-traces/inferred-services.rst

@@ -54,8 +54,6 @@ What information about inferred services is available?
 
 Because inferred services are not instrumented, Splunk APM only has access to spans from instrumented services that call inferred services. Splunk APM places inferred services in their inferred locations in the service map and provides :ref:`Troubleshooting MetricSets<troubleshooting-metricsets>` (TMS), or request, error, and duration (RED) metrics, based on the associated client-side spans. You can continue to troubleshoot with these metrics in :ref:`Tag Spotlight<apm-tag-spotlight>`. If the inferred service is servicing other uninstrumented services, these reported RED metrics might not provide a complete picture of the inferred service's interactions. 
 
-Splunk APM does not create :ref:`Monitoring MetricSets<monitoring-metricsets>` (MMS) for inferred services because MMS are created in real time from spans where ``span.kind = SERVER`` or ``span.kind = CONSUMER``, and Splunk APM does not receive server or consumer-side spans from uninstrumented services. This means that even though inferred services appear in the list of services on the landing page, you can't get real-time MMS data on inferred services in the APM landing page or in dashboards and alerts, and you can't create detectors based on inferred services. 
-
 .. _inferred-service-map:
 
 Viewing inferred services in the service map
@@ -76,6 +74,13 @@ Inferred services, pub/subs, and databases also appear as spans in the Trace Wat
 
 When you select an inferred span in the Trace Waterfall, it expands to show the metadata of the corresponding parent span. The length of the operation represented as the gray striped bar in the waterfall visualization is also inherited from the parent span and might not be exactly representative of the operation duration in the inferred service.
 
+Create Monitoring MetricSets to chart and alert on inferred services
+-----------------------------------------------------------------------
+
+.. note:: Only 3rd-party or uninstrumented HTTP services are supported for MMS.
+
+You can create a Monitoring MetricSet (MMS) for inferred services. MMS enable you to chart inferred service metrics in dashboards and create detectors to alert on your inferred services. See :ref:`inferred-service-mms`.
+
 .. _how-apm-infers-services:
 
 How does Splunk APM identify inferred services?

apm/span-tags/cmms.rst

@@ -1,22 +1,22 @@
 .. _cmms:
 
 *******************************************************
-Generate a Monitoring MetricSet with a custom dimension
+Create a Monitoring MetricSet with a custom dimension
 *******************************************************
 
 .. meta::
-   :description: Learn how to generate a Monitoring MetricSet with a custom dimension. 
+   :description: Learn how to create a Monitoring MetricSet with a custom dimension. 
 
-Generate Monitoring MetricSets (MMS) with custom dimensions to monitor your services in real time. Splunk APM provides a default set of MMS, but you can configure additional MMS using indexed span tags to create custom dimensions. Use your custom MMS to create custom charts, dashboards, and detectors to track what matters most to your team. See :ref:`charts-dashboards-alerts`. 
+Create Monitoring MetricSets (MMS) with custom dimensions to monitor your services in real time. Splunk APM provides a default set of MMS, but you can configure additional MMS using indexed span tags to create custom dimensions. Use your custom MMS to create custom charts, dashboards, and detectors to track what matters most to your team. See :ref:`charts-dashboards-alerts`. 
 
 What are MetricSets?
 =========================
 
 MetricSets are metric time series (MTS) you can use to track the performance of specific metrics over time. To help you analyze your service performance, APM provides 2 categories of MetricSets: Troubleshooting MetricSets (TMS) and Monitoring MetricSets (MMS). Use Troubleshooting MetricSets for high-cardinality troubleshooting. Use Monitoring MetricSets for real-time monitoring and alerting. 
 
-APM provides a collection of MetricSets by default. You can also index additional span tags or processes to generate additional TMS and, as part of the indexing process, you can opt to create custom dimensionalized Monitoring MetricSets (MMS). See :ref:`apm-MetricSets` for an overview of the types of MetricSets in APM and a list of the MetricSets available by default. 
+APM provides a collection of MetricSets by default. You can also index additional span tags or processes to create additional TMS and, as part of the indexing process, you can opt to create custom dimensionalized Monitoring MetricSets (MMS). See :ref:`apm-MetricSets` for an overview of the types of MetricSets in APM and a list of the MetricSets available by default. 
 
-Use this topic to learn how to index a span tag or process to generate Monitoring MetricSets. See :ref:`apm-index-span-tags` to learn how to generate Troubleshooting MetricSets. 
+Use this topic to learn how to index a span tag or process to create Monitoring MetricSets. See :ref:`apm-index-span-tags` to learn how to create Troubleshooting MetricSets. 
 
 Prerequisites 
 ==============
@@ -32,18 +32,18 @@ Prerequisites
 Use custom Monitoring MetricSets to monitor important metrics 
 ======================================================================
 
-In addition to the MMS APM generates by default, you can create additional MMS with custom dimensions to filter and aggregate the default generated metrics by a specific indexed span tag or process such as ``version`` or ``cloud.provider``.
+In addition to the MMS APM creates by default, you can create additional MMS with custom dimensions to filter and aggregate the default created metrics by a specific indexed span tag or process such as ``version`` or ``cloud.provider``.
 
-You can create custom MMS at the service level and the endpoint or span level. When you create a custom dimension for a service-level MMS, APM generates an MMS that includes the service-level metrics with your chosen indexed span tag or process as a custom dimension. If you add endpoint-level metrics, APM generates MMS that include span-level metrics, with your chosen indexed span tag or process as a custom dimension. 
+You can create custom MMS at the service level and the endpoint or span level. When you create a custom dimension for a service-level MMS, APM creates an MMS that includes the service-level metrics with your chosen indexed span tag or process as a custom dimension. If you add endpoint-level metrics, APM creates MMS that include span-level metrics, with your chosen indexed span tag or process as a custom dimension. 
 
 When you create an MMS with a custom dimension, you can use the custom dimension to create charts, dashboards, and alerts. See :ref:`charts-dashboards-alerts`. 
 
 To learn more about a specific scenario for custom MMS: :ref:`custom-metricset`.
 
-Generate a Monitoring MetricSet with a custom dimension 
-====================================================================
+Create a Monitoring MetricSet with a custom dimension for an instrumented service
+=======================================================================================
 
-Before you generate an MMS, first consider:
+Before you create an MMS, first consider:
 
 * How you want to allocate the cardinality of your data to make your MMS most efficient. 
 * Your organization's cardinality limits. See :ref:`troubleshoot-mms` for guidance on editing your MetricSet configuration to optimize your overall cardinality.
@@ -53,7 +53,7 @@ Before you generate an MMS, first consider:
 Select how you want to configure your Monitoring MetricSet
 ------------------------------------------------------------------
 
-You can generate a Monitoring MetricSet in 3 ways:
+You can create a Monitoring MetricSet in 3 ways:
 
 * service-level MMS only
 * endpoint-level MMS for specific endpoints within a given service
@@ -73,20 +73,20 @@ You can also filter by specific tag values to further reduce the cardinality of
    * - service-level MMS and endpoint-level MMS for all endpoints within a service
      -  This option creates a custom dimensionalized MMS for each selected service, as well as an MMS for each endpoint in each selected service. This is the highest cardinality option. 
    * - service-level MMS and endpoint-level MMS for specific endpoints
-     - This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you're interested in. This helps you to minimize unnecessary cardinality`.
+     - This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to create MMS for specific endpoints you're interested in. This helps you to minimize unnecessary cardinality.
    * - Service-level MMS only
      - This option creates an MMS for each selected service and no endpoint-level MMS. This is the lowest cardinality option. 
 
 
 Scope of Monitoring MetricsSets 
 ---------------------------------------------------------------
 
-You can create custom MMS for endpoints (metrics starting with ``span*``) and services (metrics starting with ``service.request.*``), but not for workflows (metrics starting with ``workflow*``) or traces (metrics starting with ``trace*``) at this time. Custom MMS aren't supported for global tags.
+You can create custom MMS for endpoints (metrics starting with ``span*``) and services (metrics starting with ``service.request.*``), but not for workflows (metrics starting with ``workflow*``) or traces (metrics starting with ``trace*``) at this time. Custom MMS aren't supported for global tags. See :ref:`default-mms`.
 
 Add a Monitoring MetricSet
 ------------------------------
 
-Follow these steps to create a Monitoring MetricSet. 
+Follow these steps to create a Monitoring MetricSet for an instrumented service:
 
 1. To get to the :guilabel:`APM MetricSets` page, do one of the following: 
 
@@ -106,17 +106,16 @@ Follow these steps to create a Monitoring MetricSet.
         :width: 70%
         :alt: This image shows the MetricSet creation dialog box. 
 
-
 5. Select how you want to add tag or process data to your Monitoring MetricSet from the dropdown. See :ref:`mms-conf` for more details. 
     a. :guilabel:`Service and all endpoint MMS:` Create an MMS for each selected service, as well as an MMS for each endpoint in each selected service. 
-    b. :guilabel:`Service and specific endpoint MMS:` Create an MMS for each selected service and an MMS for specific endpoints you select. To add :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you're interested in. 
+    b. :guilabel:`Service and specific endpoint MMS:` Create an MMS for each selected service and an MMS for specific endpoints you select. To add :guilabel:`Endpoint Filters`, provide a list of endpoints or a regular expression pattern to create MMS for specific endpoints you're interested in. 
     c. :guilabel:`Service MMS only:` Create an MMS for each selected service and no endpoint-level MMS. 
 
 6. (Optional) In the field labeled :guilabel:`Filter by tag values`, add tag filters to narrow the scope of your MMS to include only data associated with the tag values you enter. This can help further reduce cardinality in your MetricSets.
 
 7. Select :strong:`Start Analysis` to submit the configuration. Splunk APM runs an analysis of the span tag or process to calculate the potential cardinality contribution of the MetricSet to determine whether it is within your entitlement limit. 
 
-8. Your new metric set shows under :strong:`Pending MetricSets` section of the table with an :guilabel:`Analyzing` status while the cardinality check runs. You might have to wait a few moments for the check to run. 
+8. Your new MetricSet shows under :strong:`Pending MetricSets` section of the table with an :guilabel:`Analyzing` status while the cardinality check runs. You might have to wait a few moments for the check to run. 
 
    .. caution:: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again. 
 
@@ -132,26 +131,52 @@ Follow these steps to create a Monitoring MetricSet.
       :width: 60%
       :alt: This image shows the cardinality check for a Monitoring MetricSet. 
 
-.. note:: Splunk APM replaces dots with underscores in dimension names for MMS time series.
+.. note:: For standard MMS Splunk APM replaces dots with underscores in dimension names for MMS time series. For histogram MMS underscores are preserved.
+.. _inferred-service-mms:
+
+Create a Monitoring MetricSet with a custom dimension for an inferred service
+=======================================================================================
+
+.. note:: Only 3rd-party or uninstrumented HTTP services are supported for MMS.
+
+Follow these steps to create a Monitoring MetricSet for an inferred service:
+
+1. To get to the :guilabel:`APM MetricSets` page, do one of the following: 
+  
+   a. In Splunk APM, select :guilabel:`APM Configuration` and select :guilabel:`APM MetricSets` from the menu. The APM MetricSets page opens.
+   b. From anywhere in Splunk Observability Cloud, select :guilabel:`Settings` in the navigation bar and select :guilabel:`APM MetricSets` under :guilabel:`Data Configuration`.   
+
+2. On the :guilabel:`APM MetricSets` page, scroll to the :guilabel:`Inferred Service Monitoring MetricSets` section and select :guilabel:`Add Inferred Service MMS`.
+3. Select the inferred service for which you want to create a Monitoring MetricSet. 
+4. Select :guilabel:`Save`.
+5. Your new MetricSet shows under :strong:`Pending MetricSets` section of the table with an :guilabel:`Analyzing` status while the cardinality check runs. You might have to wait a few moments for the check to run. 
+
+   .. caution:: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again. 
+
+6.  After the cardinality check has run, review the cardinality and activate your new MetricSet.
+
+    * If the cardinality impact of indexing this new MetricSet is within your subscription limit you see a green check mark and "Within Entitlement." Select the check mark under :guilabel:`Actions` to activate your Monitoring MetricSet. 
+    * If the check times out before you activate the MetricSet, run the analysis again.
+    * If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See :ref:`cardinality-check-mms`. 
 
 .. _cardinality-check-mms:
 
 Cardinality contribution of indexed span tags and processes
 ================================================================
 
-When you index a new span tag or process to generate Monitoring MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span tag or process. This gives you control of what you index and helps you to account for organization subscription limits.
+When you index a new span tag or process to create Monitoring MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span tag or process. This gives you control of what you index and helps you to account for organization subscription limits.
 
-If you try to index a span tag or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes by modifying or removing indexed span tags or processes.
+If you try to index a span tag or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes for instrumented services by modifying or removing indexed span tags or processes.
 
-To see your TMS subscription limit, go to :guilabel:`Settings` then :guilabel:`Subscription Usage`. Depending on your organization subscription, you might need to go to :guilabel:`Settings` then :guilabel:`Billing and Usage`. Select the :guilabel:`APM` tab and then select the :guilabel:`Troubleshooting MetricSets` panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see :ref:`apm-billing-usage-index`.
+To see your TMS subscription limit, go to :guilabel:`Settings` then :guilabel:`Subscription Usage`. Depending on your organization subscription, you might need to go to :guilabel:`Settings` then :guilabel:`Billing and Usage`. Select the :guilabel:`APM` tab and then select the :guilabel:`Monitoring MetricSets` panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see :ref:`apm-billing-usage-index`.
 
 .. _charts-dashboards-alerts:
 
 Set up charts, dashboards, and alerts for custom dimensions
 ==================================================================
 You can create charts, dashboards, and alerts based on your custom Monitoring MetricSet. 
 
-To use the custom dimensionalized Monitoring MetricSets you have created, apply the filter ``sf_dimensionalized:true``. This filters out the metrics generated by the default Monitoring Metricset. To filter your metrics even more, use the new dimension you have created which is the tag name you indexed.
+To use the custom dimensionalized Monitoring MetricSets you have created, apply the filter ``sf_dimensionalized:true``. This filters out the metrics created by the default Monitoring MetricSet. To filter your metrics even more, use the new dimension you have created which is the tag name you indexed.
 
 .. list-table::
    :header-rows: 1