Merge pull request #2770 from splunk/adasplunk-O11YDOCS-6517-new

[O11YDOCS-6517-new] More updates and review comments

Author: adasplunk

private-preview/aopt/aopt-derived-metrics.rst

@@ -28,6 +28,12 @@ The ratio of how many days of information is available for a workload compared t
 Application Optimization calculates an overall confidence level by taking the lowest confidence level across all containers, where each container's confidence level is an average of the separate confidence levels for CPU and memory.
 
 
+Why the confidence level matters
+----------------------------------------------------------
+
+It's a good idea to match the confidence level to your workload's importance or criticality. In other words, if your workload is a test or you just need to preview the recommendations, a confidence level of :guilabel:`Low` is okay. But if your workload is a production or business critical workload, it's best to wait for a confidence level of :guilabel:`High` before applying the recommendations.
+
+
 .. _aopt-glossary-efficiency:
 
 Efficiency
@@ -37,9 +43,19 @@ The balance between over-provisioning and under-provisioning to optimize resourc
 
 Application Optimization is a powerful tool for achieving and maintaining efficiency. It calculates efficiency as the average of the pod-wide usage of a resource's ``request`` setting, capped at 100%. Its calculation only includes metrics within the analysis window, which is the lesser of 14 days and the time since the last resource change (or the initial deployment). Note that rather than finding the utilization (usage over requests) of each container within a pod, all of the containers' usage and requests are added up first. The averages for each CPU and memory ``request`` setting are then weight-averaged based on the assumed resource cost weights.
 
+Best practices call for resource utilization in the 60-80% range. Having efficiency above 70-80% presents resource starvation risks. 
+
 When values are unset for a particular resource, this tool assumes those ``request`` settings to be at usage (in other words, 100% efficient) to more accurately weigh multi-container rates.
 
-When the main container has an unset resource, this tool considers the efficiency rate to be nullified.
+When the main container has an unset resource, this tool considers the efficiency rate to be undefined.
+
+
+.. _aopt-glossary-resource-footprint:
+
+Resource Footprint
+==========================================================
+
+A workload's resource footprint is the sum of its pods' ``request`` settings for that resource or its average usage if it exceeds its ``request`` settings. If the ``request`` value is not set, the footprint represents the sum of actual usage instead. This tile displays the sum of all resource footprints of all the pods of all your workloads. 
 
 
 .. _aopt-glossary-starvation-risk:
@@ -49,12 +65,17 @@ Starvation risk
 
 A workload's average risk of running out of CPU or memory:
 
-* :guilabel:`High`: Any container in which usage is greater than or equal to 95% of its ``limit`` settings.
+* :guilabel:`High`: The workload has tried to use more resources than were available, so its performance and reliability have likely been impacted. Application Optimization marks any container in which usage is greater than or equal to 95% of its ``limit`` settings as :guilabel:`High`.
 
-* :guilabel:`Medium`: At least one resource (CPU or memory) of one container is not defined OR (all ``request`` settings are defined AND actual usage of at least one resource of one container exceeds its ``request`` setting for any time slot).
+* :guilabel:`Medium`: The workload has used more than its allocated resources (``request`` settings). While this may not have an impact on its performance and reliability due to Kubernetes bursting into additional resources, future occurrences of overusage may have an impact, since extra resources are not guaranteed to exist. 
+  
+  Application Optimization sets :guilabel:`Starvation risk` to :guilabel:`Medium` for any container in which either of these is true:
+  
+    * At least one resource (CPU or memory) of is undefined.
+    
+    * All ``request`` settings are defined and actual usage of at least one resource of exceeds its ``request`` setting for any time slot.
 
-* :guilabel:`Low`: For either CPU or memory, the recommendation is greater than the baseline value. For example, the usage is greater than target utilization (0.85).
+* :guilabel:`Low`: The workload hasn't exceeded its allocated resources but doesn't have enough headroom to absorb spikes or delays in scale-out when traffic increases. Application Optimization marks any container in which, for either CPU or memory, the recommendation is greater than the baseline value. For example, the usage is greater than target utilization (0.85).
 
 * :guilabel:`Minimal`: None of the above conditions are detected. In other words, all containers have ``request`` settings for both CPU and memory, and neither of these resources has had usage exceeding its target utilization. 
 
-

private-preview/aopt/aopt-glossary.rst

@@ -17,9 +17,12 @@ By using Application Optimization together with :new-page:`Splunk Infrastructure
 Key features
 ==========================================================
 
-* :guilabel:`Kubernetes Profiler` (the :guilabel:`Application Optimization` dashboard) provides comprehensive insights into the efficiency of your CPU and memory settings for your Kubernetes workloads.	
+* :guilabel:`Kubernetes Profiler` (the :guilabel:`Application Optimization` dashboard) provides insights into the efficiency of your CPU and memory settings for your Kubernetes workloads. You can use the profiler to:
 
-* :guilabel:`Instant Recommendations` provide suggestions for CPU and memory settings based on historical utilization data (metrics you've sent to Splunk IM). You can apply these suggestions directly to your pods using the YAML snippets it provides. 
+   * Obtain insight summaries by applying filters of your choice (environment, cluster, namespace, and so on).
+   * Find the workloads most in need of attention based on your priorities, which could be performance, reliability, or cost savings.
+
+* :guilabel:`Instant Recommendations` provide suggestions for CPU and memory settings based on historical utilization across all pods of a workload. Utilization data comes from the metrics you've sent to Splunk IM. You can apply these suggestions directly to your workloads using the YAML snippets it provides. For workloads that use autoscaling, it also suggests how to update the autoscaling configuration.
 
 
 Requirements
@@ -33,7 +36,7 @@ Requirements
 
 * All metrics that the :new-page:`Splunk IM Kubernetes cluster receiver collects by default <https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-kubernetes/install-k8s.html#helm-chart-supported-distros>` must be present in your data. Since these metrics are enabled by default on your Kubernetes collector you don't need to take any action unless you've disabled them. 
 
-* Horizontal pod autoscaler (HPA) telemetry: Optional, but if you do have HPAs and you send :new-page:`k8s.hpa.* metrics <https://docs.splunk.com/observability/en/gdi/opentelemetry/components/kubernetes-cluster-receiver.html>` to Splunk IM, :guilabel:`Instant Recommendations` can help you to improve them.
+* Horizontal pod autoscaler (HPA) telemetry: Optional, but if you do have HPAs and you send :new-page:`k8s.hpa.* metrics <https://docs.splunk.com/observability/en/gdi/opentelemetry/components/kubernetes-cluster-receiver.html>` to Splunk IM, :guilabel:`Instant Recommendations` can help you to improve them.  See :ref:`aopt-workload-hpa`.
 
 
 Enable Application Optimization

private-preview/aopt/aopt-intro.rst

@@ -0,0 +1,303 @@
+:orphan:
+
+.. _aopt-metric-reference:
+
+.. include:: /private-preview/aopt/toc.rst
+    :start-after: :orphan:
+
+**********************************************************
+Metric reference
+**********************************************************
+
+Application Optimization's workload analysis produces the following metics. All metrics have at least the same dimensions as the workload metrics (for example ``aws-region`` and so on) and use the same attribute names and values.
+
+.. find out what the prefix is and add it to the metric name. ask daniel for the name.
+
+All metric names have a prefix of either ``sf`` or ``o11y``.
+
+.. note::
+    Memory is specified in GiB. CPU is specified in cores.
+
+
+
+.. list-table::
+   :widths: 40 5 55
+   :width: 100%
+   :header-rows: 1
+
+   - 
+
+      - **Metric**
+      - **Scope\***
+      - **Description**
+   - 
+
+      - ``sf.report.available``
+      - W
+      - Synthetic metric. The value is ``0`` for failed, ``1`` for success. This metric may have additional attributes that represent the report outcome as a whole. At least a ``aopt.profile_report.error_reason`` code.
+   - 
+
+      - ``sf.report.window_days``
+      - W
+      - Number of days (possibly fractional) that were considered in the analysis. In general, this is the smaller of 14 and the number of days since the last resource configuration change for the workload. This is used to determine the validity and confidence level of the report.
+   - 
+
+      - ``sf.report.coverage_ratio``
+      - W
+      - Window coverage with metrics: the ratio of number of actual metrics values found compared to the number of timeslots in the window. This should represent the worst case value (in other words, the minimum of the coverage of each input timeseries we use). This is used to determine the validity and confidence level of the report.
+   - 
+
+      - ``sf.report.average_replicas``
+      - W
+      - Average number of replicas during the analysis window. Does not include pods that allocate resources, such as those scheduled but not started.
+   - 
+
+      - ``sf.report.pod.qos_class``
+      - W
+      - Pod's quality of service (QoS) class, as defined in Kubernetes docs, encoded as an integer.
+   - 
+
+      - ``sf.report.footprint.cpu_cores``
+      - W
+      - Number of the allocated CPU cores for all replicas (averaged based on ``average_replicas``). Does not account for usage above request (bursting).
+   - 
+
+      - ``sf.report.footprint.memory_gib``
+      - W
+      - GiB allocated memory for all replicas (averaged based on the average_replicas). Does not account for usage above request (bursting).
+   - 
+
+      - ``sf.report.efficiency_rate``
+      - W
+      - Resource efficiency rate, as percent. Weighted average of resource utilization of CPU and memory. CPU and memory weights according to AWS on-demand cost. Capped at 100%, rounded to whole percent.
+   - 
+
+      - ``sf.report.starvation_risk``
+      - W
+      - Resource starvation risk: Minimal, Low, Medium, High (encoded as ``0``, ``1``, ``2``, ``3`` respectively). 
+         Risk levels defined elsewhere:
+          - Minimal: no starvation detected
+          - Low: could benefit from more overhead
+          - Medium: actually bursting but not being limited
+          - High: CPU throttled and/or at resource limits.
+   - 
+
+      - ``sf.recommendation.available``
+      - W
+      - Indicates whether a recommendation is available for at least one container. This value is ``0`` or ``1``.
+   - 
+
+      - ``sf.recommendation.confidence_level``
+      - W
+      - Recommendations overall confidence level: Unknown, Low, Medium, High (encoded as ``0``, ``1``, ``2``, ``3`` respectively). Aggregated from ``container.confidence_level``, by taking the lowest confidence value (or the confidence value of the main or largest container).
+   - 
+
+      - ``sf.recommendation.container.available``
+      - C
+      - Indicates whether a recommendation is available: ``0`` or ``1``. A recommendation that matches the baseline is considered available.
+   - 
+
+      - ``sf.recommendation.container.confidence_level``
+      - C
+      - Recommendation confidence level: Unknown, Low, Medium, High (encoded as ``0``, ``1``, ``2``, ``3`` respectively).
+   - 
+
+      - ``sf.recommendation.container.cpu_request``
+      - C
+      - Per-container recommendation.
+   - 
+
+      - ``sf.recommendation.container.memory_request``
+      - C
+      - Per-container recommendation.
+   - 
+
+      - ``sf.recommendation.container.cpu_limit``
+      - C
+      - Per-container recommendation.
+   - 
+
+      - ``sf.recommendation.container.memory_limit``
+      - C
+      - Per-container recommendation.
+   - 
+
+      - ``sf.recommendation.footprint.cpu_cores``
+      - W
+      - Total footprint of recommendation.
+   - 
+
+      - ``sf.recommendation.footprint.memory_gib``
+      - W
+      - Total footprint of recommendation.
+   - 
+
+      - ``sf.recommendation.footprint_change.cpu_cores``
+      - W
+      - Footprint change of CPU requests, assuming the CPU request recommendations are applied for all containers. May be ``0``, ``missing``, or ``NaN`` if requests are not defined.
+   - 
+
+      - ``sf.recommendation.footprint_change.memory_gib``
+      - W
+      - Footprint change of memory requests, assuming the memory request recommendations are applied for all containers. May be ``0``, ``missing``, or ``NaN`` if requests are not defined.
+   - 
+
+      - ``sf.baseline.pod.cpu_request``
+      - W
+      - Pod-level sum of the baseline for the configuration being analyzed. The ``request`` for a container is considered defined if its ``limit`` is defined, even if the ``request`` is reported as missing or ``0``.
+   - 
+
+      - ``sf.baseline.pod.memory_request``
+      - W
+      - Pod-level sum of the baseline for the configuration being analyzed.  The ``request`` for a container is considered defined if its ``limit`` is defined, even if the ``request`` is reported as missing or ``0``.
+   - 
+
+      - ``sf.baseline.pod.cpu_limit``
+      - W
+      - Pod-level sum of the baseline for the configuration being analyzed. This value is ``0`` or ``NaN`` if at least one ``limit`` is missing; as a result, the whole pod doesn't have a ``limit`` for this resource.
+   - 
+
+      - ``sf.baseline.pod.memory_limit``
+      - W
+      - Pod-level sum of the baseline for the configuration being analyzed. This value is ``0`` or ``NaN`` if at least one ``limit`` is missing; as a result, the whole pod doesn't have a ``limit`` for this resource.
+   - 
+
+      - ``sf.baseline.container.cpu_request``
+      - C
+      - Per-container baseline for the configuration being analyzed.
+   - 
+
+      - ``sf.baseline.container.memory_request``
+      - C
+      - Per-container baseline for the configuration being analyzed.
+   - 
+
+      - ``sf.baseline.container.cpu_limit``
+      - C
+      - Per-container baseline for the configuration being analyzed.
+   - 
+
+      - ``sf.baseline.container.memory_limit``
+      - C
+      - Per-container baseline for the configuration being analyzed.
+
+
+
+\*Scope is W for workload and C for container. See :ref:`Dimensions <aopt-derived-metrics_dimensions>` for attributes that apply to each scope.
+
+
+
+.. _aopt-derived-metrics_dimensions:
+
+Dimensions
+==========================================================
+
+
+Workload-level attributes
+----------------------------------------------------------
+
+The following dimensions are applied to all metrics (both workload and container scope):
+
+.. list-table::
+   :widths: 40 60
+   :width: 100%
+   :header-rows: 1
+
+   - 
+
+      - **Attribute name**
+      - **Description**
+   - 
+
+      - ``environment``
+      - Splunk Observability Cloud-specific attribute.
+   - 
+
+      - ``k8s.cluster.name``
+      - Kubernetes cluster name.
+   - 
+
+      - ``k8s.namespace.name``
+      - 
+   - 
+
+      - ``k8s.workload.name``
+      - This is our own generic workload info.
+   - 
+
+      - ``k8s.workload.kind``
+      - Kind of workload: ``deployment``, ``statefulset`` or ``daemonset``. This is our own generic workload info.
+   - 
+
+      - ``k8s.workload.uid``
+      - This is our own generic workload info.
+   - 
+
+      - ``k8s.deployment.name``
+      - Present only for ``workload.kind`` == ``deployment``. Same as ``k8s.workload.name``.
+   - 
+
+      - ``k8s.deployment.uid``
+      - Present only for ``workload.kind`` == ``deployment``. Same as ``k8s.object_uid``.
+   - 
+
+      - ``k8s.statefulset.name``
+      - Present only for ``workload.kind`` == ``statefulset``. Same as ``k8s.workload.name``.
+   - 
+
+      - ``k8s.statefulset.uid``
+      - Present only for ``workload.kind`` == ``statefulset``. Same as ``k8s.object_uid``.
+   - 
+
+      - ``k8s.daemonset.name``
+      - Present only for ``workload.kind`` == ``daemonset``. Same as ``k8s.workload.name``.
+   - 
+
+      - ``k8s.daemonset.uid``
+      - Present only for ``workload.kind`` == ``daemonset``. Same as ``k8s.object_uid``.
+   - 
+
+      - ``k8s.pod.qos``
+      - Pod-level QoS
+   - 
+
+      - ``aopt.profiler_report.success``
+      - Whether the analysis was successful and a report is provided. Values: ``0`` or ``1``.
+   - 
+
+      - ``aopt.instant_recommendation.present``
+      - Whether there is a valid recommendation. Values: ``0`` or ``1``.
+
+
+
+
+
+Container-level attributes
+----------------------------------------------------------
+
+The following additional dimensions are applied to per-container metrics (in other words, any metric named ``*.container.*``):
+
+
+.. list-table::
+   :widths: 40 60
+   :width: 100%
+   :header-rows: 1
+
+   - 
+
+      - **Attribute name**
+      - **Description**
+
+   - 
+
+      - ``k8s.container.name``
+      - 
+   - 
+
+      - ``k8s.container.pseudo_qos``
+      - Container-level pseudo-QoS.
+
+
+.. note:: 
+   This set of additional attributes matches the set of additional attributes that per-container ``k8s`` metrics (such as memory and CPU utilization), provide on top of workload-level metrics (such as replica count). This excludes metadata attributes that are per pod instance (such ``as k8s.replica.set`` and ``k8s.pod.id``, since we always aggregate metrics across instances), as well as per container instance (such as ``k8s.container.id``) for the same reason.
+