[diagnostics] Add InstrumentAdvice details to instrumentation doc (#43640)

* Add InstrumentAdvice details to instrumentation doc.

* Code review.

* Code review.

* Code review.

* Lint.

* Tweak.

Author: CodeBlanch

docs/core/diagnostics/metrics-instrumentation.md

@@ -2,7 +2,7 @@
 title: Creating Metrics
 description: How to add new metrics to a .NET library or application
 ms.topic: tutorial
-ms.date: 08/27/2024
+ms.date: 11/19/2024
 ---
 
 # Creating metrics
@@ -224,6 +224,11 @@ Types of instruments currently available:
   <xref:System.Diagnostics.Metrics.Histogram%601.Record%2A> to record these measurements during the collection tool's update interval: 1,5,2,3,10,9,7,4,6,8. A collection tool
   might report that the 50th, 90th, and 95th percentiles of these measurements are 5, 9, and 9 respectively.
 
+  > [!NOTE]
+  > For details about how to set the recommended bucket boundaries when creating
+  > a Histogram instrument see: [Using Advice to customize Histogram
+  > instruments](#using-advice-to-customize-histogram-instruments).
+
 ### Best practices when selecting an instrument type
 
 - For counting things, or any other value that solely increases over time, use Counter or ObservableCounter. Choose between Counter and ObservableCounter depending on which
@@ -542,6 +547,95 @@ Name                                                  Current Value
 > [!NOTE]
 > OpenTelemetry refers to tags as 'attributes'. These are two different names for the same functionality.
 
+## Using Advice to customize Histogram instruments
+
+When using Histograms, it is the responsibility of the tool or library
+collecting the data to decide how best to represent the distribution of values
+that were recorded. A common strategy (and the [default mode when using
+OpenTelemetry](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#explicit-bucket-histogram-aggregation))
+is to divide up the range of possible values into sub-ranges called buckets and
+report how many recorded values were in each bucket. For example a tool might
+divide numbers into three buckets, those less than 1, those between 1-10, and
+those greater than 10. If your app recorded the values 0.5, 6, 0.1, 12 then
+there would be two data points the first bucket, one in the second, and one in
+the 3rd.
+
+The tool or library collecting the Histogram data is responsible for defining
+the buckets it will use. The default bucket configuration when using
+OpenTelemetry is: `[ 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000,
+7500, 10000 ]`.
+
+The default values may not lead to the best granularity for every Histogram. For
+example, sub-second request durations would all fall into the `0` bucket.
+
+The tool or library collecting the Histogram data may offer mechanism(s) to
+allow users to customize the bucket configuration. For example, OpenTelemetry
+defines a [View
+API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#view).
+This however requires end user action and makes it the user's responsibility to
+understand the data distribution well enough to choose correct buckets.
+
+To improve the experience the `9.0.0` version of the
+`System.Diagnostics.DiagnosticSource` package introduced the
+(<xref:System.Diagnostics.Metrics.InstrumentAdvice`1>) API.
+
+The `InstrumentAdvice` API may be used by instrumentation authors to specify the
+set of recommended default bucket boundaries for a given Histogram. The tool or
+library collecting the Histogram data can then choose to use those values when
+configuring aggregation leading to a more seamless onboarding experience for
+users. This is supported in the [OpenTelemetry .NET SDK as of version
+1.10.0](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#configuring-the-aggregation-of-a-histogram).
+
+> [!IMPORTANT]
+> In general more buckets will lead to more precise data for a given Histogram
+> but each bucket requires memory to store the aggregated details and there is
+> CPU cost to find the correct bucket when processing a measurement. It is
+> important to understand the tradeoffs between precision and CPU/memory
+> consumption when choosing the number of buckets to recommend via the
+> `InstrumentAdvice` API.
+
+The following code shows an example using the `InstrumentAdvice` API to set
+recommended default buckets.
+
+```csharp
+using System;
+using System.Diagnostics.Metrics;
+using System.Threading;
+
+class Program
+{
+    static Meter s_meter = new Meter("HatCo.Store");
+    static Histogram<double> s_orderProcessingTime = s_meter.CreateHistogram<double>(
+        name: "hatco.store.order_processing_time",
+        unit: "s",
+        description: "Order processing duration",
+        advice: new InstrumentAdvice<double> { HistogramBucketBoundaries = [0.01, 0.05, 0.1, 0.5, 1, 5] });
+
+    static Random s_rand = new Random();
+
+    static void Main(string[] args)
+    {
+        Console.WriteLine("Press any key to exit");
+        while (!Console.KeyAvailable)
+        {
+            // Pretend our store has one transaction each 100ms
+            Thread.Sleep(100);
+
+            // Pretend that we measured how long it took to do the transaction (for example we could time it with Stopwatch)
+            s_orderProcessingTime.Record(s_rand.Next(5, 15) / 1000.0);
+        }
+    }
+}
+```
+
+### Additional information
+
+For more details about explicit bucket Histograms in OpenTelemetry see:
+
+* [Why Histograms?](https://opentelemetry.io/blog/2023/why-histograms/)
+
+* [Histograms vs Summaries](https://opentelemetry.io/blog/2023/histograms-vs-summaries/)
+
 ## Test custom metrics
 
 Its possible to test any custom metrics you add using <xref:Microsoft.Extensions.Diagnostics.Metrics.Testing.MetricCollector%601>. This type makes it easy to record the measurements from specific instruments and assert the values were correct.