Add troubleshooting guides for SDK and Collector sampling configuration

troubleshoot/ingest/opentelemetry/edot-collector/misconfigured-sampling-collector.md

@@ -0,0 +1,82 @@
+---
+navigation_title: Collector sampling issues
+description: Learn how to troubleshoot missing or incomplete traces in the EDOT Collector caused by sampling configuration.
+applies_to:
+  serverless: all
+  product:
+    edot_collector: ga  
+products:
+  - id: observability
+  - id: edot-collector
+---
+
+# Missing or incomplete traces due to Collector sampling
+
+If traces or spans are missing in {{kib}}, the issue might be related to the Collector’s sampling configuration. 
+
+{applies_to}`stack: ga 9.2` Tail-based sampling (TBS) allows the Collector to evaluate entire traces before deciding whether to keep them. If TBS policies are too strict or not aligned with your workloads, traces you expect to see may be dropped.
+
+Both Collector-based and SDK-level sampling can lead to gaps in telemetry if not configured correctly. See [Missing or incomplete traces due to SDK sampling](../edot-sdks/misconfigured-sampling-sdk.md) for more information.
+
+## Symptoms
+
+When Collector-based tail sampling is misconfigured or too restrictive, you might observe the following:
+
+- Only a small subset of traces reaches {{es}}/{{kib}}, even though SDKs are exporting spans.
+- Error traces are missing because they’re not explicitly included in the `sampling_policy`.
+- Collector logs show dropped spans.
+
+## Causes
+
+The following conditions can lead to missing or incomplete traces when using tail-based sampling in the Collector:
+
+- Tail sampling policies in the Collector are too narrow or restrictive.
+- The default rule set excludes key transaction types (for example long-running requests, non-error transactions).
+- Differences between head sampling (SDK) and tail sampling (Collector) can lead to fewer traces being available for evaluation.
+- Conflicting or overlapping `sampling_policy` rules might result in unexpected drops.
+- High load: the Collector might drop traces if it can’t evaluate policies fast enough.
+
+## Resolution
+
+Follow these steps to resolve sampling configuration issues:
+
+::::{stepper}
+
+:::{step} Review `sampling_policy` configuration
+
+- Check the `processor/tailsampling` section of your Collector configuration
+- Ensure policies are broad enough to capture the traces you need
+:::
+
+:::{step} Add explicit rules for critical traces
+
+- Create specific rules for important trace types
+- Example: keep all error traces, 100% of login requests, and 10% of everything else
+- Use attributes like `status_code`, `operation`, or `service.name` to fine-tune rules
+:::
+
+:::{step} Validate Collector logs
+
+- Review Collector logs for messages about dropped spans, and determine whether drops are due to sampling policy outcomes or resource limits
+:::
+
+:::{step} Differentiate head and tail sampling
+
+- Review if SDKs already applies head sampling, which reduces traces available for tail sampling in the Collector
+- Consider setting SDKs to `always_on` and managing sampling centrally in the Collector for more flexibility
+:::
+
+:::{step} Test in staging
+
+- Adjust sampling policies incrementally in a staging environment
+- Monitor trace volume before and after changes
+- Validate that critical traces are captured as expected
+:::
+
+::::
+
+## Resources
+
+- [Tail sampling processor (Collector)](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
+- [OpenTelemetry sampling concepts - contrib documentation](https://opentelemetry.io/docs/concepts/sampling/) 
+- [Missing or incomplete traces due to SDK sampling](../edot-sdks/misconfigured-sampling-sdk.md)

troubleshoot/ingest/opentelemetry/edot-sdks/misconfigured-sampling-sdk.md

@@ -0,0 +1,85 @@
+---
+navigation_title: SDK sampling issues
+description: Learn how to troubleshoot missing or incomplete traces in EDOT SDKs caused by head sampling configuration.
+applies_to:
+  serverless: all
+  product:
+    elastic-otel-sdk: ga
+products:
+  - id: observability
+  - id: edot-sdk
+---
+
+# Missing or incomplete traces due to SDK sampling
+
+If traces or spans are missing in Kibana, the issue might be related to SDK-level sampling configuration. By default, SDKs use head-based sampling, meaning the decision to record or drop a trace is made when the trace is first created.
+
+Both SDK-level and Collector-based sampling can result in gaps in telemetry if misconfigured. Refer to [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md) for more details.
+
+## Symptoms
+
+You might notice one or more of the following behaviors when SDK-level sampling is impacting your traces:
+
+- Only a small subset of traces reaches {{es}} or {{kib}}, even though SDKs are exporting spans.
+- Transactions look incomplete because some spans are missing.
+- Trace volume is unexpectedly low compared to logs or metrics.
+
+## Causes
+
+These factors can result in missing spans or traces when sampling is configured at the SDK level:
+
+- Head sampling at the SDK level drops traces before they're exported.
+- Default sampling rates (for example `1/100` or `1/1000`) might be too low for your workload.
+- Environment variables like `OTEL_TRACES_SAMPLER` or `OTEL_TRACES_SAMPLER_ARG` are not set, not recognized, or formatted in a way the SDK doesn't support.
+- Inconsistent configuration across services can lead to fragmented or incomplete traces.
+- Some SDKs enforce stricter formats for sampler arguments, which can cause values to be ignored if not matched precisely.
+
+## Resolution
+
+Follow these steps to resolve SDK sampling configuration issues:
+
+::::{stepper}
+
+:::{step} Check SDK environment variables
+
+- Confirm that `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are set correctly.
+- For testing, you can temporarily set:
+
+  ```bash
+  export OTEL_TRACES_SAMPLER=always_on
+  ```
+- In production, consider using `parentbased_traceidratio` with an explicit ratio.
+:::
+
+:::{step} Align configuration across services
+
+- Use consistent sampling configuration across all instrumented services to help avoid dropped child spans or fragmented traces.
+:::
+
+:::{step} Adjust sampling ratios for your traffic
+
+- For low-traffic applications, avoid extremely low ratios (such as `1/1000`). 
+
+    For example, the following configuration samples ~20% of traces:
+
+  ```bash
+  export OTEL_TRACES_SAMPLER=parentbased_traceidratio
+  export OTEL_TRACES_SAMPLER_ARG=0.2
+  ```
+:::
+
+:::{step} Use Collector tail sampling for advanced scenarios
+
+- Head sampling can't evaluate the full trace context before making a decision.
+- For more control (for example "keep all errors, sample 10% of successes"), use Collector tail sampling.
+
+    For more information, refer to [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md).
+:::
+
+::::
+
+## Resources
+
+- [OTEL_TRACES_SAMPLER environment variable specifications](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#otel_traces_sampler)
+- [OpenTelemetry sampling concepts - contrib documentation](https://opentelemetry.io/docs/concepts/sampling/)
+- [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md)

troubleshoot/ingest/opentelemetry/toc.yml

@@ -9,6 +9,7 @@ toc:
       - file: edot-collector/metadata.md
       - file: edot-collector/enable-debug-logging.md
       - file: edot-collector/collector-not-starting.md
+      - file: edot-collector/misconfigured-sampling-collector.md
   - file: edot-sdks/index.md
     children:
       - file: edot-sdks/android/index.md
@@ -23,5 +24,6 @@ toc:
       - file: edot-sdks/enable-debug-logging.md
       - file: edot-sdks/missing-app-telemetry.md
       - file: edot-sdks/proxy.md
+      - file: edot-sdks/misconfigured-sampling-sdk.md
   - file: no-data-in-kibana.md
   - file: contact-support.md