Add troubleshooting guides for SDK and Collector sampling configuration

troubleshoot/ingest/opentelemetry/edot-collector/misconfigured-sampling-collector.md

@@ -0,0 +1,76 @@
+---
+navigation_title: Troubleshoot Collector sampling configuration
+description: Learn how to troubleshoot missing or incomplete traces in the EDOT Collector caused by sampling configuration.
+applies_to:
+  serverless: all
+  product:
+    edot_collector: ga  
+products:
+  - id: observability
+  - id: edot-collector
+---
+
+# Missing or incomplete traces due to Collector sampling
+
+If traces or spans are missing in Kibana, the issue might be related to the Collector’s sampling configuration. Tail-based sampling in the Collector can reduce trace volume if policies are too strict or misconfigured.
+
+Both Collector-based and SDK-level sampling can lead to gaps in telemetry if not configured correctly. See [Missing or incomplete traces due to SDK sampling](../edot-sdks/misconfigured-sampling-sdk.md) for more information.
+
+## Symptoms
+
+- Only a small subset of traces reaches Elasticsearch/Kibana, even though SDKs are exporting spans.
+- Error traces are missing because they’re not explicitly included in the `sampling_policy`.
+- Collector logs show dropped spans.
+
+## Causes
+
+- Tail sampling policies in the Collector are too narrow or restrictive.
+- The default rule set excludes key transaction types (for example long-running requests, non-error transactions).
+- Differences between head sampling (SDK) and tail sampling (Collector) can lead to fewer traces being available for evaluation.
+- Conflicting or overlapping `sampling_policy` rules might result in unexpected drops.
+- High load: the Collector might drop traces if it can’t evaluate policies fast enough.
+
+## Resolution
+
+Follow these steps to resolve sampling configuration issues:
+
+::::{stepper}
+
+:::{step} Review `sampling_policy` configuration
+
+- Check the `processor/tailsampling` section of your Collector configuration
+- Ensure policies are broad enough to capture the traces you need
+:::
+
+:::{step} Add explicit rules for critical traces
+
+- Create specific rules for important trace types
+- Example: keep all error traces, 100% of login requests, and 10% of everything else
+- Use attributes like `status_code`, `operation`, or `service.name` to fine-tune rules
+:::
+
+:::{step} Validate Collector logs
+
+- Review Collector logs for messages about dropped spans, and determine whether drops are due to sampling policy outcomes or resource limits
+:::
+
+:::{step} Differentiate head vs. tail sampling
+
+- Review if SDKs apply head sampling, which reduces traces available for tail sampling
+- Consider setting SDKs to `always_on` and managing sampling centrally in the Collector for more flexibility
+:::
+
+:::{step} Test in staging
+
+- Adjust sampling policies incrementally in a staging environment
+- Monitor trace volume before and after changes
+- Validate that critical traces are captured as expected
+:::
+
+::::
+
+## Resources
+
+- [Tail sampling processor (Collector)](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
+- [OpenTelemetry sampling concepts - contrib documentation](https://opentelemetry.io/docs/concepts/sampling/) 
+- [Missing or incomplete traces due to SDK sampling](../edot-sdks/misconfigured-sampling-sdk.md)

troubleshoot/ingest/opentelemetry/edot-sdks/misconfigured-sampling-sdk.md

@@ -0,0 +1,80 @@
+---
+navigation_title: Troubleshoot SDK sampling configuration
+description: Learn how to troubleshoot missing or incomplete traces in EDOT SDKs caused by head sampling configuration.
+applies_to:
+  serverless: all
+  product:
+    elastic-otel-sdk: ga
+products:
+  - id: observability
+  - id: edot-sdk
+---
+
+# Missing or incomplete traces due to SDK sampling
+
+If traces or spans are missing in Kibana, the issue might be related to SDK-level sampling configuration. By default, SDKs use head-based sampling, meaning the decision to record or drop a trace is made when the trace is first created.
+
+Both SDK-level and Collector-based sampling can result in gaps in telemetry if misconfigured. See [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md) for more details.
+
+## Symptoms
+
+- Only a small subset of traces appears in Kibana, even under light traffic.
+- Transactions look incomplete because some spans are missing.
+- Trace volume is unexpectedly low compared to logs or metrics.
+
+## Causes
+
+- Head sampling at the SDK level drops traces before they're exported.
+- Default sampling rates (for example `1/100` or `1/1000`) might be too low for your workload.
+- Environment variables like `OTEL_TRACES_SAMPLER` or `OTEL_TRACES_SAMPLER_ARG` are not set, not recognized, or formatted in a way the SDK doesn't support.
+- Inconsistent configuration across services can lead to fragmented or incomplete traces.
+- Some SDKs enforce stricter formats for sampler arguments, which can cause values to be ignored if not matched precisely.
+
+## Resolution
+
+Follow these steps to resolve SDK sampling configuration issues:
+
+::::{stepper}
+
+:::{step} Check SDK environment variables
+
+- Confirm that `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are set correctly
+- For testing, you can temporarily set:
+  ```bash
+  export OTEL_TRACES_SAMPLER=always_on
+  ```
+- In production, consider using `parentbased_traceidratio` with an explicit ratio
+:::
+
+:::{step} Align configuration across services
+
+- Use consistent sampling configuration across all instrumented services to help avoid dropped child spans or fragmented traces
+:::
+
+:::{step} Adjust sampling ratios for your traffic
+
+- For low-traffic applications, avoid extremely low ratios (such as `1/1000`) 
+
+    For example, the following configuration samples ~20% of traces:
+
+  ```bash
+  export OTEL_TRACES_SAMPLER=parentbased_traceidratio
+  export OTEL_TRACES_SAMPLER_ARG=0.2
+  ```
+:::
+
+:::{step} Use Collector tail sampling for advanced scenarios
+
+- Head sampling can't evaluate the full trace context before making a decision
+- For more control (for example "keep all errors, sample 10% of successes"), use Collector tail sampling
+
+    For more information, refer to [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md).
+:::
+
+::::
+
+## Resources
+
+- [OTEL_TRACES_SAMPLER environment variable specifications](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#otel_traces_sampler)
+- [OpenTelemetry sampling concepts - contrib documentation](https://opentelemetry.io/docs/concepts/sampling/)
+- [Missing or incomplete traces due to Collector sampling](../edot-collector/misconfigured-sampling-collector.md)

troubleshoot/ingest/opentelemetry/toc.yml

@@ -9,6 +9,7 @@ toc:
       - file: edot-collector/metadata.md
       - file: edot-collector/enable-debug-logging.md
       - file: edot-collector/collector-not-starting.md
+      - file: edot-collector/misconfigured-sampling-collector.md
   - file: edot-sdks/index.md
     children:
       - file: edot-sdks/android/index.md
@@ -23,5 +24,6 @@ toc:
       - file: edot-sdks/enable-debug-logging.md
       - file: edot-sdks/missing-app-telemetry.md
       - file: edot-sdks/proxy.md
+      - file: edot-sdks/misconfigured-sampling-sdk.md
   - file: no-data-in-kibana.md
   - file: contact-support.md