feat(tracing): support 128-bit trace ids [does not include support for b3 and w3c] (#5326)

## Description 

Currently Datadog spans are correlated using 64 bit trace ids. However,
as industry standards have sprung up around distributed tracing
([OpenTracing](https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md#trace-context-http-headers),
[OpenCensus](https://github.com/census-instrumentation/opencensus-specs/blob/master/trace/Span.md#traceid),
and now
[OpenTelemetry](https://opentelemetry.io/docs/reference/specification/trace/api/#spancontext))
the accepted standard length for trace IDs has settled at 128 bits. This
PR introduced a configuration which can enable the generation and
propagation of 128bit trace ids
(`DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED`).

With this change 128bit trace ids will be `opt-in`. 128bit trace ids
will become the default standard in a future PR.

## Components

### Format

Current format: trace ids are 64bit integers with the following binary
representation: <64 random bits>
Proposed format: trace ids are 128bit integers with the following binary
representation: `<32-bit unix seconds><32 bits of zero><64 random bits>`

### Encoding

128bit integers are not compatible with ddtrace trace encoders and agent
endpoints. As a workaround 128bit trace ids will be encoded in two
fields.
- The 64 lowest order bits will be stored in the `trace_id` field in
Json and MsgPack Encoders.
- The 64 highest order bits will be encoded into hex and stored as a
span tag with the key `_dd.p.tid`.

#### Example

Span:
`<Span(id=3822442170818112150,trace_id=133030438088573679178230931306392465947,parent_id=None,name=bit128)>`
trace_id as binary:
`01100100000101001011101011111101000000000000000000000000000000001101100111101101100000011001111100101111111100100011001000011011`

trace_id lower order bits:
`1101100111101101100000011001111100101111111100100011001000011011`
trace_id lower order bits as integer: `15703349996414972443`
trace_id higher order bits:
`0110010000010100101110101111110100000000000000000000000000000000`
trace_id higher order bits as hex: `6414bafd00000000`

Encoded Span: `span_id=3822442170818112150` ,
`trace_id=15703349996414972443` , `name=bit128`, ` _meta={"_dd.tid":
"6414bafd00000000")>`

### Distributed Tracing

This PR only adds support for propagating 128bit trace ids using the
datadog propagation mode. Although b3 and w3c trace header formats
support 128bit the Datadog tracer truncates all trace ids to 64 bits.
Moving forward this is unacceptable. Supporting 128bit trace ids in b3
and w3c will be added in a future PR.

#### Datadog Distributed Tracing Headers

Similar to the encoding example above the `x-datadog-trace-id` header
will propagate the 64 lower order bits as an integer and
`x-datadog-tags` header will propagate the higher order bits as hex
(using the `_dd.tid` tag).

##### Example

Span:
`<Span(id=3822442170818112150,trace_id=133030438088573679178230931306392465947,parent_id=None,name=bit128)>`

distributed tracing headers: `{"x-datadog-tags":
"t.tid:6414bafd00000000", "x-datadog-trace-id": 15703349996414972443,
"x-datadog-parent-id": 3822442170818112150}`

### Sampling

The 64 lowest order bits are random but the 64 highest order bits
correspond to the unix time and are not random. When 128bit trace ids
are generated we should only use the lowest order 64 bits (random
component) to determine whether a span should be sampled. This will
ensure when trace ids mapped to values from 0 to 1 we get a uniform
random distribution.

## Testing Strategy

- Run the tracer test suite with
`DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=true`. This will ensure all
tracing operations work as expected when 128bit trace ids are generated
(ex: sampling, distributed tracing, encoding).
- Ensure all 128bit trace id system tests pass.
- Add integration tests for the following scenarios
- 128bit trace ids are encoded without data loss and raising an
OverflowError.
- trace_id field should only contain a 64bit integer and the remaining
bits are stored in the `_dd.p.tid` tag.
- 128bit trace ids are propagated by Datadog distributing tracing
headers.
- Ensure the full 128bit trace id propagated and reconstructed by
downstream services.
  - Ensure Spans with 128bit trace ids are sampled at the expected rate

### Performance Testing

1. There is no performance regression when 128bit trace id generation is
disabled. This is the default mode.
2. When 128bit trace id generation is enabled there is a ~60ns increase
(578ns -> 640ns, 10%) to span creation (`Span.__init__(name, ......)`).
This performance regression does not appear avoidable.
- This overhead was measured on M1 using python 3.8. Results vary across
platforms and python versions.

## Next Steps

1. Support 128bit trace id propagation in b3 and w3c headers
- The code change in straight forward (ie avoid truncating incoming
trace ids) but this change will require a significant refactor of
existing tests.
2. Add support for the `DD_TRACE_128_BIT_TRACEID_LOGGING_ENABLED`
environment variable
- This supports logging 64bit trace ids even when
`DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=true`
4. Enable 128bit trace id on a sample application (ex: internal services
using the ddtrace library)
   - Ensure sampling rates are respected
- Ensure logs correlation works for traces with 128bit trace ids (this
is a concern raised in the RFC)
- Ensure distributed tracing works across tracers that only support
64bit trace ids
- Ensure the full 128bit trace id is reconstructed and is viewable in
the Datadog product
5. Enable support for 128bit trace id by default and add public
documentation.

## Checklist

- [x] Change(s) are motivated and described in the PR description.
- [x] Testing strategy is described if automated tests are not included
in the PR.
- [x] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines)
are followed.
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [x] Author is aware of the performance implications of this PR as
reported in the benchmarks PR comment.

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer is aware of, and discussed the performance implications
of this PR as reported in the benchmarks PR comment.

---------

Co-authored-by: Kyle Verhoog <[email protected]>

Author: mabdinur

ddtrace/internal/_encoding.pyx

@@ -604,7 +604,7 @@ cdef class MsgpackEncoderV03(MsgpackEncoderBase):
         if ret == 0:
             ret = pack_bytes(&self.pk, <char *> b"trace_id", 8)
             if ret != 0: return ret
-            ret = pack_number(&self.pk, span.trace_id)
+            ret = pack_number(&self.pk, span._trace_id_64bits)
             if ret != 0: return ret
 
             if has_parent_id:
@@ -718,7 +718,7 @@ cdef class MsgpackEncoderV05(MsgpackEncoderBase):
         ret = self._pack_string(span.resource)
         if ret != 0: return ret
 
-        _ = span.trace_id
+        _ = span._trace_id_64bits
         ret = msgpack_pack_uint64(&self.pk, _ if _ is not None else 0)
         if ret != 0: return ret
 

ddtrace/internal/_rand.pyi

@@ -1,2 +1,3 @@
 def seed() -> None: ...
 def rand64bits(check_pid: bool = True) -> int: ...
+def rand128bits(check_pid: bool = True) -> int: ...

ddtrace/internal/_rand.pyx

@@ -54,6 +54,8 @@ test_rand64bits_pid_check     121.8156 (2.03)     168.9837 (1.71)     130.3854 (
 import os
 import random
 
+from libc.time cimport time
+
 from ddtrace.internal import compat
 from ddtrace.internal import forksafe
 
@@ -87,4 +89,9 @@ cpdef rand64bits():
     return <uint64_t>(state * <uint64_t>2685821657736338717)
 
 
+cpdef rand128bits():
+    # Returns a 128bit integer with the following format -> <32-bit unix seconds><32 bits of zero><64 random bits>
+    return int(time(NULL)) << 96 | rand64bits()
+
+
 seed()

ddtrace/internal/constants.py

@@ -18,6 +18,8 @@
 DEFAULT_SERVICE_NAME = "unnamed_python_service"
 # Used to set the name of an integration on a span
 COMPONENT = "component"
+HIGHER_ORDER_TRACE_ID_BITS = "_dd.p.tid"
+MAX_UINT_64BITS = (1 << 64) - 1
 
 APPSEC_BLOCKED_RESPONSE_HTML = """
 <!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <meta name="viewport"

ddtrace/internal/encoding.py

@@ -52,9 +52,9 @@ def encode(self, obj):
 
     @staticmethod
     def _span_to_dict(span):
-        # type: () -> Dict[str, Any]
+        # type: (Span) -> Dict[str, Any]
         d = {
-            "trace_id": span.trace_id,
+            "trace_id": span._trace_id_64bits,
             "parent_id": span.parent_id,
             "span_id": span.span_id,
             "service": span.service,

ddtrace/internal/processor/trace.py

@@ -12,13 +12,16 @@
 from ddtrace import config
 from ddtrace.constants import SAMPLING_PRIORITY_KEY
 from ddtrace.constants import USER_KEEP
+from ddtrace.internal.constants import HIGHER_ORDER_TRACE_ID_BITS
+from ddtrace.internal.constants import MAX_UINT_64BITS
 from ddtrace.internal.logger import get_logger
 from ddtrace.internal.processor import SpanProcessor
 from ddtrace.internal.sampling import SpanSamplingRule
 from ddtrace.internal.sampling import is_single_span_sampled
 from ddtrace.internal.service import ServiceStatusError
 from ddtrace.internal.writer import TraceWriter
 from ddtrace.span import Span
+from ddtrace.span import _get_64_highest_order_bits_as_hex
 from ddtrace.span import _is_top_level
 
 
@@ -127,6 +130,10 @@ def process_trace(self, trace):
 
         ctx._update_tags(chunk_root)
         chunk_root.set_tag_str("language", "python")
+        # for 128 bit trace ids
+        if chunk_root.trace_id > MAX_UINT_64BITS:
+            trace_id_hob = _get_64_highest_order_bits_as_hex(chunk_root.trace_id)
+            chunk_root.set_tag_str(HIGHER_ORDER_TRACE_ID_BITS, trace_id_hob)
         return trace
 
 

ddtrace/propagation/http.py

@@ -21,6 +21,8 @@
 from ..internal._tagset import encode_tagset_values
 from ..internal.compat import ensure_str
 from ..internal.compat import ensure_text
+from ..internal.constants import HIGHER_ORDER_TRACE_ID_BITS as _HIGHER_ORDER_TRACE_ID_BITS
+from ..internal.constants import MAX_UINT_64BITS as _MAX_UINT_64BITS
 from ..internal.constants import PROPAGATION_STYLE_B3
 from ..internal.constants import PROPAGATION_STYLE_B3_SINGLE_HEADER
 from ..internal.constants import PROPAGATION_STYLE_DATADOG
@@ -31,6 +33,8 @@
 from ..internal.logger import get_logger
 from ..internal.sampling import validate_sampling_decision
 from ..span import _MetaDictType
+from ..span import _get_64_highest_order_bits_as_hex
+from ..span import _get_64_lowest_order_bits_as_int
 from ._utils import get_wsgi_header
 
 
@@ -154,7 +158,16 @@ def _inject(span_context, headers):
             log.debug("tried to inject invalid context %r", span_context)
             return
 
-        headers[HTTP_HEADER_TRACE_ID] = str(span_context.trace_id)
+        if span_context.trace_id > _MAX_UINT_64BITS:
+            # set lower order 64 bits in `x-datadog-trace-id` header. For backwards compatibility these
+            # bits should be converted to a base 10 integer.
+            headers[HTTP_HEADER_TRACE_ID] = str(_get_64_lowest_order_bits_as_int(span_context.trace_id))
+            # set higher order 64 bits in `_dd.p.tid` to propagate the full 128 bit trace id.
+            # Note - The higher order bits must be encoded in hex
+            span_context._meta[_HIGHER_ORDER_TRACE_ID_BITS] = _get_64_highest_order_bits_as_hex(span_context.trace_id)
+        else:
+            headers[HTTP_HEADER_TRACE_ID] = str(span_context.trace_id)
+
         headers[HTTP_HEADER_PARENT_ID] = str(span_context.span_id)
         sampling_priority = span_context.sampling_priority
         # Propagate priority only if defined
@@ -197,11 +210,18 @@ def _inject(span_context, headers):
     @staticmethod
     def _extract(headers):
         # type: (Dict[str, str]) -> Optional[Context]
-        trace_id = _extract_header_value(
-            POSSIBLE_HTTP_HEADER_TRACE_IDS,
-            headers,
-        )
-        if trace_id is None:
+        trace_id_str = _extract_header_value(POSSIBLE_HTTP_HEADER_TRACE_IDS, headers)
+        if trace_id_str is None:
+            return None
+        try:
+            trace_id = int(trace_id_str)
+        except ValueError:
+            trace_id = 0
+
+        if trace_id == 0 or trace_id > _MAX_UINT_64BITS:
+            log.warning(
+                "Invalid trace id: %r. `x-datadog-trace-id` must be greater than zero and less than 2**64", trace_id_str
+            )
             return None
 
         parent_span_id = _extract_header_value(
@@ -246,6 +266,20 @@ def _extract(headers):
                 }
                 log.debug("failed to decode x-datadog-tags: %r", tags_value, exc_info=True)
 
+        if meta is not None and config._128_bit_trace_id_enabled:
+            # When 128 bit trace ids are propagated the 64 lowest order bits are encoded as an integer
+            # and set in the `x-datadog-trace-id` header (this was done for backwards compatibility).
+            # The 64 highest order bits are encoded in base 16 and store in the `_dd.p.tid` tag.
+            # Here we reconstruct the full 128 bit trace_id.
+            trace_id_hob_hex = meta.get(_HIGHER_ORDER_TRACE_ID_BITS)  # type: Optional[str]
+            if trace_id_hob_hex is not None:
+                # convert lowest order bits in trace_id to base 16
+                trace_id_lod_hex = "{:016x}".format(trace_id)
+                # combine highest and lowest order hex values to create a 128 bit trace_id
+                trace_id = int(trace_id_hob_hex + trace_id_lod_hex, 16)
+                # After the full trace id is reconstructed this tag is no longer required
+                del meta[_HIGHER_ORDER_TRACE_ID_BITS]
+
         # Try to parse values into their expected types
         try:
             if sampling_priority is not None:
@@ -258,7 +292,7 @@ def _extract(headers):
 
             return Context(
                 # DEV: Do not allow `0` for trace id or span id, use None instead
-                trace_id=int(trace_id) or None,
+                trace_id=trace_id or None,
                 span_id=int(parent_span_id) or None,  # type: ignore[arg-type]
                 sampling_priority=sampling_priority,  # type: ignore[arg-type]
                 dd_origin=origin,

ddtrace/sampler.py

@@ -25,6 +25,7 @@
 from .constants import USER_REJECT
 from .internal.compat import iteritems
 from .internal.compat import pattern_type
+from .internal.constants import MAX_UINT_64BITS as _MAX_UINT_64BITS
 from .internal.logger import get_logger
 from .internal.rate_limiter import RateLimiter
 from .internal.sampling import SamplingMechanism
@@ -44,8 +45,11 @@
 
 log = get_logger(__name__)
 
-MAX_TRACE_ID = 2 ** 64
-
+# All references to MAX_TRACE_ID were replaced with _MAX_UINT_64BITS.
+# Now that ddtrace supports generating 128bit trace_ids,
+# the max trace id should be 2**128 - 1 (not 2**64 -1)
+# MAX_TRACE_ID is no longer used and should be removed.
+MAX_TRACE_ID = _MAX_UINT_64BITS
 # Has to be the same factor and key as the Agent to allow chained sampling
 KNUTH_FACTOR = 1111111111111111111
 
@@ -101,11 +105,11 @@ def __init__(self, sample_rate=1.0):
     def set_sample_rate(self, sample_rate):
         # type: (float) -> None
         self.sample_rate = float(sample_rate)
-        self.sampling_id_threshold = self.sample_rate * MAX_TRACE_ID
+        self.sampling_id_threshold = self.sample_rate * _MAX_UINT_64BITS
 
     def sample(self, span):
         # type: (Span) -> bool
-        return ((span.trace_id * KNUTH_FACTOR) % MAX_TRACE_ID) <= self.sampling_id_threshold
+        return ((span._trace_id_64bits * KNUTH_FACTOR) % _MAX_UINT_64BITS) <= self.sampling_id_threshold
 
 
 class RateByServiceSampler(BasePrioritySampler):
@@ -431,7 +435,7 @@ def sample_rate(self):
     def sample_rate(self, sample_rate):
         # type: (float) -> None
         self._sample_rate = sample_rate
-        self._sampling_id_threshold = sample_rate * MAX_TRACE_ID
+        self._sampling_id_threshold = sample_rate * _MAX_UINT_64BITS
 
     def _pattern_matches(self, prop, pattern):
         # If the rule is not set, then assume it matches
@@ -501,7 +505,7 @@ def sample(self, span):
         elif self.sample_rate == 0:
             return False
 
-        return ((span.trace_id * KNUTH_FACTOR) % MAX_TRACE_ID) <= self._sampling_id_threshold
+        return ((span._trace_id_64bits * KNUTH_FACTOR) % _MAX_UINT_64BITS) <= self._sampling_id_threshold
 
     def _no_rule_or_self(self, val):
         return "NO_RULE" if val is self.NO_RULE else val

ddtrace/settings/config.py

@@ -225,6 +225,8 @@ def __init__(self):
 
         self._telemetry_metrics_enabled = asbool(os.getenv("_DD_TELEMETRY_METRICS_ENABLED", default=False))
 
+        self._128_bit_trace_id_enabled = asbool(os.getenv("DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED", False))
+
         # Propagation styles
         self._propagation_style_extract = self._propagation_style_inject = _parse_propagation_styles(
             "DD_TRACE_PROPAGATION_STYLE", default=_PROPAGATION_STYLE_DEFAULT

ddtrace/span.py

@@ -28,7 +28,8 @@
 from .context import Context
 from .ext import http
 from .ext import net
-from .internal import _rand
+from .internal._rand import rand64bits as _rand64bits
+from .internal._rand import rand128bits as _rand128bits
 from .internal.compat import NumericType
 from .internal.compat import StringIO
 from .internal.compat import ensure_text
@@ -37,6 +38,7 @@
 from .internal.compat import numeric_types
 from .internal.compat import stringify
 from .internal.compat import time_ns
+from .internal.constants import MAX_UINT_64BITS as _MAX_UINT_64BITS
 from .internal.logger import get_logger
 from .internal.sampling import SamplingMechanism
 from .internal.sampling import update_sampling_decision
@@ -50,6 +52,18 @@
 log = get_logger(__name__)
 
 
+def _get_64_lowest_order_bits_as_int(large_int):
+    # type: (int) -> int
+    """Get the 64 lowest order bits from a 128bit integer"""
+    return _MAX_UINT_64BITS & large_int
+
+
+def _get_64_highest_order_bits_as_hex(large_int):
+    # type: (int) -> str
+    """Get the 64 highest order bits from a 128bit integer"""
+    return "{:032x}".format(large_int)[:16]
+
+
 class Span(object):
 
     __slots__ = [
@@ -137,8 +151,13 @@ def __init__(
         self.duration_ns = None  # type: Optional[int]
 
         # tracing
-        self.trace_id = trace_id or _rand.rand64bits()  # type: int
-        self.span_id = span_id or _rand.rand64bits()  # type: int
+        if trace_id is not None:
+            self.trace_id = trace_id  # type: int
+        elif config._128_bit_trace_id_enabled:
+            self.trace_id = _rand128bits()
+        else:
+            self.trace_id = _rand64bits()
+        self.span_id = span_id or _rand64bits()  # type: int
         self.parent_id = parent_id  # type: Optional[int]
         self._on_finish_callbacks = [] if on_finish is None else on_finish
 
@@ -176,6 +195,10 @@ def _get_ctx_item(self, key):
             return None
         return self._store.get(key)
 
+    @property
+    def _trace_id_64bits(self):
+        return _get_64_lowest_order_bits_as_int(self.trace_id)
+
     @property
     def start(self):
         # type: () -> float