aiohttp-server: add support for headers capture for request and response (open-telemetry#3916)

* Collect request attributes once

* Doen't add request attributes to span if it is not recording

While at it also avoid copying empty dictionaries

* Add support for collecting custom request and response headers

* update type annotations

* Add changelog

* Add PR number in changelog

* Add documentation

* Apply suggestions from code review

Co-authored-by: Tammy Baylis <[email protected]>

---------

Co-authored-by: Tammy Baylis <[email protected]>

Author: xrmx

CHANGELOG.md

@@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   ([#3885](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3885))
 - `opentelemetry-instrumentation-django`: improve readthedocs for sqlcommenter configuration.
   ([#3884](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3884))
+- `opentelemetry-instrumentation-aiohttp-server`: add support for custom header captures via `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST` and `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`
+  ([#3916](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3916))
 
 ### Fixed
 

instrumentation/opentelemetry-instrumentation-aiohttp-server/src/opentelemetry/instrumentation/aiohttp_server/__init__.py

@@ -54,11 +54,104 @@ async def hello(request):
 
 will exclude requests such as ``https://site/client/123/info`` and ``https://site/xyz/healthcheck``.
 
+Capture HTTP request and response headers
+*****************************************
+You can configure the agent to capture specified HTTP headers as span attributes, according to the
+`semantic conventions <https://github.com/open-telemetry/semantic-conventions/blob/main/docs/http/http-spans.md#http-server-span>`_.
+
+Request headers
+***************
+To capture HTTP request headers as span attributes, set the environment variable
+``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to a comma delimited list of HTTP header names.
+
+For example,
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="content-type,custom_request_header"
+
+will extract ``content-type`` and ``custom_request_header`` from the request headers and add them as span attributes.
+
+Request header names in aiohttp are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment
+variable will capture the header named ``custom-header``.
+
+Regular expressions may also be used to match multiple headers that correspond to the given pattern.  For example:
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="Accept.*,X-.*"
+
+Would match all request headers that start with ``Accept`` and ``X-``.
+
+To capture all request headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to ``".*"``.
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST=".*"
+
+The name of the added span attribute will follow the format ``http.request.header.<header_name>`` where ``<header_name>``
+is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
+list containing the header values.
+
+For example:
+``http.request.header.custom_request_header = ["<value1>, <value2>"]``
+
+Response headers
+****************
+To capture HTTP response headers as span attributes, set the environment variable
+``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to a comma delimited list of HTTP header names.
+
+For example,
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="content-type,custom_response_header"
+
+will extract ``content-type`` and ``custom_response_header`` from the response headers and add them as span attributes.
+
+Response header names in aiohttp are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment
+variable will capture the header named ``custom-header``.
+
+Regular expressions may also be used to match multiple headers that correspond to the given pattern.  For example:
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="Content.*,X-.*"
+
+Would match all response headers that start with ``Content`` and ``X-``.
+
+To capture all response headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to ``".*"``.
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE=".*"
+
+The name of the added span attribute will follow the format ``http.response.header.<header_name>`` where ``<header_name>``
+is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
+list containing the header values.
+
+For example:
+``http.response.header.custom_response_header = ["<value1>, <value2>"]``
+
+Sanitizing headers
+******************
+In order to prevent storing sensitive data such as personally identifiable information (PII), session keys, passwords,
+etc, set the environment variable ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS``
+to a comma delimited list of HTTP header names to be sanitized.  Regexes may be used, and all header names will be
+matched in a case-insensitive manner.
+
+For example,
+::
+
+    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS=".*session.*,set-cookie"
+
+will replace the value of headers such as ``session-id`` and ``set-cookie`` with ``[REDACTED]`` in the span.
+
+Note:
+    The environment variable names used to capture HTTP headers are still experimental, and thus are subject to change.
+
+API
+---
 """
 
+from __future__ import annotations
+
 import urllib
 from timeit import default_timer
-from typing import Dict, List, Tuple, Union
 
 from aiohttp import web
 from multidict import CIMultiDictProxy
@@ -91,7 +184,17 @@ async def hello(request):
 )
 from opentelemetry.semconv.metrics import MetricInstruments
 from opentelemetry.trace.status import Status, StatusCode
-from opentelemetry.util.http import get_excluded_urls, redact_url
+from opentelemetry.util.http import (
+    OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS,
+    OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST,
+    OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE,
+    SanitizeValue,
+    get_custom_headers,
+    get_excluded_urls,
+    normalise_request_header_name,
+    normalise_response_header_name,
+    redact_url,
+)
 
 _duration_attrs = [
     HTTP_METHOD,
@@ -134,15 +237,15 @@ def _parse_active_request_count_attrs(req_attrs):
     return active_requests_count_attrs
 
 
-def get_default_span_details(request: web.Request) -> Tuple[str, dict]:
+def get_default_span_name(request: web.Request) -> str:
     """Default implementation for get_default_span_details
     Args:
         request: the request object itself.
     Returns:
-        a tuple of the span name, and any attributes to attach to the span.
+        The span name.
     """
     span_name = request.path.strip() or f"HTTP {request.method}"
-    return span_name, {}
+    return span_name
 
 
 def _get_view_func(request: web.Request) -> str:
@@ -158,7 +261,7 @@ def _get_view_func(request: web.Request) -> str:
         return "unknown"
 
 
-def collect_request_attributes(request: web.Request) -> Dict:
+def collect_request_attributes(request: web.Request) -> dict:
     """Collects HTTP request attributes from the ASGI scope and returns a
     dictionary to be used as span creation attributes."""
 
@@ -203,6 +306,42 @@ def collect_request_attributes(request: web.Request) -> Dict:
     return result
 
 
+def collect_request_headers_attributes(
+    request: web.Request,
+) -> dict[str, list[str]]:
+    sanitize = SanitizeValue(
+        get_custom_headers(
+            OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS
+        )
+    )
+
+    return sanitize.sanitize_header_values(
+        request.headers,
+        get_custom_headers(
+            OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST
+        ),
+        normalise_request_header_name,
+    )
+
+
+def collect_response_headers_attributes(
+    response: web.Response,
+) -> dict[str, list[str]]:
+    sanitize = SanitizeValue(
+        get_custom_headers(
+            OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS
+        )
+    )
+
+    return sanitize.sanitize_header_values(
+        response.headers,
+        get_custom_headers(
+            OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE
+        ),
+        normalise_response_header_name,
+    )
+
+
 def set_status_code(span, status_code: int) -> None:
     """Adds HTTP response attributes to span using the status_code argument."""
 
@@ -225,7 +364,7 @@ def set_status_code(span, status_code: int) -> None:
 class AiohttpGetter(Getter):
     """Extract current trace from headers"""
 
-    def get(self, carrier, key: str) -> Union[List, None]:
+    def get(self, carrier, key: str) -> list | None:
         """Getter implementation to retrieve an HTTP header value from the ASGI
         scope.
 
@@ -241,7 +380,7 @@ def get(self, carrier, key: str) -> Union[List, None]:
             return None
         return headers.getall(key, None)
 
-    def keys(self, carrier: Dict) -> List:
+    def keys(self, carrier: dict) -> list:
         return list(carrier.keys())
 
 
@@ -256,11 +395,13 @@ async def middleware(request, handler):
     ):
         return await handler(request)
 
-    span_name, additional_attributes = get_default_span_details(request)
+    span_name = get_default_span_name(request)
 
-    req_attrs = collect_request_attributes(request)
-    duration_attrs = _parse_duration_attrs(req_attrs)
-    active_requests_count_attrs = _parse_active_request_count_attrs(req_attrs)
+    request_attrs = collect_request_attributes(request)
+    duration_attrs = _parse_duration_attrs(request_attrs)
+    active_requests_count_attrs = _parse_active_request_count_attrs(
+        request_attrs
+    )
 
     duration_histogram = meter.create_histogram(
         name=MetricInstruments.HTTP_SERVER_DURATION,
@@ -279,14 +420,22 @@ async def middleware(request, handler):
         context=extract(request, getter=getter),
         kind=trace.SpanKind.SERVER,
     ) as span:
-        attributes = collect_request_attributes(request)
-        attributes.update(additional_attributes)
-        span.set_attributes(attributes)
+        if span.is_recording():
+            request_headers_attributes = collect_request_headers_attributes(
+                request
+            )
+            request_attrs.update(request_headers_attributes)
+            span.set_attributes(request_attrs)
         start = default_timer()
         active_requests_counter.add(1, active_requests_count_attrs)
         try:
             resp = await handler(request)
             set_status_code(span, resp.status)
+            if span.is_recording():
+                response_headers_attributes = (
+                    collect_response_headers_attributes(resp)
+                )
+                span.set_attributes(response_headers_attributes)
         except web.HTTPException as ex:
             set_status_code(span, ex.status_code)
             raise