|
| 1 | +## ADOT AWS SDK Instrumentation |
| 2 | + |
| 3 | +### Overview |
| 4 | +The aws-sdk instrumentation is an SPI-based implementation that extends the upstream OpenTelemetry AWS Java SDK instrumentation. |
| 5 | + |
| 6 | +##### _v1.11 Initialization Workflow_ |
| 7 | +1. OpenTelemetry Agent Starts |
| 8 | + - Loads default instrumentations |
| 9 | + - Loads aws-sdk v1.11 instrumentations |
| 10 | + - Injects **TracingRequestHandler** into constructor |
| 11 | +2. Scans for other SPI implementations |
| 12 | + - Finds ADOT’s **AdotAwsSdkInstrumentationModule** |
| 13 | + - Injects code that: |
| 14 | + - Checks for TracingRequestHandler |
| 15 | + - If present, adds **AdotAwsSdkTracingRequestHandler** |
| 16 | +3. AWS SDK Client Created |
| 17 | + - Constructor runs with injected code: |
| 18 | + [AWS Handlers] → TracingRequestHandler → AdotAwsSdkTracingRequestHandler |
| 19 | + |
| 20 | +##### _v2.2 Initialization Workflow_ |
| 21 | + |
| 22 | +1. OpenTelemetry Agent starts |
| 23 | + - Loads default instrumentations |
| 24 | + - Loads aws-sdk instrumentation from opentelemetry-java-instrumentation |
| 25 | + - Registers **TracingExecutionInterceptor** (order = 0) |
| 26 | +2. Scans for other SPI implementations |
| 27 | + - Finds ADOT’s **AdotAwsSdkInstrumentationModule** |
| 28 | + - Registers **AdotAwsSdkTracingExecutionInterceptor** (order > 0) |
| 29 | + |
| 30 | +#### _Note on Attribute Collection:_ |
| 31 | +AWS SDK v1.11 and v2.2 handle attribute collection differently: |
| 32 | + |
| 33 | +**V1.11:** |
| 34 | +- Maintains a separate AttributesBuilder during request/response lifecycle |
| 35 | +- Collects ADOT-specific attributes alongside upstream processing without interference |
| 36 | +- Injects collected attributes into span at the end of the request and response lifecycle hooks |
| 37 | + |
| 38 | + |
| 39 | +**V2.2:** |
| 40 | +- FieldMapper directly modifies spans during request/response processing |
| 41 | +- Attributes are added to spans immediately when discovered |
| 42 | +- Direct integration with span lifecycle |
| 43 | + |
| 44 | +This architectural difference exists due to upstream AWS SDK injecting attributes into spans differently for v1.11 and v2.2 |
| 45 | + |
| 46 | +### AWS SDK v1 Instrumentation Summary |
| 47 | +The AdotAwsSdkInstrumentationModule uses the instrumentation (specified in AdotAwsClientInstrumentation) to register the AdotAwsSdkTracingRequestHandler through `typeInstrumentations`. |
| 48 | + |
| 49 | +Key aspects of handler registration: |
| 50 | +- `order` method ensures ADOT instrumentation runs after OpenTelemetry's base instrumentation. It is set to the max integer value, as precaution, in case upstream aws-sdk registers more handlers. |
| 51 | +- `AdotAwsSdkClientInstrumentation` class adds ADOT handler to list of request handlers |
| 52 | + |
| 53 | +**AdotAwsSdkClientInstrumentation** |
| 54 | + |
| 55 | +AWS SDK v1.11 instrumentation requires ByteBuddy because, unlike v2.2, it doesn't provide an SPI for adding request handlers. While v2.2 uses the ExecutionInterceptor interface and Java's ServiceLoader mechanism, v1.11 maintains a direct list of handlers that can't be modified through a public API. Therefore, we use ByteBuddy to modify the AWS client constructor and inject our handler directly into the requestHandler2s list. |
| 56 | + |
| 57 | + - `AdotAwsSdkClientAdvice` registers our handler only if the upstream aws-sdk span is enabled (i.e. it checks if the upstream handler is present when an AWS SDK client is |
| 58 | + initialized). |
| 59 | + - Ensures the OpenTelemetry handler is registered first. |
| 60 | + |
| 61 | +**AdotAwsSdkTracingRequestHandler** |
| 62 | + |
| 63 | +The AdotAwsSdkTracingRequestHandler hooks onto OpenTelemetry's spans during specific phases of the SDK request and response life cycle. These hooks are strategically chosen to ensure proper ordering of attribute injection. |
| 64 | + |
| 65 | +1. `beforeRequest`: the latest point where the SDK request can be obtained after it is modified by the upstream aws-sdk v1.11 handler |
| 66 | +2. `afterAttempt`: the latest point to access the SDK response before the span closes in the upstream afterResponse/afterError methods |
| 67 | + - _NOTE:_ We use afterAttempt not because it's ideal, but because it our last chance to add attributes, even though this means our logic runs multiple times during retries. |
| 68 | + - This is a trade-off: |
| 69 | + - We get to add our attributes before span closure |
| 70 | + - But our code runs redundantly on each retry attempt |
| 71 | + - We're constrained by when upstream closes the span |
| 72 | + |
| 73 | +All the span lifecycle hooks provided by AWS SDK RequestHandler2 can be found [here.](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/handlers/RequestHandler2.html#beforeMarshalling-com.amazonaws.AmazonWebServiceRequest) |
| 74 | + |
| 75 | +_**Important Notes:**_ |
| 76 | +- The upstream interceptor's last point of request modification occurs in [beforeRequest](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-sdk/aws-sdk-1.11/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v1_11/TracingRequestHandler.java#L58). |
| 77 | +- The upstream interceptor closes the span in [afterResponse](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-sdk/aws-sdk-1.11/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v1_11/TracingRequestHandler.java#L116) and/or [afterError](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/aws-sdk/aws-sdk-1.11/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v1_11/TracingRequestHandler.java#L131). These hooks are inaccessible for span modification. |
| 78 | + `afterAttempt` is our final hook point, giving us access to both the fully processed response and active span. |
| 79 | + |
| 80 | +**High-Level Sequence Diagram:** |
| 81 | + |
| 82 | + |
| 83 | +_Class Functionalities:_ |
| 84 | +- `AdotAwsSdkTracingRequestHandler` |
| 85 | + - Hooks into AWS SDK request/response lifecycle |
| 86 | + - Adds ADOT-specific attributes to spans extracted by AwsSdkExperimentalAttributesExtractor |
| 87 | +- `AwsSdkExperimentalAttributesExtractor` |
| 88 | + - Extracts attributes from AWS requests/responses and enriches spans |
| 89 | + - Uses RequestAccess to get field values |
| 90 | + - Special handling for Bedrock services |
| 91 | +- `RequestAccess` |
| 92 | + - Provides access to AWS SDK object fields |
| 93 | + - Caches method handles for performance |
| 94 | + - Uses BedrockJsonParser for parsing LLM payloads |
| 95 | +- `BedrockJsonParser` |
| 96 | + - Custom JSON parser for Bedrock payloads |
| 97 | + - Handles different LLM model formats |
| 98 | +- `AwsBedrockResourceType` |
| 99 | + - Maps Bedrock class names to resource types |
| 100 | + - Provides attribute keys and accessors for each type |
| 101 | + |
| 102 | +### AWS SDK v2 Instrumentation Summary |
| 103 | + |
| 104 | +**AdotAwsSdkInstrumentationModule** |
| 105 | + |
| 106 | +The AdotAwsSdkInstrumentationModule registers the AdotAwsSdkTracingExecutionInterceptor in `registerHelperResources`. |
| 107 | + |
| 108 | +Key aspects of interceptor registration: |
| 109 | +- AWS SDK's ExecutionInterceptor loads global interceptors from files named '/software/amazon/awssdk/global/handlers/execution.interceptors' in the classpath |
| 110 | +- Interceptors are executed in the order they appear in the classpath - earlier entries run first |
| 111 | +- `order` method ensures ADOT instrumentation runs after OpenTelemetry's base instrumentation, maintaining proper sequence of interceptor registration in AWS SDK classpath |
| 112 | + |
| 113 | +**AdotAwsSdkTracingExecutionInterceptor** |
| 114 | + |
| 115 | +The AdotAwsSdkTracingExecutionInterceptor hooks onto OpenTelemetry's spans during specific phases of the SDK request and response life cycle. These hooks are strategically chosen to ensure proper ordering of attribute injection. |
| 116 | + |
| 117 | +1. `beforeTransmission`: the latest point where the SDK request can be obtained after it is modified by the upstream's interceptor |
| 118 | +2. `modifyResponse`: the latest point to access the SDK response before the span closes in the upstream afterExecution method |
| 119 | + |
| 120 | +All the span lifecycle hooks provided by AWS SDK ExecutionInterceptor can be found [here.](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/interceptor/ExecutionInterceptor.html) |
| 121 | + |
| 122 | +_**Important Notes:**_ |
| 123 | +- The upstream interceptor's last point of request modification occurs in [beforeTransmission](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/release/v2.11.x/instrumentation/aws-sdk/aws-sdk-2.2/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2/internal/TracingExecutionInterceptor.java#L237). |
| 124 | +- The upstream interceptor closes the span in [afterExecution](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/release/v2.11.x/instrumentation/aws-sdk/aws-sdk-2.2/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2/internal/TracingExecutionInterceptor.java#L348). That hook is inaccessible for span modification. |
| 125 | +`modifyResponse` is our final hook point, giving us access to both the fully processed response and active span. |
| 126 | + |
| 127 | +**High-Level Sequence Diagram:** |
| 128 | + |
| 129 | + |
| 130 | +_Class Functionalities:_ |
| 131 | +- `AdotAwsSdkTracingExecutionInterceptor` |
| 132 | + - Intercepts AWS SDK calls to create and enrich OpenTelemetry spans with AWS attributes |
| 133 | + - Coordinates the attribute mapping process |
| 134 | +- `FieldMapper` |
| 135 | + - Maps the AWS SDK fields to span attributes |
| 136 | + - Coordinates with Serializer for value conversion |
| 137 | +- `FieldMapping` |
| 138 | + - Defines what fields to map from SDK to spans |
| 139 | + - Groups mappings by type (REQUEST/RESPONSE) |
| 140 | +- `MethodHandleFacotry` |
| 141 | + - Provides fast, cached access to AWS SDK object fields for better performance |
| 142 | + - Used by FieldMapper for efficient field value extraction |
| 143 | +- `Serializer` |
| 144 | + - Converts AWS SDK objects and Bedrock objects into string values that can be used as span attributes |
| 145 | + - Works with BedrockJsonParser for LLM responses |
| 146 | +- `AwsJsonProtocolFactoryAccess` |
| 147 | + - Enables access to AWS SDK's internal JSON serialization capabilities for complex SDK objects |
| 148 | + - Uses reflection to access internal SDK classes |
| 149 | + - Caches method handles for performance |
| 150 | +- `BedrockJasonParser` |
| 151 | + - Parses and extracts specific attributes from Bedrock LLM responses for GenAI telemetry |
| 152 | + |
| 153 | +### Commands for Running Groovy Tests |
| 154 | + |
| 155 | +#### aws-sdk v1.11 |
| 156 | +To run the `BedrockJsonParserTest`: |
| 157 | +```` |
| 158 | +./gradlew :instrumentation:aws-sdk:test --tests "software.amazon.opentelemetry.javaagent.instrumentation.awssdk_v1_11.BedrockJsonParserTest" |
| 159 | +```` |
| 160 | + |
| 161 | +#### aws-sdk v2.2 |
| 162 | +To run the `BedrockJsonParserTest`: |
| 163 | +```` |
| 164 | +./gradlew :instrumentation:aws-sdk:test --tests "software.amazon.opentelemetry.javaagent.instrumentation.awssdk_v2_2.BedrockJsonParserTest" |
| 165 | +```` |
| 166 | + |
| 167 | +### Commands for Running Java Tests |
| 168 | + |
| 169 | +#### aws-sdk v1.11 |
| 170 | +To run the `AwsSdkExperimentalAttributesInjectionTest`: |
| 171 | +```` |
| 172 | +./gradlew :instrumentation:aws-sdk:test --tests "software.amazon.opentelemetry.javaagent.instrumentation.awssdk_v1_11.AwsSdkExperimentalAttributesInjectionTest" |
| 173 | +```` |
| 174 | + |
| 175 | +To run the `AdotAwsSdkClientAdviceTest`: |
| 176 | +```` |
| 177 | +./gradlew :instrumentation:aws-sdk:test --tests "software.amazon.opentelemetry.javaagent.instrumentation.awssdk_v1_11.AdotAwsSdkClientAdviceTest" |
| 178 | +```` |
| 179 | + |
| 180 | +#### aws-sdk v2.2 |
| 181 | +To run the `AwsSdkExperimentalAttributesInjectionTest`: |
| 182 | +```` |
| 183 | +./gradlew :instrumentation:aws-sdk:test --tests "software.amazon.opentelemetry.javaagent.instrumentation.awssdk_v2_2.AwsSdkExperimentalAttributesInjectionTest" |
0 commit comments