Add OpenTelemetry base classes #4982

rli · 2024-10-17T23:37:46Z

The OpenTelemetry SDK can automatically attach the same trace ID to spans recorded under the same 'context'. This is somewhat challenging because the JetBrains platform frequently requires switching threads and coroutine scope contexts, which loses the ThreadLocal state needed to connect parent->child when spans are started.

This on its own is not too difficult, but becomes a significant ergonomic challenge when we also want to bring some type-safety to how span metadata is built. This PR allows spans generated in aws/aws-toolkit-common#899 to automatically perform context propagation.

See internal documentation for details. A contributor-friendly version of the doc will be written after we validate this pattern with some initial metrics.

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2024-10-17T23:54:56Z

Qodana Community for JVM

8 new problems were found

Inspection name	Severity	Problems
`Unused symbol`	🔶 Warning	3
`Unstable API Usage`	🔶 Warning	1
`Non-distinguishable logging calls`	◽️ Notice	2
`Function or property has platform type`	◽️ Notice	1
`Redundant 'requireNotNull' or 'checkNotNull' call`	◽️ Notice	1

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at [email protected]

Or via our issue tracker: https://jb.gg/qodana-issue
Or share your feedback: https://jb.gg/qodana-discussions

...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+            try {
+                val item = TraceRequestMarshaler.create(listOf(data))
+
+                httpPost(traceUrl, contentLength = item.binarySerializedSize.toLong(), contentType = ContentType.XProtobuf) {


...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+            } catch (e: CancellationException) {
+                throw e
+            } catch (e: ConnectException) {
+                thisLogger().warn("Cannot export (url=$traceUrl): ${e.message}")


...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+            } catch (e: CancellationException) {
+                throw e
+            } catch (e: ConnectException) {
+                thisLogger().warn("Cannot export (url=$traceUrl): ${e.message}")


.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+import java.io.ByteArrayOutputStream
+import java.net.ConnectException
+
+private class BasicOtlpSpanProcessor(


...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+    }
+}
+
+private class SigV4OtlpSpanProcessor(


...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+    }
+
+    companion object {
+        fun getSdk() = service<OTelService>().sdk


.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

+import com.intellij.platform.diagnostic.telemetry.helpers.use as ijUse
+import com.intellij.platform.diagnostic.telemetry.helpers.useWithScope as ijUseWithScope
+
+val AWS_PRODUCT_CONTEXT_KEY = ContextKey.named<AWSProduct>("pluginDescriptor")


.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

+            }
+            setParent(parent)
+        }
+        requireNotNull(parent)


.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

manodnyab · 2024-10-29T23:23:19Z

...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+                    it.identity(creds.resolveIdentity().get())
+                    it.request(httpRequest)
+                    it.payload(payload)
+                    it.putProperty(AwsV4HttpSigner.SERVICE_SIGNING_NAME, "osis")


What does this property name stand for

open search ingestion service

manodnyab · 2024-11-01T20:55:48Z

...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+                    .setResource(
+                        Resource.create(
+                            Attributes.builder()
+                                .put(AttributeKey.stringKey("os.type"), SystemInfoRt.OS_NAME)


Where would we use this recorded information other than telemetry?
Should this have all the fields that we record for the telemetry publisher?

these are common fields understood by OTLP processors

manodnyab · 2024-11-01T20:57:58Z

...brains-community/tst/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBaseTest.kt

+    fun `context propagates from parent to child when child overrides context`() {
+        spanBuilder("tracer", "parentSpan").use {
+            // parent->child relationship is still maintained because Context.current() will return parent context
+            spanBuilder("anotherTracer", "childSpan").setParent(Context.current().with(AWS_PRODUCT_CONTEXT_KEY, AWSProduct.AMAZON_Q_FOR_VS_CODE)).use {}


Should we change this to AMAZON_Q_FOR_JETBRAINS?

intentional so we know for sure it's working and not just picking up a default field somewhere

manodnyab · 2024-11-01T20:58:22Z

...brains-community/tst/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBaseTest.kt

+            assertThat(parent.parentSpanContext.traceId).isEqualTo(TraceId.getInvalid())
+            assertThat(child.parentSpanContext.traceId).isEqualTo(parent.spanContext.traceId)
+            assertThat(parent.getAttribute(PLUGIN_ATTRIBUTE_KEY)).isNotEqualTo(child.getAttribute(PLUGIN_ATTRIBUTE_KEY))
+            assertThat(child.getAttribute(PLUGIN_ATTRIBUTE_KEY)).isEqualTo("Amazon Q For VS Code")


this would also change to JB?

manodnyab · 2024-11-01T20:59:21Z

...brains-community/tst/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBaseTest.kt

+                }
+
+                override fun onEnd(span: ReadableSpan) {
+                    println(span)


do we need this?

helpful for debugging without a debugger but this one can be removed

manodnyab · 2024-11-04T23:19:29Z

...tbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OTelService.kt

+            } catch (e: CancellationException) {
+                throw e
+            } catch (e: ConnectException) {
+                thisLogger().warn("Cannot export (url=$traceUrl): ${e.message}")


nit: should this also be log.error?

manodnyab · 2024-11-04T23:21:16Z

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

+abstract class AbstractSpanBuilder<
+    BuilderType : AbstractSpanBuilder<BuilderType, SpanType>,
+    SpanType : AbstractBaseSpan<SpanType>,
+    >(


nit: should the > be on the previous line?

can't get formatter to do it

manodnyab · 2024-11-04T23:25:26Z

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

+            parent = if (s is AbstractBaseSpan<*> && s.context != null) {
+                s.context.with(Span.fromContext(parent))
+            } else {
+                parent.with(AWS_PRODUCT_CONTEXT_KEY, resolvePluginName())


Should resolvePluginName be under a try catch?

returns null if fails

## Problem At the moment, metrics in the JetBrains extensions are collected as singular datapoints independent of each other. Modern observability frameworks, as well as our long-term goals are to be able to correlate these data flows over a single user flow. This requires the 'trace ID' concept which is high-friction in the current model. ## Solution Generate 'fluent builders' that wrap the OpenTelemetry SDK to allow us to leverage the SDK to record trace/span IDs automatically, but still expose a good amount of type safety based on the metric schema. This PR references base classes proposed in aws/aws-toolkit-jetbrains#4982. ## License By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…5027) * Depends on #4982 * Depends on aws/aws-toolkit-common#899 * Rename Unit -> MetricUnit directly in SDK codegen * Use newer provider convention syntax for input/output in generator task * expose `passive`/`unit`/`value` as special fields * add smoke test for `validateRequiredAttributes`

Add OpenTelemetry base classes

9a3743d

github-advanced-security bot found potential problems Oct 17, 2024

View reviewed changes

rli added 2 commits October 21, 2024 11:32

Merge remote-tracking branch 'origin/main' into rli/otel

1ae437e

lint

0769a77

github-advanced-security bot found potential problems Oct 21, 2024

View reviewed changes

rli added 2 commits October 22, 2024 10:46

Merge remote-tracking branch 'origin/main' into rli/otel

5d087e3

build

b1ef95f

github-advanced-security bot found potential problems Oct 22, 2024

View reviewed changes

rli added 2 commits October 22, 2024 15:02

tst

0a4e9ca

lint

c34c51c

github-advanced-security bot found potential problems Oct 22, 2024

View reviewed changes

hack in 233

ddf6db9

github-advanced-security bot found potential problems Oct 24, 2024

View reviewed changes

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/telemetry/otel/OtelBase.kt

}

setParent(parent)

}

requireNotNull(parent)

Check notice

Code scanning / QDJVMC

Redundant 'requireNotNull' or 'checkNotNull' call Note

Redundant 'requireNotNull' call

rli added 8 commits October 24, 2024 13:32

lint

7d90796

wip

f5e4c31

Merge remote-tracking branch 'origin/main' into rli/otel

f3d4c32

cleanup

eeb5b78

Merge remote-tracking branch 'origin/main' into rli/otel

201590c

revert

68ac68f

revert

37d92e3

redundant

8389744

rli marked this pull request as ready for review October 28, 2024 20:53

rli requested a review from a team as a code owner October 28, 2024 20:53

github-advanced-security bot found potential problems Oct 28, 2024

View reviewed changes

rli added 2 commits October 28, 2024 16:01

build

332ecb2

Merge remote-tracking branch 'origin/main' into rli/otel

1f4cb7d

This was referenced Oct 30, 2024

Tweak otelbase classes slightly to meet requirements from generator #5027

Merged

jetbrains: Generate OpenTelemetry-style metric builders aws/aws-toolkit-common#899

Merged

lint

551b153

manodnyab reviewed Nov 1, 2024

View reviewed changes

manodnyab approved these changes Nov 4, 2024

View reviewed changes

Update OtelBaseTest.kt

c457e1f

rli merged commit 06181dd into main Nov 5, 2024
16 of 17 checks passed

rli deleted the rli/otel branch November 5, 2024 00:48

Add OpenTelemetry base classes #4982

Add OpenTelemetry base classes #4982

Uh oh!

Conversation

rli commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License

Uh oh!

github-actions bot commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Qodana Community for JVM

Uh oh!

Uh oh!

Check warning

Check notice

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Check warning

Uh oh!

Uh oh!

Check warning

Uh oh!

Check notice

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rli Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rli commented Oct 17, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024 •

edited

Loading

rli Nov 4, 2024 •

edited

Loading