OpenTelemetry - AWS ID Generator

[akil-rails]

In the quarkus docs there's an example of a custom IDGenerator for AWS using AwsXrayIdGenerator.getInstance(); To compile this, a dependency is needed, and I added the following

    <dependency>
      <groupId>io.opentelemetry.contrib</groupId>
      <artifactId>opentelemetry-aws-xray</artifactId>
      <version>1.25.1</version>
    </dependency>

The JVM version works fine, But with this, the native compilation fails with the following exception:

Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing io.opentelemetry.contrib.awsxray.AwsXrayRemoteSamplerBuilder.build() 
Parsing context:
   at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSamplerBuilder.build(AwsXrayRemoteSamplerBuilder.java:95)
   at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSamplerProvider.createSampler(AwsXrayRemoteSamplerProvider.java:32)
   at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration$$Lambda$1209/0x00000007c1a27b58.apply(Unknown Source)
   at io.opentelemetry.sdk.autoconfigure.SpiUtil.lambda$loadConfigurable$0(SpiUtil.java:48)
   at io.opentelemetry.sdk.autoconfigure.SpiUtil$$Lambda$1202/0x00000007c1a26bb8.get(Unknown Source)
   at com.oracle.svm.core.jdk.SystemPropertiesSupport.initializeLazyValue(SystemPropertiesSupport.java:219)
   at com.oracle.svm.core.jdk.SystemPropertiesSupport.getProperty(SystemPropertiesSupport.java:172)
   at com.oracle.svm.core.jdk.Target_java_lang_System.getProperty(JavaLangSubstitutions.java:305)
   at sun.security.action.GetPropertyAction.privilegedGetProperty(GetPropertyAction.java:106)

	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:153)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:104)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:83)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraph(MethodTypeFlow.java:65)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultSpecialInvokeTypeFlow.onObservedUpdate(DefaultSpecialInvokeTypeFlow.java:61)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:562)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$1.run(PointsToAnalysis.java:488)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:193)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:177)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected an instance of Random/SplittableRandom class in the image heap. Instances created during image generation have cached seed values and don't behave as expected.  To see how this object got instantiated use --trace-object-instantiation=java.util.Random. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
	at parsing io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler.<init>(AwsXrayRemoteSampler.java:100)

Is this not supported? (the docs say that native compilation will work).

[quarkus-bot[bot]]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[akil-rails]

I tried --initialize-at-run-time=io.opentelemetry.contrib.awsxray, but this results in another error

Error: Classes that should be initialized at run time got initialized during image building:
 io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator the class was requested to be initialized at run time (from command line with 'io.opentelemetry.contrib.awsxray'). To see why io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator got initialized use --trace-class-initialization=io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator
To see how the classes got initialized, use --trace-class-initialization=io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator
com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
 io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator the class was requested to be initialized at run time (from command line with 'io.opentelemetry.contrib.awsxray'). To see why io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator got initialized use --trace-class-initialization=io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator
To see how the classes got initialized, use --trace-class-initialization=io.opentelemetry.contrib.awsxray.AwsXrayIdGenerator
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ProvenSafeClassInitializationSupport.checkDelayedInitialization(ProvenSafeClassInitializationSupport.java:273)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.duringAnalysis(ClassInitializationFeature.java:164)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:748)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:85)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:748)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.AbstractAnalysisEngine.runAnalysis(AbstractAnalysisEngine.java:162)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:745)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:578)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:535)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:580)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)

[akil-rails]

The issue appears with version 3.0.1-Final (GraalVM 22.3.2.1-Final Java 17 Mandrel Distribution) ; the compilation to native is successful with version 2.9.1-Final (GraalVM 22.0.0.2 Java 17 CE).

A sample is at https://github.com/akil-rails/opentelemetry-microservices (project is_odd)

1. main branch: 2.9.1-Final
2. q3 branch: 3.0.1-Final

Looking for advice on what I can do/try to resolve this.

[akil-rails]

Removing the dependency and adding just the IdGenerator implementation works and looks to be the way ahead for now .

[brunobat]

@akil-rails What do you mean by just adding the IdGenerator?
Can you please post an example?

[brunobat]

The problem is the static init of the random at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler
These are the solutions: https://foivos.zakkak.net/tutorials/working-with-randoms-native-images/
Most likely we need to commit updates to the lib.

[akil-rails]

But we don't need the Sampler, just the IdGenerator as per the Quarkus/open-telemetry docs. I had tried to defer to runtime initialisation but that didn't work either

[brunobat]

The sampler get's instantiated when the Id Generator is used.
... And yes, using the 1st simple trick in the article doesn't work.

[akil-rails]

Should I try and submit a PR to the library (not sure if they will accept) or do you suggest an extension (but I'll need some guidance on how to do the extension).

[brunobat]

I'm trying an update to the lib... Will submit a PR if it works... We shouldn't need an extension for this

[brunobat]

I could fix the random but I found another, bigger problem. The OkHttp client is being used in the XraySamplerClient to get the sampling rules and it requires OkHttp v 4.11. Quarkus only allows 3.x and the called method is not available in 3.x.
This will never work on native either without removing OkHttp from the lib (preferable) or creating an extension to exclude the classes using it (not very maintainable and will cut functionality).

Created this: open-telemetry/opentelemetry-java-contrib#863
and this: open-telemetry/opentelemetry-java-contrib#862

[DSkoufis]

Has this been resolved in a newer Quarkus version? I'm trying to enable this in the quarkus-lambda extension.

In the docs there are some references to AWS X-ray but trying to use the AwsXrayIdGenerator still doesn't work with native, a bit lost with what's supported an what not.

• https://quarkus.io/guides/opentelemetry#id-generator
• https://quarkus.io/guides/aws-lambda#tracing-with-aws-xray-and-graalvm

[brunobat]

Hi @DSkoufis Not sure of the current state of that AwsXrayIdGenerator.
Can you please create a Bug report?