Replace apm-java-agent with OTEL SDK in Elasticsearch

[pgomulka]

Description

elasticsearch is using apm-java-agent as the underlying implementation in the apm module.we are using our own apm api, implemented in apm-module with OTEL api. This should not change.
What should change is the binding between otel api and the implementation. Which should be otel sdk. Otel SDK will allow us to get more flexibility on configuring how our metrics and traces are sent to apm server (apm server support otel sdk).
With Otel sdk we will be able to implement features like 'tee-ing' (splitting to two apm server) of the export or some additional buffering, retries when apm server is overloaded.

I worked on a simple very dirty PoC where this proves to work #110263
Things that need more investigation and work:

• 1. configuring the otel sdk with the apm-java-agent settings (things like metrics_interval, transactioN-sample_rate, server_url etc)
• 2. rework of RunTask - for localhost testing 73c0e5f
• 3. rework of logging - otel sdk is using JUL logging. We already have a bridge in server https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/common/logging/JULBridge.java If we refactor it into a lib it should be possible to use it in the module. I wasn't able to make it work without adding a JUL -log4j bridge dependency (which we don't want) but at the same time I was rushing it.. We want to make sure we use the same apm_agent.json file (probably not worth renaming). This is a good example where plugin's might want to add an appender.
• 4. careful review of the dependencies. otel sdk requires quite a lot, some of the dependencies like okhttp are not working with java9's modules. some are introducing a 'clash version' dependencies (netty is already a dependency of server). It all shouldn't be a problem if we hide the implementation behind the embedded classloader like we do for x-content and jackson
• 5. due to use of java beans the java9's module require a java.desktop. This feels awkward, but  I am not sure how to go around it.
• 6. OTEL sdk buildAndRegister can only be called once. If it is called twice and exception will be thrown. We need to make sure that starting/stopping the metering (this is possible now) will not throw this exception. 
• 7. apm-java-agent gives us a bunch of out of the box metrics for the jvm. I copied the JvmJdMetrics from apm-java-agent repo. Perhaps we need to work with apm team to have this as a lib? Just copying the JvmFdMetrics,  JvmGcMetrics, JvmMemoryMetrics could work initially, but feels dirty. The naming there has to also comply with our naming convention (we would register them using our with own api)
• 8. BIG - review the security manager permission. It would be a relatively tedious and long task, as there is a loot of new dependencies. For the PoC I have disabled the security manager
• 9. New Exporters - we could simply configure the out of the box available exporters (simply adding 2 for the support of exporting to 2 apm servers) or implement our own so that we have more control of logging etc
• 10. otel sdk exporters support http and grpc protocoles. APM server works with both. Need to decide on one.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

additional features to consider:

1. being able to set custom interval for certain metrics

• it is possible to create multiple MetricReaders(those trigger exporting) at different interval. So therefore we could maybe have some custom filtering what metrics are read by what MetricsReader (and thus exported at different intervals). does not seem trivial though.

3. change metric interval dynamically

• it is possible to provide a custom java.util.concurency.Scheduler. so if only we implement a logic that cancel's previous scheduled task and submit a new one with different interval it should be doable.

4. make sure metrics are sent upon node shutdown

• there is a force flush mechanism. I am not sure if it is possible to set a timeout on it though

[pgomulka]

we should also consider how to turn the exporting on and off

[SylvainJuge]

Hi !

For the JVM metrics, I am currently working on making them defined in the instrumentation as a yaml file (open-telemetry/opentelemetry-java-instrumentation#13392) this won't be 100% equivalent to the ones captured by the OpenTelemetry instrumentation agent or EDOT due to limitations of the JMX interface, but would still be compliant with the stable JVM metrics defined in semantic conventions.

With that, you should be able to easily reuse the jmx-metrics library module from instrumentation (it only depends on the SDK) and capture the JVM metrics in a semconv-compliant way. A good example of reusing this is the jmx-scraper in contrib that is a CLI tool that connects to a remote JVM to extract metrics from the JMX interface and push them to an otel collector endpoint.

As an alternative, the runtime-telemetry/runtime-telemetry-java8/library (JMX) or runtime-telemetry/runtime-telemetry-java17 (JMX + JFR) library modules in instrumentation could also provide an easier setup without the limitations of the JMX + YAML implementation and they are available today.

Something worth considering is that the metrics mapping might be a bit different from what they currently do, which means having potentially breaking changes on anything that relies on them (dashboards, alerts, ...).

In addition, if we have to deal with breaking changes here, it's probably a good idea to bundle this with changing the ingestion path to use the EDOT collector instead of the apm endpoint. Doing so should bring a more otel-native storage of data while preserving compatibility with APM UI.

[SylvainJuge]

For the Security Manager, this is something that goes away with next release.

Another reason to prioritize this is that some security scanning tools inspect the dependencies of the Elastic APM Java agent and are not able to distinguish between compile-time dependencies (which are required to build instrumentation) and the dependencies that are actually included in the product.

As a consequence there are some false-positive security reports that we have to deal with from time to time which is not ideal. Using the SDK, it's just a regular library, so we would not have those potentially unsafe compile-time-only dependencies and thus it would solve those issues for good.

[JonasKunz]

10. otel sdk exporters support http and grpc protocoles. APM server works with both. Need to decide on one.

The default exporter is http/protobuf, due to the easier, more widespread support than gRPC. For the detailed reasoning please see this issue.
Therefore I'd suggest to stick with http, as it is the default.

[JonasKunz]

Here's a demo project showing how to:

• Create and globally register an OpenTelemetrySDK instance
• Have the exporters dynamically configurable on that instance

If you need any help with understanding the demo or anything else let me know!