Support for Serialization and De-serialization of Metrics and Logs using Spring Data

[FWinkler79]

Is your feature request related to a problem? Please describe.

The current metric (potentially also log) types of OTel Java Instrumentation do not support serialization / de-serialization and persistence using Spring Data. The persistence fails due to the classes of OTel Java Instrumentation not having been compiled with the -parameters compiler flag and partially because they don't always provide a default constructor.

Persistence of metrics and logs becomes relevant for scenarios where the metrics and logs must not be lost, i.e. some delivery guarantees need to be given. Examples of such scenarios are logs used for audit logging, or metrics used in commercial metering environments.

Please also see this discussion thread I had opened some time ago: #7554

Describe the solution you'd like

The OpenTelemetry classes for metrics and logs, potentially also trace spans should be serializable / de-serializable and support persistence via Spring Data. This will allow application developers wanting to use OpenTelemetry in scenarios where delivery guarantees are required (e.g. audit logging, metering) to temporarily persist OTel data and export it later, in case of temporary error situations that make sending the data impossible.

Describe alternatives you've considered
Serializing classes via frameworks like Jackson to XML or JSON and persisting that instead. Still need to try out, if this would be a valid option.

Additional context
More details, also on what would have to change in the OTel Java instrumentation to support this can be found here.

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[zeitlinger]

There is a module for telemetry serialization: https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/disk-buffering

[jack-berg]

The SpanData, MetricData, LogRecordData interfaces are only meant to be in-memory representations of those data types. They are intentionally opinionated about serialization, which is a task normally reserved for SpanExporter, MetricExporter, LogRecordExporter.

However, OpenTelemetry does define a standard serialization format for these data types in the form of OTLP, and our Otlp(Grpc|Http)(Span|Metric|LogRecord)Exporters internally use this marshaling code to perform this serialization. We hand roll the serialization to improve performance and minimize dependencies. Notably, there is no corresponding deserialization code since OTLP exporters don't need to do that.

If you generate bindings from the OTLP proto definitions (or use those from opentelemetry-proto-java), its much easier to serialize (although less performant and with extra dependencies). All you need to do is perform the mapping from SpanData, MetricData, LogRecordData to the respective proto bindings and use the built-in binary or JSON serialization tools from the proto library dependency. This is what the disk buffering library @zeitlinger links to is doing.

I'm open to making the internal hand-rolled serialization code public, but as mentioned, there is equivalent hand-rolled deserialization code. So someone would have to do that tedious / careful work, and we'd have to decide where it should live. Its a bit of an odd thing to publish because its not actually needed by anything in opentelemetry-java.