[disk-buffering] performance improvements

[zeitlinger]

From @tylerbenson

1. it seems like there's a potentially unnecessary step of deserializing and reserializing assuming the delegate exporter is serializing to otlp/protobuf. Any reason it can't just stream the bytes directly?
2. I think it could be a lot more efficient by using the hand coded protobuf serializers that the OTLP exporters currently use, instead of fully generating the whole byte array at once.

[zeitlinger]

unnecessary step of deserializing and reserializing

this is necessary if we want to use the existing exporters which work on object level, e.g.

opentelemetry-java-contrib/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordFromDiskExporter.java

Line 29 in 2b31033

.setExportFunction(exporter::export)

[zeitlinger]

could be a lot more efficient

I got the serialization part compiling (see below) - but there's no deserialize method in the hand written serializers

@jack-berg do you know it it's possible to deserialize the hand written protobuf?

Subject: [PATCH] fix
---
Index: disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/LogRecordDataSerializer.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/LogRecordDataSerializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/LogRecordDataSerializer.java
--- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/LogRecordDataSerializer.java	(revision d2ebf96a6b2e08d9bd7bd30f51f7139cb5b18535)
+++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/LogRecordDataSerializer.java	(date 1748266961119)
@@ -5,9 +5,7 @@
 
 package io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers;
 
-import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.logs.ProtoLogsDataMapper;
-import io.opentelemetry.contrib.disk.buffering.internal.utils.ProtobufTools;
-import io.opentelemetry.proto.logs.v1.LogsData;
+import io.opentelemetry.exporter.internal.otlp.logs.LogsRequestMarshaler;
 import io.opentelemetry.sdk.logs.data.LogRecordData;
 import java.io.ByteArrayOutputStream;
 import java.io.IOException;
@@ -24,11 +22,12 @@
 
   @Override
   public byte[] serialize(Collection<LogRecordData> logRecordData) {
-    LogsData proto = ProtoLogsDataMapper.getInstance().toProto(logRecordData);
+    // for immutable data
+    // todo check reusable data, as it's more efficient:
+    // https://github.com/open-telemetry/opentelemetry-java/blob/cd3b0e7f9d950127759d2b0d34fed84ea438e938/exporters/otlp/common/src/main/java/io/opentelemetry/exporter/internal/otlp/logs/LogReusableDataMarshaler.java#L40
+
     try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
-      int size = LogsData.ADAPTER.encodedSize(proto);
-      ProtobufTools.writeRawVarint32(size, out);
-      proto.encode(out);
+      LogsRequestMarshaler.create(logRecordData).writeBinaryTo(out);
       return out.toByteArray();
     } catch (IOException e) {
       throw new IllegalStateException(e);
Index: disk-buffering/build.gradle.kts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/disk-buffering/build.gradle.kts b/disk-buffering/build.gradle.kts
--- a/disk-buffering/build.gradle.kts	(revision d2ebf96a6b2e08d9bd7bd30f51f7139cb5b18535)
+++ b/disk-buffering/build.gradle.kts	(date 1748266758921)
@@ -22,6 +22,7 @@
 
 dependencies {
   api("io.opentelemetry:opentelemetry-sdk")
+  implementation("io.opentelemetry:opentelemetry-exporter-otlp-common")
   compileOnly("com.google.auto.value:auto-value-annotations")
   annotationProcessor("com.google.auto.value:auto-value")
   signature("com.toasttab.android:gummy-bears-api-21:0.6.1:coreLib@signature")

[tylerbenson]

As Jack said, the deserialization would need to be implemented, which is probably a bigger task than is warranted for this use case. One potential option... perhaps there could be a lazy implementation of the pipeline data objects that wait until they're accessed before deserialization. Then, maybe the protobuf exporter implementation could have a special case to handle this scenario where the object is already serialized.

[tylerbenson]

Following up here... while I think this is an improvement, it is still using an intermediary byte array. It would be much more efficient if the marshaler could write directly to the file output stream.

[jack-berg]

@jack-berg do you know it it's possible to deserialize the hand written protobuf?

Someone would have to hand roll de-serialization logic to pair with the existing hand rolled serialization logic