-
Couldn't load subscription status.
- Fork 168
Open
Labels
Description
Context: There is a LOT of context in #2183 and #2190.
This issue is intended to help us keep track of these things:
- We agree that there are cases where it would be useful to export data previously stored on disk without the need to deserialize/unmarshal it back into Java objects.
a. This is beneficial, because reading, deserializing, and then reserializing back to the network is higher cost/overhead.
b. Typically, we would just want the stored protobufs (or json) to just read from disk in order to be put into an http request body and forwarded.
c. A stretch goal (which I would be fine spilling over into another follow-up issue) would be to facilitate streaming directly from disk to http request. This streaming approach should have lower total memory use and not require loading entire files into memory.
d. One downside is that existing java exporters all operate on otel data objects (spans, logs, metric data) and not raw bytes, so existing exporters cannot be leveraged directly here. - We agree that there will need to be some additional work performed (perhaps across several PRs) to refactor the APIs to make this possible.
a. A design goal is to try and limit the overall api surface (for maintainers)
b. A design goal is to make the common use case simple (sane defaults)
c. A design goal is to make the other more complicated use cases possible
I think the scope of this issue should NOT include situations where the serialized on-disk format differs from the wire format. That is, if protobuf byte arrays are written to disk, then raw protobuf byte arrays would be read and sent. Same for json or any other format.
This effort should also not limit/prevent the use of existing, upstream, off-the-shelf exporters (grpc, http, etc)