Instrumentation agent directly export the trace to kakfa in otlp format

[diracchehn]

Is your feature request related to a problem? Please describe.
Currently instrumentation agent will build a keep-alived connection with the collector, if one agent capture too much trace in a certain period, it will lead the instrumentation agent out of memory or the collector has been killed by oom.
Especially when the application has been hosted in an on-premise data center and the collector has been hosted on the cloud. If the trace data is relatively big and the network latency is high, it will happen above issues.

Describe the solution you'd like
I suggest make the java instrumentation able to directly send trace to Kafka or a message queue like skywalking. It will decouple the agent and the collector and give better performance.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[mateuszrzeszutek]

Hey @diracchehn ,

Do I understand it correctly that you'd like to have a Java implementation of the collector's Kafka exporter?

[diracchehn]

Yes, a kafka exporter will be great. Meanwhile, the instrumentation shall able to define the topic and some kafka parameters.

[trask]

Meanwhile, the instrumentation shall able to define the topic and some kafka parameters.

can you expand on what you mean by this? thx

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[diracchehn]

For full stack monitoring, the APM software usually hosted on cloud, but the application needs to be mointored will hosted in DC and other cloud. If the entire applications are in a hybrid cloud environment, the network latency will be high. Usually the java auto-instrumentation will export the trace and metrics to a otel-collector. How ever it consume too much resource on the application side. Just a suggestion, let the agent direct export the trace to a Kafka, then the collector could consume it in a controled frequency, the collector could be in another cloud or in DC. That is my idea, hope I explain it properly.

[mateuszrzeszutek]

Hey @diracchehn ,
I get your use case -- the problem is, this issue is still to vague to even consider it. Do you want to reimplement the collector's Kafka exporter (that I linked a couple posts above) in Java? Or do you have another ideas about the desired behavior/format/specification of the exporter?

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.