Request to add environment variable for Span size limitation

[leon4652]

Is your feature request related to a problem? Please describe.

Hello, I would like to suggest an idea inspired by a problem that occurred during the process of Agent and Span collection. It would be helpful if there were an environment variable or feature to limit the size of Span objects.

During the additional Span and Attributes collection process, a memory leak issue occurred. This was caused by the Span collected by the Agent, and the Old Gen region significantly increased throughout the collection period.

I suspect that this issue happened due to 'large-sized Span - where the content of Attributes grew longer, causing premature promotion from Eden to Old Gen.' Upon examining the collected data, I found that a number of abnormal Spans exceeding a certain size were collected. (Of course, this was caused by my individual Extension and collection logic.) I suspect that this is what triggered the memory leak.

In fact, I didn’t anticipate the Span size would grow so large until I analyzed it using tools, which led to the server overload.

Heap Used Graph(Metric), Max Heap size 8gb

Major GC occurred

Describe the solution you'd like

Could you please consider adding an option to set a maximum size for individual Span instances through environment variables? For instance, if a Span exceeds the specified size in KB, it could be automatically discarded or the memory could be released immediately. This feature could also be configured based on GC settings (e.g., XX:PretenureSizeThreshold).

Describe alternatives you've considered

As described above.

Additional context

I find this Agent very useful. Thank you for reading.

[laurit]

While it is not possible to limit span size it is possible to limit the number of attributes and their size, see https://opentelemetry.io/docs/languages/java/configuration/#properties-general Limiting based on span size doesn't seem too useful to me because the span size becomes know only after it has been serialized by the exporter, so a lot of work would need to be done before such limit could be applied.

During the additional Span and Attributes collection process, a memory leak issue occurred.

Usually a memory leak means that memory is accumulated somewhere and never freed, https://en.wikipedia.org/wiki/Memory_leak. I assume here you must be meaning something else when you say memory leak. If you really observed a memory leak then limiting the span size would be unlikely to fix it.

I suspect that this issue happened

Usually issues that present strong evidence rather than a gut feeling are more likely to get resolved.

Looking at the screenshot from memory analyzer the first thing I'd note is that we have done some work to lower the allocation rate in exporters. See https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters setting the experimental memory mode option to reusable_data could help. Secondly I'd interpret the screenshot so that you are exporting 120537 span with total size of 78mb, the average size of a span is 650 bytes. Is my understand correct? To me the average size of a span doesn't seem large. On the contrary the total number of spans is large. The default exported batch size 512, have you increased it?

Finally I'd like to note that the exporters are in https://github.com/open-telemetry/opentelemetry-java if you so wish we can move this issue there.

[leon4652]

Hello,
Regarding the memory leak, I realize that my choice of words might not have accurately conveyed the issue. To clarify, the problem arises because large Span objects are being collected and queued by the process, leading to an increase in Major-GC and Full-GC activities, rather than the Span objects being released.

I have also reviewed the contents of Eclipse MAT. It seems there was an error during the verification process. The MAT screenshot you referred to appears to be from a load testing server rather than the production server. Your response is correct. The attached images are results from a similar load testing server. There are differences in the individual sizes of the Spans, and the batch intervals at the time of the issue remain the same.

To be specific, the detailed environment variables set in the Batch Span Processor are as follows:

otel.bsp.schedule.delay=1000
otel.bsp.max.queue.size=16384 ~ 163840 
otel.bsp.max.export.batch.size=32768 ~ 327680
otel.bsp.export.timeout=10000

I have also reviewed the environment variable related to otel.java.experimental.exporter.memory_mode as mentioned in the documentation. Thank you for your assistance. It seems that by using this parameter to enable the pooling method, we can effectively resolve the current issue.

Additionally, as you mentioned, the exporter repository seems to be the correct place for further questions. If any related issues arise in the future, I will move them to that repository.

Thank you for your kind response. I wish you a wonderful day and hope you stay healthy during this seasonal transition.

Best regards, leon