Solution for emitting subsegment larger than 65507 bytes

[aws-kj]

This is a feature request for cases where SDK (not only Java SDK but for any other XRay SDKs) cannot send (sub)segments if they are larger than 65507 bytes.

Currently com.amazonaws.xray.emitters.UDPEmitter best effort to send large segments to the agent is to dissolve into subsegments and eventually throw an exception with the details if the subsegment is still larger than the maximum UDP datagram size as per this code and will eventually lead to a java.io.IOException

(details behind this limitation of the UDP protocol are here)

However, there are scenarios when we cannot avoid emitting large (sub)segments due to automatic instrumentation (for example large SQL queries). For these cases there could be some mitigation/workaround to still receive the subsegment or at least a part of it.

Options:

• Truncate segment workload (SQL query text in the particular case)
• Implement TCP communication between SDK and the agent
• Implement programmatic datagram fragmentation (as UDP doesn't support fragmentation by design)

[wangzlei]

Thanks for raising this idea. The single X-Ray packet limit from SDK UDPEmitter is the UDP MTU is the hard limit by design. To break this limit we have to create TCP(or grpc, http whatever) emitter from both SDKs and X-Ray daemon, which would also impact existing xray service design since X-Ray service also has segment size limit, please refer to https://docs.aws.amazon.com/general/latest/gr/xray.html#limits_xray

So, truncate segment might be a feasible solution but how to truncate the segment attribute intelligently without breaking the segment integrity is a challenge. Right now our recommendation solution is by migrating to https://opentelemetry.io/ , which the SDK and Collector are by http/grpc. But at this moment, xray service side is still the bottleneck. We need a new feature from xray service to be able to digest larger Segment and Trace. I will forward this request to X-Ray service.