Feature request: Client-Side sampling or similar mechanism to avoid throttling for traces

[slavaschmidt]

Use case

Context

We are currently evaluating AWS Powertools as a potential replacement for our custom X-Ray trace publisher. Our existing solution is tailored for high-throughput batch processing workloads, where we frequently encounter throttling issues with AWS X-Ray during peak load periods.

Use Case

In our current setup, we mitigate X-Ray throttling by implementing a custom sampling mechanism using SQS. This allows us to decouple trace generation from trace submission and apply post-processing sampling logic before sending data to X-Ray.
However, AWS Powertools does not currently offer a way to configure or intercept trace processing in a way that would allow us to implement similar throttling mitigation strategies.

Solution/User Experience

We would greatly benefit from a mechanism in Powertools that allows more control over how traces are handled before being sent to X-Ray. Two possible approaches could include:

• Configurable Sampling Logic: Allow users to define custom sampling strategies (e.g., probabilistic, rate-based, or dynamic sampling).
• Pluggable Trace Destination: Provide an option to redirect trace data to a custom destination (e.g., SQS, Kinesis, or a local buffer) for deferred or filtered submission.

We are open to any other solution that would enable similar flexibility in managing trace throughput under high-load conditions.

Thank you for considering this enhancement!

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Java) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, TypeScript, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[sdangol]

Hi @slavaschmidt, thank you for creating a feature request for this.
On a side note, the X-Ray SDK has announced a deprecation date and we're collecting customer anecdotes and data points to start working on OpenTelemetry (OTEL) support for Tracer, we have an ongoing RFC for OTEL support aws-powertools/powertools-lambda-python#7004. It would be great to have your input on what you'd like to see.

[sdangol]

Tagging other language maintainers to get feedback if this feature would be useful across other languages.
@dreamorosi @svozza @leandrodamascena @hjgraca

[phipag]

Hi @slavaschmidt,

thanks again for opening this feature request and also for sharing your pom.xml on your other issue.

As @sdangol pointed out, X-RAY SDK, which is used by Powertools, is currently on the deprecation path (details here). Therefore, we are not inclined to invest more effort on X-RAY SDK specific features.

Currently, we are considering to add OpenTelemetry support to our utilities which can open the door for Trace Sampling. You can find our public RFC for OTEL here.

We would love to learn more about your use-case to understand how we could help you in the best possible way. Would you be interested in a short meeting with us to discuss your use-case in more detail? If yes, please send an email to aws-powertools-maintainers@amazon.com and we can set this up.

[slavaschmidt]

Hi @phipag,

thank you so much for your interest in our use-case and invitation to participate. And sorry for the late response.

I'm currently trying to get one of the principal engineers and head of technology on board, and it probably will take some time.

[phipag]

Hey @slavaschmidt,

no problem. We've marked this issue as an idea for now and can re-visit once you have an update.