Memory leak with python sentry-sdk 3.0.0a6

[mukesh-hira]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

3.0.0a6

Steps to Reproduce

Enable sentry-sdk with default parameters in our python FastAPI server with a sampling rate of 0.1. Run a load test using k6 that emulates a 1000 virtual users with spikes of search APIs. Python RSS and pod memory usage continues to grow up to 3GB and beyond as long as the load test keeps running. At the end of the load test, RSS reaches a plateau but doesn't come back down even hours after the load test stops. We are seeing high memory utilization in our prod environment too, but the isolated load test provides a better perspective as in that environment activity returns to zero and thus memory utilization should clearly come back down.

The memory leak was narrowed down by running tracemalloc periodically and it showed open telemetry responsible for bulk of the allocations. Changing sentry-sdk sampling rate to 0 makes the system behave very well with very low memory utilization, eliminating the possibility of memory leaks in other parts of the code.

It seems that use of open telemetry in sentry-sdk is new in 3.0 and the current stable 2.86 release doesn't use open telemetry under the hood. Thus it is possible that the mem leak we are experiencing has to do with the specific release of sentry-sdk we are running with new open telemetry integration.

Expected Result

Expected result - memory utilization should come back down to a small level after stopping load test.

Actual Result

Memory utilization stays at the level it reaches at the stop of load test.

[linear]

PY-1856 Memory leak with python sentry-sdk 3.0.0a6

[sentrivana]

@mukesh-hira Hey, thanks a ton for trying out the alpha and reporting! This is very valuable feedback. v3.0 does indeed use OpenTelemetry under the hood for tracing instead of our home-baked Sentry solution.

Based on your description, I agree that tracing in the new SDK seems like the most likely culprit since setting traces_sample_rate=0 eliminates the issue. Would you be able to run the load test using the current stable version (2.38.0 at the point of writing) just to completely rule out anything else?

Also, any further details regarding the load test setup and the tracemalloc would be valuable, for instance:

• Did you see what specifically was using up most of the memory / what stayed allocated after the load test? Can you share the output?
• Can you tell us more about your FastAPI server:
  • Is there a database behind it, or something else the SDK is instrumenting? A list of installed packages would also be helpful if you can share that.
  • Can you post your sentry_sdk.init()? This might also help us determine which specific parts of the SDK are active.

We did some load tests on the alpha previously and memory utilization returned to the baseline afterwards, so we need to figure out what case we missed.

[mukesh-hira]

Update - we ran the load tests with sentry-sdk 2.38 and the system behaves very well - memory growth is much more limited even during the load tests which probably implies that references are being freed up and gc actively frees up memory. It turns out that another issue observed in load tests of big spikes in response latencies for some API calls were also related to sentry-sdk alpha version and/or open-telemetry. We didn't think these would be related to sentry and were chasing down other paths like examining our DB etc. But with sentry-sdk reverted to 2.38 all API response latencies are back to good levels and there is no more RSS memory growth to large levels. Here is the sentry_sdk.init - pretty standard stuff, nothing special sentry_sdk.init(
dsn=server_config.sentry_dsn,
environment=server_config.production_environment,
traces_sample_rate=server_config.sentry_traces_sample_rate,
send_default_pii=server_config.sentry_send_default_pii)

sentry_traces_sample_rate: float = 0.1
sentry_send_default_pii: bool = True

[sentrivana]

Thank you @mukesh-hira! We appreciate the feedback and will look into this before we make a stable release of 3.0.