Skip to content

Conversation

@cmanallen
Copy link
Member

@cmanallen cmanallen commented Oct 8, 2025

msgspec allows us to partially deserialize a JSON payload. By removing the requirement to deserialize full and incremental snapshots, we a significantly increase the performance of the deserialization and significantly reduce the memory consumption.

This is particularly important as our consumer suffers from memory usage issue and our single-threaded processing logic is a bottleneck to throughput improvements. Experiments demonstrated a 10x performance uplift and memory usage declined to, essentially, nil.

There is a downside. We can no longer read data from incremental snapshots or full snapshots. To do so would require allocating memory and defeat the purpose of this PR. Presently we don't do anything with those events except log some canvas events which, I think, are unimportant enough we can remove them.

Full plan: https://www.notion.so/sentry/Q4-25-Recording-Consumer-Improvements-28c8b10e4b5d803cb212e19aebefee81

Related: getsentry/pypi#1695

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 8, 2025
@codecov
Copy link

codecov bot commented Oct 8, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##           master   #101195    +/-   ##
=========================================
  Coverage   81.02%    81.03%            
=========================================
  Files        8701      8703     +2     
  Lines      385959    386418   +459     
  Branches    24413     24413            
=========================================
+ Hits       312737    313126   +389     
- Misses      72871     72941    +70     
  Partials      351       351            

@cmanallen cmanallen marked this pull request as ready for review October 14, 2025 13:36
@cmanallen cmanallen requested a review from a team as a code owner October 14, 2025 13:36
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

@cmanallen cmanallen requested a review from a team as a code owner October 15, 2025 13:21
@cmanallen cmanallen merged commit b7f489c into master Oct 15, 2025
67 checks passed
@cmanallen cmanallen deleted the cmanallen/replays-consumer-parsing-perf branch October 15, 2025 16:14
@github-actions github-actions bot locked and limited conversation to collaborators Oct 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants