[backend] SSE message backpressure mechanism

[xfournet]

Proposed changes

• handle SSE write backpressure

Related issues

• Fix Unbounded buffering on SSE responses can cause OOM with slow clients #13908

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case (coverage and e2e)
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

[codecov]

Codecov Report

❌ Patch coverage is 1.53846% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.79%. Comparing base (e6863b5) to head (3d4d3e1).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
...tform/opencti-graphql/src/graphql/sseMiddleware.js	1.53%	64 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #13909      +/-   ##
==========================================
- Coverage   30.86%   30.79%   -0.08%     
==========================================
  Files        2911     2912       +1     
  Lines      192486   193960    +1474     
  Branches    39265    39880     +615     
==========================================
+ Hits        59419    59726     +307     
- Misses     133067   134234    +1167     

Flag	Coverage Δ
opencti	30.79% <1.53%> (-0.08%)	⬇️
opencti-front	2.48% <ø> (+0.01%)	⬆️
opencti-graphql	68.33% <1.53%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xfournet]

i prefer to use function to group the code

[xfournet]

one flush per chunk is not necessary. should be done only at the end of all chunks

[xfournet]

if chunk is no more necessary, i prefer to remove it, since we can easily break the SSE message due to race conditions caused by this await. this race condition may already exist with heartbeat that are issued in //

[richard-julien]

Tested successfully.

[STREAM] Buffer draining { buffer: 65937, limit: 65536 }
[STREAM] Buffer draining { buffer: 66905, limit: 65536 }
[STREAM] Buffer draining { buffer: 65984, limit: 65536 }
[STREAM] Buffer draining { buffer: 65545, limit: 65536 }
[STREAM] Buffer draining { buffer: 66708, limit: 65536 }
[STREAM] Buffer draining { buffer: 65589, limit: 65536 }
[STREAM] Buffer draining { buffer: 65641, limit: 65536 }
[STREAM] Buffer draining { buffer: 67237, limit: 65536 }
[STREAM] Buffer draining { buffer: 66249, limit: 65536 }
[STREAM] Buffer draining { buffer: 66954, limit: 65536 }
[STREAM] Buffer draining { buffer: 65891, limit: 65536 }
[STREAM] Buffer draining { buffer: 66537, limit: 65536 }
[STREAM] Buffer draining { buffer: 66396, limit: 65536 }