fix(FIR-50188): memory leak in streaming

[ptiurin]

This pull request significantly improves the streaming query result processing pipeline for large datasets, focusing on efficient memory usage, proper backpressure handling, and robust performance. It also introduces comprehensive integration and performance tests to benchmark and validate the new streaming behavior against traditional result fetching.

Streaming and Backpressure Improvements:

• Refactored the ServerSideStream class in src/statement/stream/serverSideStream.ts to implement a pending rows buffer, limit memory usage with a maxPendingRows threshold, and properly pause/resume the streaming interface to handle backpressure. The stream now pushes data downstream as consumers are ready, preventing memory buildup and ensuring efficient processing. Also, improved error handling and resource cleanup during stream destruction.

• Optimized the normalizeResponseRowStreaming function in src/statement/normalizeResponse.ts for performance by replacing Array.map with a manual for-loop and pre-allocated result array, reducing overhead for large datasets.

Testing and Validation:

• Added a new integration test suite in test/integration/v2/performance.test.ts to compare performance and correctness between normal (fetchResult) and streaming (streamResult) executions across various dataset sizes, including pipeline processing scenarios. The tests ensure that streaming is not more than 10% slower than normal execution and that both methods yield identical results.

Before (testing 1000000 rows with different data types):

After (memory now is stable over a long time with no large spikes):

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud