Skip to content

Commit ea5d292

Browse files
perf: optimize memory usage in denoised-rows query by conditionally including processedRows (#1177)
When the searching row limits is set very high (ex the max of 100k) the app quickly consumes all available memory and crashes. This adds some improvements to help mitigate the problem: 1. **QueryKey Issues** - The `queryKey` is generating a ton of extra entries every time the `processedRows` changes (which is every 5s when in live mode). The queryKey and result is cached regardless of if enabled is true or false. The base hashFn strategy is to [stringify the objects](https://github.com/TanStack/query/blob/2a00fb6504e777fa1a9c9a46ce9f1b7ccdb3835a/packages/query-core/src/utils.ts#L216-L217) which creates a very large string to be stored in memory. I tried to fix this by providing a custom `queryKeyHashFn` to `useQuery` but it was too slow, and the faster browser based hashing fns return a promise which isn't supported by `useQuery` at this time. The easiest solution I found was to short circuit the hash generation if we are not denoising. 2. **Sync `gcTime`** - We already set `gcTime` in `useOffsetPaginatedQuery` so I added that field here too, this helps keep the memory usage lower while denoising rows (but the memory still is much higher). **The app still uses very high memory usage, just from the sheer number of rows being captured and processed**, but it doesn't crash anymore. There is definitely further optimizations we could make to reduce this. One solution that comes to mind is storing a hash/unique id of each row server side before sending to the client, then our app can leverage this key instead of a stringified object. Before (after 1 min): <img width="645" height="220" alt="Screenshot 2025-09-17 at 4 05 59 PM" src="https://github.com/user-attachments/assets/dab0ba34-4e92-42ce-90a0-fefadd9f0556" /> After (after 5 mins): <img width="1887" height="940" alt="Screenshot 2025-09-17 at 3 52 23 PM" src="https://github.com/user-attachments/assets/bd969d2a-f0ec-4a5a-9858-409ff4a1eaa1" /> Fixes: HDX-2409
1 parent 8a1762e commit ea5d292

File tree

2 files changed

+15
-1
lines changed

2 files changed

+15
-1
lines changed

.changeset/sixty-mangos-join.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@hyperdx/app": patch
3+
---
4+
5+
Improve memory efficiency in high row cound envs

packages/app/src/components/DBRowTable.tsx

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import cx from 'classnames';
1010
import { format, formatDistance } from 'date-fns';
1111
import { isString } from 'lodash';
1212
import curry from 'lodash/curry';
13+
import ms from 'ms';
1314
import { useHotkeys } from 'react-hotkeys-hook';
1415
import {
1516
Bar,
@@ -1268,11 +1269,18 @@ function DBSqlRowTableComponent({
12681269
queryKey: [
12691270
'denoised-rows',
12701271
config,
1271-
processedRows,
1272+
denoiseResults,
1273+
// Only include processed rows if denoising is enabled
1274+
// This helps prevent the queryKey from getting extremely large
1275+
// and causing memory issues, when it's not used.
1276+
...(denoiseResults ? [processedRows] : []),
12721277
noisyPatternIds,
12731278
patternColumn,
12741279
],
12751280
queryFn: async () => {
1281+
if (!denoiseResults) {
1282+
return [];
1283+
}
12761284
// No noisy patterns, so no need to denoise
12771285
if (noisyPatternIds.length === 0) {
12781286
return processedRows;
@@ -1296,6 +1304,7 @@ function DBSqlRowTableComponent({
12961304
}
12971305
return undefined;
12981306
},
1307+
gcTime: isLive ? ms('30s') : ms('5m'), // more aggressive gc for live data, since it can end up holding lots of data
12991308
enabled:
13001309
denoiseResults &&
13011310
noisyPatterns.isSuccess &&

0 commit comments

Comments
 (0)