Fix SearchContext CB memory accounting #138002
Conversation
| } | ||
| if (context.checkRealMemoryCB(locallyAccumulatedBytes[0], "fetch source")) { | ||
| // if we checked the real memory breaker, we restart our local accounting | ||
| locallyAccumulatedBytes[0] = 0; |
There was a problem hiding this comment.
Most of the time these batches were too small, so this didn't trigger.
| } | ||
| RankFeatureShardPhase.prepareForFetch(searchContext, request); | ||
| fetchPhase.execute(searchContext, docIds, null); | ||
| fetchPhase.execute(searchContext, docIds, null, i -> {}); |
There was a problem hiding this comment.
These did not account for memory before anyway, but now they could.
The hard part is releasing the CB; I don't see close+relase logic around here, and I'm not very familiar with this code. Maybe this could be a follow-up
| docIdsToLoad[i] = topDocs.scoreDocs[i].doc; | ||
| } | ||
| FetchSearchResult fetchResult = runFetchPhase(subSearchContext, docIdsToLoad); | ||
| FetchSearchResult fetchResult = runFetchPhase(subSearchContext, docIdsToLoad, this::addRequestCircuitBreakerBytes); |
There was a problem hiding this comment.
Should we batch here and avoid invoking the CB for every document?
Maybe addRequestCircuitBreakerBytes should take care of this?
I suspect that fetching source is way more expensive than invoking the CB, so I'm not sure we want more complication here.
elasticsearchmachine
added
the
Team:Analytics
|
Hi @luigidellaquila, I've created a changelog YAML for you. |
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
| public final boolean checkRealMemoryCB(int locallyAccumulatedBytes, String label) { | ||
| if (locallyAccumulatedBytes >= memAccountingBufferSize()) { | ||
| circuitBreaker().addEstimateBytesAndMaybeBreak(0, label); | ||
| circuitBreaker().addEstimateBytesAndMaybeBreak(locallyAccumulatedBytes, label); |
There was a problem hiding this comment.
This was the crux
2 participants
Fixing TopHits memory management
Problems and TODO (spotted so far):
SearchContext.checkRealMemoryCBdoesn't account for CB memory (always zero)FetchPhase.buildSearchHitsbatches are too small, the memory buffer never accumulates enough to be trackedTopHitsAggregatormemory management lifecycleInnerHitsPhasememory management lifecycleSearchService.execute*Phase()memory management lifecycleTopHitsAggregator.subSearchContext.closeFuturegrows too much - this is due to this block, so it's irrelevant in prod.Fixes: #136836