Skip to content

Conversation

@bekadavis9
Copy link
Contributor

The current implementation of FragmentConsolidation always sets up a very large buffer space (~ 10GB) and performs consolidation within that workspace. This work aims to reduce the memory footprint and latency of fragment consolidation through use of a ProducerConsumerQueue for concurrent reads/writes. At present, the buffers are (somewhat arbitrarily) sized at 10MB, and the queue is capped at size 10. As such, there are quantitatively more allocations, but the overall size (and runtime) is drastically reduced, as small operations need not construct / destruct the full workspace.

The following test saw an improvement in runtime of 16.500s -> 0.130s:
./test/tiledb_unit --durations yes --vfs=native "C++ API: Test consolidation that respects the current domain"


TYPE: IMPROVEMENT
DESC: Use a concurrent buffer deque in FragmentConsolidation.


Resolves CORE-411.

@bekadavis9 bekadavis9 force-pushed the rd/core-411 branch 2 times, most recently from e70da9b to 47ea0fc Compare December 5, 2025 18:35
@bekadavis9 bekadavis9 requested a review from rroelke December 5, 2025 19:19
string_size = 60 * 10485760; // 6 GB (over default memory budget of 5GB)
}

consolidate_sparse_string(string_size, true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this write only the one input cell?
If so then the write queue is always empty for this test.
We should have additional tests which contain huge cells which occur when the write queue has data in it. e.g. the buffer is 4G, the writer is processing 1G, there are two buffers queued at 1G each, and the next cell is 2G.
We must have coverage for waiting for there to be enough space to read more data.

Copy link
Member

@rroelke rroelke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to add new tests for some of the edge cases of the new implementation before we can merge this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants