-
Notifications
You must be signed in to change notification settings - Fork 201
Use a concurrent buffer deque in FragmentConsolidation. #5700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
c3a3e3a to
358d1b3
Compare
e70da9b to
47ea0fc
Compare
| string_size = 60 * 10485760; // 6 GB (over default memory budget of 5GB) | ||
| } | ||
|
|
||
| consolidate_sparse_string(string_size, true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this write only the one input cell?
If so then the write queue is always empty for this test.
We should have additional tests which contain huge cells which occur when the write queue has data in it. e.g. the buffer is 4G, the writer is processing 1G, there are two buffers queued at 1G each, and the next cell is 2G.
We must have coverage for waiting for there to be enough space to read more data.
rroelke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to add new tests for some of the edge cases of the new implementation before we can merge this.
5010ce2 to
0ed52b6
Compare
0ed52b6 to
77c355d
Compare
The current implementation of
FragmentConsolidationalways sets up a very large buffer space (~ 10GB) and performs consolidation within that workspace. This work aims to reduce the memory footprint and latency of fragment consolidation through use of aProducerConsumerQueuefor concurrent reads/writes. At present, the buffers are (somewhat arbitrarily) sized at 10MB, and the queue is capped at size 10. As such, there are quantitatively more allocations, but the overall size (and runtime) is drastically reduced, as small operations need not construct / destruct the full workspace.The following test saw an improvement in runtime of
16.500s->0.130s:./test/tiledb_unit --durations yes --vfs=native "C++ API: Test consolidation that respects the current domain"TYPE: IMPROVEMENT
DESC: Use a concurrent buffer deque in
FragmentConsolidation.Resolves CORE-411.