Support DocIdSetBuilder with partition bounds #15383

prudhvigodithi · 2025-10-30T15:57:23Z

Description

Coming from #14485 and #13745 (Initial implementation of intra-segment search concurrency #13542), when splitting a segment into partitions for intra segment search, each partition would create a DocIdSetBuilder that allocates memory based on the entire segment size, even though it only collects documents within a small partition range. This PR adds partition aware support to DocIdSetBuilder which creates bitsets and buffers scoped to its doc ID range instead of the entire segment size, this change will have memory efficiency during intra segment search.

Example for a Segment with 1M documents split into 4 partitions of 250K docs each and now each partition creates a FixedBitSet(1M) which is not required.

`PartitionAwareBufferAdder`:

Filters documents to only accept those within minDocId, maxDocId range.
Stores absolute doc IDs in buffers (used for sparse results below threshold) and rejects not part of of the partition range.

`PartitionAwareFixedBitSetAdder`

Filters documents to only accept those within partition range.
Uses partition sized bitset instead of segment sized.

`OffsetBitDocIdSet` & `OffsetDocIdSetIterator`

- FixedBitSet uses the doc ID parameter directly as an array index. When we create partition sized bitsets to save memory, we store documents using relative indices (0 to partitionSize-1) internally, but the Lucene API requires iterators to return absolute doc IDs. These wrapper classes handle the conversion automatically.
So these wrapper classes adds offset during iteration (when PartitionAwareFixedBitSetAdder is used). This is to convert partition relative indices back to absolute doc IDs.
Callers should always receive absolute doc IDs.

Segment: 100,000 documents
Partition: [50,000 to 60,000) - only 10,000 docs

Without Optimization (Old Way):

Create bitset for ENTIRE segment:
FixedBitSet(100,000 bits)

Bit position:  0     1     2  ... 50000 ... 50500 ... 55000 ... 59999 ... 99999
                ↓     ↓     ↓       ↓        ↓         ↓         ↓         ↓
Bit value:      0     0     0       1        1         1         1         0

With Optimization (New Way):

Create bitset with ONLY partition size:
FixedBitSet(10,000 bits)

Bit position:  0    1    2    ... 500  ... 5000 ... 9000 ... 9999
               ↓    ↓    ↓        ↓        ↓        ↓        ↓
Bit value:     0    0    0        1        1        1        0
               └───────────────────────────────────────────────┘
                All bits used efficiently!
                
Storage mapping (with offset):
  Doc 50,000 → Bit[0]     (50,000 - 50,000 = 0)
  Doc 50,500 → Bit[500]   (50,500 - 50,000 = 500)
  Doc 55,000 → Bit[5,000] (55,000 - 50,000 = 5,000)
  Doc 59,999 → Bit[9,999] (59,999 - 50,000 = 9,999)

Signed-off-by: Prudhvi Godithi <[email protected]>

prudhvigodithi · 2025-10-31T15:34:36Z

Hey all, pending to add some tests/validations and code clean up from my end but before this I would like to get some early feedback on the approach to see if the idea would make sense.

prudhvigodithi · 2025-10-31T16:10:53Z

Adding @jainankitk @getsaurabh02 to the conversation.

Signed-off-by: Prudhvi Godithi <[email protected]>

github-actions · 2025-11-03T12:09:17Z