Commit 44e10f4
feat: Support stripe level batched index read (#16360)
Summary:
Pull Request resolved: #16360
X-link: facebookincubator/nimble#480
This diff implements stripe-level batched index read support for Nimble files in Velox. The key motivation is to improve index lookup performance by processing multiple lookup requests in batches at the stripe level, rather than processing each request individually.
**New `SelectiveNimbleIndexReader` class**: A format-specific index reader that handles:
- Encoding index bounds into Nimble-specific encoded keys
- Looking up stripes and row ranges using the tablet index
- Managing stripe iteration and data reading with batched processing
- Returning results in request order via an iterator pattern (`startLookup`/`hasNext`/`next`)
**Batched stripe processing**: Instead of loading stripes per-request, the reader:
- Maps all lookup requests to their matching stripes upfront
- Merges overlapping row ranges within stripes for efficient reading (without filters)
- Splits overlapping ranges into non-overlapping segments to preserve filter semantics (with filters)
- Tracks output references with ref-counting to share read data across requests
**HiveIndexReader refactoring**: Simplified to focus on index bounds creation and result assembly, delegating format-specific control logic to `SelectiveNimbleIndexReader`.
**KeyEncoder enhancements**: Added support for encoding index bounds with constant values for efficient multi-row range queries, and extended test coverage for edge cases.
**New runtime stats**: Added metrics for tracking index lookup performance:
- `kNumIndexLookupRequests`: Total lookup requests processed
- `kNumIndexLookupStripes`: Number of stripes accessed
- `kNumIndexLookupReadSegments`: Number of read segments processed
Reviewed By: HuamengJiang, tanjialiang
Differential Revision: D92848948
fbshipit-source-id: d6f5c78ef23dca1d4d8faa2d27acf85f776128051 parent 54f4662 commit 44e10f4
File tree
11 files changed
+1609
-541
lines changed- velox
- connectors/hive
- docs/monitoring
- dwio/common
- serializers
- tests
11 files changed
+1609
-541
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
371 | 371 | | |
372 | 372 | | |
373 | 373 | | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
374 | 398 | | |
375 | 399 | | |
376 | 400 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
90 | 109 | | |
91 | 110 | | |
92 | 111 | | |
| |||
0 commit comments