Commit 684bcc1
Refactor Velox Writer to Use New Flush Policy Contract (facebookincubator#242)
Summary:
Pull Request resolved: facebookincubator#242
This should be a no-op since no chunking flush policy is currently being used in Prod. but we make three changes in this dif:
1. `writeChunk` now returns a boolean to indicate whether any stream was successfully chunked
2. The previous raw size of the encoded stripe data in the writer context is now stored in the Writer context
3. We update and pass down the memory stats needed by the new flush policy contract
TODO: We will be introducing two more VeloxWriter changes in the next diffs in this stack to:
1. Support per stream chunking instead of always chunking all eligible streams
2. Support breaking down large stream into multiple smaller chunks
Reviewed By: helfman
Differential Revision: D81545433
fbshipit-source-id: 20f22d90ac23706a8a020990a7c865d03c4e091b1 parent 888deea commit 684bcc1
File tree
4 files changed
+547
-100
lines changed- dwio/nimble
- common
- velox
- tests
4 files changed
+547
-100
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
77 | 80 | | |
78 | | - | |
79 | | - | |
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
| |||
0 commit comments