Commit f36586d
Integrate stats builder into field writer (facebookincubator#444)
Summary:
Pull Request resolved: facebookincubator#444
Use the stats builder to collect stats in field writer for the respective writer types, replacing the current stats. Additionally fixing:
* missing key stats in Flatmaps
* wrong value stats attribution
* thread safety in chunking write paths
Notably we also added a thread safe wrapper for flatmap value writers to merge their stats concurrently for now, to be extended to collect separate feature/map stats later. Collecting separate streams right now and merging later is additional costs in both wall time and memory that we don't need to pay unless one wants feature/map stats.
NOTE: this diff currently disables the physical aggregation, and it will be fully restored with the diff replacing it with the new physical aggregation call sites.
Reviewed By: helfman
Differential Revision: D90166052
fbshipit-source-id: 87b7b1b7fc6c1c9682383fd446a0a52bd6cf81651 parent 388ca35 commit f36586d
File tree
11 files changed
+2130
-326
lines changed- dwio/nimble/velox
- selective/tests
- stats
- tests
11 files changed
+2130
-326
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
| 70 | + | |
69 | 71 | | |
70 | 72 | | |
71 | 73 | | |
| |||
172 | 174 | | |
173 | 175 | | |
174 | 176 | | |
175 | | - | |
| 177 | + | |
176 | 178 | | |
| 179 | + | |
177 | 180 | | |
178 | 181 | | |
179 | 182 | | |
| |||
0 commit comments