Skip to content

Commit f36586d

Browse files
Huameng (Michael) Jiangmeta-codesync[bot]
authored andcommitted
Integrate stats builder into field writer (facebookincubator#444)
Summary: Pull Request resolved: facebookincubator#444 Use the stats builder to collect stats in field writer for the respective writer types, replacing the current stats. Additionally fixing: * missing key stats in Flatmaps * wrong value stats attribution * thread safety in chunking write paths Notably we also added a thread safe wrapper for flatmap value writers to merge their stats concurrently for now, to be extended to collect separate feature/map stats later. Collecting separate streams right now and merging later is additional costs in both wall time and memory that we don't need to pay unless one wants feature/map stats. NOTE: this diff currently disables the physical aggregation, and it will be fully restored with the diff replacing it with the new physical aggregation call sites. Reviewed By: helfman Differential Revision: D90166052 fbshipit-source-id: 87b7b1b7fc6c1c9682383fd446a0a52bd6cf8165
1 parent 388ca35 commit f36586d

File tree

11 files changed

+2130
-326
lines changed

11 files changed

+2130
-326
lines changed

dwio/nimble/velox/CMakeLists.txt

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ add_library(
6666
)
6767
target_link_libraries(
6868
nimble_velox_field_writer
69+
nimble_column_stats
70+
nimble_column_stats_utils
6971
nimble_velox_schema
7072
nimble_velox_stream_data
7173
nimble_velox_schema_builder
@@ -172,8 +174,9 @@ add_library(
172174
)
173175
target_link_libraries(
174176
nimble_velox_writer
175-
nimble_common
177+
nimble_column_stats
176178
nimble_column_stats_utils
179+
nimble_common
177180
nimble_encodings
178181
nimble_index
179182
nimble_tablet_writer

0 commit comments

Comments
 (0)