Skip to content

Commit 5cdd56b

Browse files
authored
fix: change prefix ordering to align with sort order (#549)
With the PR #548 we introduced natural ordering of the parquet files, with the latest events at the top of a file. This warrants a change in the prefix order passed to the data fusion table provider. This PR ensures the prefix order is as expected.
1 parent 418db72 commit 5cdd56b

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

server/src/query.rs

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,8 @@ impl Query {
8484
self._get_prefixes()
8585
.into_iter()
8686
.map(|key| format!("{}/{}", self.stream_name, key))
87+
// latest first
88+
.rev()
8789
.collect()
8890
}
8991

@@ -176,7 +178,8 @@ impl Query {
176178
.with_file_extension(".parquet")
177179
.with_file_sort_order(file_sort_order)
178180
.with_collect_stat(true)
179-
.with_target_partitions(32);
181+
// enforce distribution will take care of parallelization
182+
.with_target_partitions(1);
180183

181184
let config = ListingTableConfig::new_with_multi_paths(prefixes)
182185
.with_listing_options(listing_options)

0 commit comments

Comments
 (0)