-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Description
Add support for partitioned tables by leveraging directory-based partitioning (e.g. year=2024/month=01/) to prune files during query planning. Partition values are extracted from file paths and used to skip non-matching partitions based on query predicates. Partition columns are materialized as virtual columns without reading Parquet data. This enables significant scan reduction for partitioned datasets and aligns DataFusion–DuckLake with common lakehouse partitioning semantics.
Current behavior: Scans all files regardless of partition filters
Desired behavior: Skip files in non-matching partitions
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request