Introduce max_compacted_partitions

[utay]

Currently, max_compacted_files in ducklake_merge_adjacent_files limits the number of compaction operations created per partition group, not globally across all partitions.

This means with a partitioned table having 3 partitions with 2 files each:

• max_compacted_files => 1 will still create up to 3 compaction operations (one per partition)
• There's no way to limit compaction to only the first N partitions

For incremental compaction, it would be useful to compact just 1 file from 1 partition.

I see two possible approaches:

1. Make max_compacted_files apply globally across all partitions
2. Introduce a new max_compacted_partitions setting to control this independently

Happy to hear your thoughts!

[Alex-Monahan]

Another option I have been thinking about is if a compaction operation could target a specific partition. That could enable something like "Compact last month's data by compacting partition year(ts)=2026 month(ts)=01" or similar. What do you think?

[utay]

@Alex-Monahan I would still like to have something like max_compacted_partitions when I don't want to target a specific partition, but I think it would be a great complementary feature! Sounds very much like #777 (cc @admin-one211) and could also help #778 (cc @tanejagagan) if there's support for both including and excluding partition paths

[tanejagagan]

added my comment to #777
This is how i am working on implementation