Skip to content

Conversation

@gabotechs
Copy link
Collaborator

@gabotechs gabotechs commented Oct 20, 2025

Enhances the way we configure the network_coalesce_tasks and network_shuffle_tasks parameters. Rather than having these parameters as part of the DistributedPhysicalOptimizerRule, they are now it's own structure that implements ConfigExtension, the standard DataFusion way of passing configuration.

This allows us to ship two main enhancements:

  • Now, both parameters can be set in SQL statements as any other parameters, for example:
SET distributed.network_coalesce_tasks = 2;
SET distributed.network_shuffle_tasks = 4;

SELECT ...
  • Now, upon setting them programatically, a dynamic number that depends on the the underlaying plan can be specified:
        let state = SessionStateBuilder::new()
            .with_default_features()
            .with_physical_optimizer_rule(Arc::new(DistributedPhysicalOptimizerRule))
            .with_distributed_network_coalesce_tasks(|plan: &Arc<dyn ExecutionPlan>| {
                if plan.as_any().is::<Something>() {
                    1
                } else {
                    4
                }
            })
            .build();

@gabotechs gabotechs force-pushed the gabrielmusat/add-dynamic-task-config branch from e2fee7f to c5698c2 Compare October 20, 2025 17:34
Copy link
Collaborator

@NGA-TRAN NGA-TRAN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.
I only have minor questions to remind myself

Base automatically changed from gabrielmusat/refactor-distributed-planner to main October 22, 2025 14:50
# Conflicts:
#	src/distributed_planner/distributed_physical_optimizer_rule.rs
#	src/distributed_planner/mod.rs
#	src/lib.rs
@gabotechs gabotechs merged commit 4ead8f1 into main Oct 22, 2025
4 checks passed
@gabotechs gabotechs deleted the gabrielmusat/add-dynamic-task-config branch October 22, 2025 15:11
@gabotechs
Copy link
Collaborator Author

I'm finding this API a bit annoying to work with in the end...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants