Allow batch executors to share the parallel pool with discrete tasks #36308
jgannonjr
started this conversation in
Feature Requests
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Description
Batch executors (
batchImplementation/preferBatch) currently monopolize the task orchestrator: while a batch is running, no discrete tasks start, even when--parallel/nx.jsonparallelhas spare capacity.Add an opt-in executor setting (e.g.
batchParallelismCost) so a batch counts as N parallel slots (typically1) instead of blocking the coordinator. Discrete tasks whose dependencies are satisfied can then run in the remaining capacity.Motivation
In mixed monorepos (e.g. TypeScript + Cargo),
preferBatch: trueis valuable for coalescing many crate tasks into onecargo check -p a -p b -p c, but it serializes the entirerun-many/affectedpipeline behind that batch. TypeScripttypecheck/lint/buildsit Waiting until Cargo finishes.Today the tradeoff is:
preferBatch→ lose multi--pcoalescingCore support for “batch occupies 1 parallel slot” fixes that for any batch executor (Cargo plugins, Gradle/Maven if they opt in, custom batch executors)—without changing default batch behavior.
Suggested Implementation
On executor registration (
executors.json/ExecutorConfig):{ "implementation": "./executor", "batchImplementation": "./batch", "preferBatch": true, "batchParallelismCost": 1 }All reactions