Skip to content

Conversation

@clintropolis
Copy link
Member

Description

This PR fixes an issue with MSQ input file validation that results in creating aTooManyInputFilesFault from the number of an individual worker instead of totals across all workers. This lead to incorrect error messaging, since the error message advises on the minimum number of workers required to run the work order, so by only considering a single workers inputs it would advise to have at least the number of workers required to run that workers inputs, even though the number of actual workers might far exceed that.

For example, the adjusted test in this PR has 8 workers to run 100k inputs, before the changes in this PR the error message would be something like:

Too many input files/segments [14286] encountered. Maximum input files/segments per worker is set to [10000]. Try breaking your query up into smaller queries, or increasing the number of workers to at least [2] by...

The number of files is less than the 100k total for the job, and the "... increasing the number of workers to at least [2] ..." isn't helpful because we already have 8 workers.

@clintropolis clintropolis added Bug Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Dec 2, 2025
@clintropolis clintropolis merged commit 2534e62 into apache:master Dec 5, 2025
194 of 198 checks passed
@clintropolis clintropolis deleted the fix-msq-too-many-input-files-fault-reporting branch December 5, 2025 09:51
@kgyrtkirk kgyrtkirk added this to the 36.0.0 milestone Jan 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants