Make MaximumTaskCall configurable

[mjpieters]

Description

I see that this was punted before but the hard-coded limit was simply raised rather that made configurable.

Now that there are matrix loops I am hitting the limit again, and want to be able to raise this for certain tasks.

My use case is that I have a matrix of Python versions, project names, and per-project versions (generated) to run a task against. Because the per-project version list is generated based on project name and Python version, I have to use a 3-tiered task approach:

• 'generate all' task, matrix over python versions (list of 3) and project names (list of 11 names currently), so executing 33 times
  • 'generate project data', generates a list of versions for a given Python version and project name. This loops anywhere between 20 and 85 times, depending on the exact inputs. The average is 44
    • 'generate project data for a given version' task. This task has a status command, for a given project name and version, only a single file needs to be generated so usually only one of the given Python versions would produce output, the others would be skipped.

The innermost task can be executed up to 33 times 44 is 1452 times, ultimately producing 484 files. The hardcoded 1000 items limit is in the way here.

I can make this work by copying the status configuration to the intermediate task, cutting this down to 484 invocations, but this project is expected to grow some more and I forsee a future where even the middle task is going to be invoked more often than 1000 times, so please let me lift this limit when I know what I am doing?

[mjpieters]

Correction: I can't work around this with a status configuration on the middle task, because that requires the project version variable to exist and we can't test conditionally skip for loop iterations while invoking a dependency task.

So my current work-around is to have the script that generates the versions filter the versions down to those that have no generated file present, defeating the purpose of putting this in task definitions: to have a single source of truth about what files need generating. :-(

[trulede]

You might try this PR and see if it helps: #2287.

Although I wonder about your approach ... if its really the best way.

[trulede]

Duplicate of #820

There is a PR available that fixes the underlying issue. If you try it and it improves your situation, please leave a comment.

Adding an ENVAR to make this configurable would be trivial, however that might bring with it unintended consequences.