generate tasks within a kind concurrently

[bhearsum]

In #5 we've discussed parallelizing the loading of kinds to improve decision task performance. One of the discoveries in that thread was that for Gecko (where we have the longest decision task), this would only incrementally improve performance. As it turns out, most of the work is inside of the test kind, and kind-level concurrency won't help there at all.

Something we could do to improve generate within a kind is to concurrently run transforms for individual tasks. This is currently done serially over here, and AFAICT, there should be nothing stopping us from running those across multiple threads or processes. It's unclear to me if this will be a massive win, but in a quick test it appears there are ~22,000 calls to transforms for the tests kind, so presumably it would be something.

We may need some sort of intelligent way to push these into worker threads/processes; it's possible that for kinds with just a few tasks that the overhead of setup/teardown would be more than the reduction we'd get from the concurrency. Perhaps a minimum number tasks should be needed before spawning workers (either an absolute number, or one relative to the number of workers we'd spawn, which would presumably be based on the # of cores available).

[ahal]

Arguably this is something the test transforms in Gecko should handle, as in general at that point of the process there are only as many tasks as there are stanzas in the kind.yml file, which typically isn't that many, and then the # of tasks get split out later.

Then again, the Gecko tests do use a specialized loader, so in their case they would benefit from parallelizing there much more so than any other kinds.

[bhearsum]

Arguably this is something the test transforms in Gecko should handle, as in general at that point of the process there are only as many tasks as there are stanzas in the kind.yml file, which typically isn't that many, and then the # of tasks get split out later.

This is a good point. The kind the next highest number of tasks is beetmover-repackage with 1,584. However, all of those are generated in transforms from the single entry in the kind...so this idea would not help there at all. I haven't looked at all of the other kinds with high numbers of tasks, but unless they're using custom loaders the same is likely true for them.

Then again, the Gecko tests do use a specialized loader, so in their case they would benefit from parallelizing there much more so than any other kinds.

Given how significant tests are to taskgraph generation, this might be reason alone to it in the short term. I imagine that in the future the way test tasks are defined and generated might be different, so the calculus might change in the future?

The other obvious way to improve test generation is to split it into multiple kinds (as suggested in #5). I haven't dug into that, but it feels like it could be a rats nest of complexity. Perhaps there's other ways to improve this that we haven't thought of yet.