UI to visualize and reprioritize scheduled+pending tasks

[escapewindow]

I'm not sure if this is filed elsewhere. We've wanted this for a long time, to both help debug what's causing large pending queues, and to bump more urgent tasks to the front of the queue rather than keep it buried behind thousands of less-urgent tasks. We held off on this until we were on Postgres. Aiui that project is now over, and we are now able to implement something like this.

[lotas]

@escapewindow do you have something concrete in mind?
Can you please give more details, I would be happy to implement this, if it will help debug things faster.

[escapewindow]

I don't know what the UI and/or API should look like specifically, no :)

We probably want use cases like:

• Which tasks are running/unscheduled/pending per repository? commit?
• Which tasks are running/unscheduled/pending per pool? when/why were they scheduled? what are their priorities?
• Can we adjust the priorities of a subset of those?

We'll probably find that certain parts of the task definitions are of particular interest. We may also find that for this to work more programmatically, we need additional metadata.

(For instance, I'm not sure if there's a consistent way of determining the repo and revision. For Firefox Desktop, we set task.payload.env.{GECKO_HEAD_REF,GECKO_HEAD_REV,GECKO_HEAD_REPOSITORY} and we have an hg-style task.metadata.source that we can regex parse, but for Firefox Mobile we'd have MOBILE_HEAD_REF etc and a github-style task.metadata.source that we can regex parse. Most likely we'll implement something that works with the current data, and file bugs for Releng to add additional metadata to each task.)

[lotas]

I will add a comment from @Archaeopteryx which seems to touch the same topic:

Task load is opaque: scheduled, pending, running and completed tasks cannot be grouped by worker group

[lotas]

This is also related to taskcluster/taskcluster#2939