Concourse worker tags priority or weight

[danpilch]

Hi all,

I want to know if there is any solution for worker tags priority/weighting. I'll give you an example:

We have a tag called integration this type of (high resource) worker runs integration tests and is only available on a time schedule e.g. 9-5 UTC)

We want to be able to failover any emergency builds to another worker IF there is no high resource integration worker available.

Is there any possible way to achieve this in concourse v7?

Thanks

[taylorsilva]

Hey, currently there is no way to do that. This workflow makes sense though and would be a nice use-case to support since I'm sure not having those heavy workers around 24/7 saves a lot of money.

[haydonryan]

@taylorsilva would you consider this as a feature now that you're full time on concourse (congratulations by the way!). If I can get enough time, I might even take a stab at it.

There are so many use cases for this including running it.

I was thinking of just a simple environment variable that gets applied to the worker. It reports this to atc?(I think) then when selecting a worker the free worker with highest priority number wins. That would be a great v1. V2 could be to round robin if there were priority numbers that were the same.

[taylorsilva]

I'd want more details about what you're thinking about here, especially since OP mentioned worker tags. Right now if you try and run a step with a tag and there are no workers with that tag, the step and job will fail. How would weighing tags work with this? How will Concourse know, for any given step with a tag, if it should only schedule the step on a worker with that tag, or of it's okay to schedule that step on another worker without that tag?

[haydonryan]

So this would be separate to the tag, and be something along the lines of:

Gather all workers with the tag specified
    if none available, then fail.
    Select worker with highest priority
       if more than one worker has highest priority then round robin / select first in list.
if no tags, just specify the highest priority worker.

So lets say we have 4 workers. In this example the priority is lower is higher)

• 1. Dell rack mount server very fast. (tags= none, priority= 1)
• 2. Mac mini (tags= macos, priority= 2)
• 3. Old mac mini (tags=macos, priority= 4)
• 4. Old mac mini (tags=macos, priority= 4)
• 5. Low power nas ( tags= none, priority= 5).

This way if you had a job that was tagged for macos, then worker2 would be selected, as it has the highest priority with the right tag. If this was busy then worker3 or worker4 would be used.

If no tags were specified, then it would select worker1, if that was busy, the mac mini would be selected, etc.
If no tags were specified, worker2-4 were offline, then it would choose worker1 first, and if worker1 was busy then worker5.

This way you're always getting the highest priority worker that would be capable of building that pipeline (at the highest speed). I hope this makes sense.

This would cover the tag and failover scenarios that OP mentioned.

[taylorsilva]

Okay, I understand what you're suggesting. My first thought is how would folks who don't want this behaviour opt-out of it? Even in this case from OP, his worker has a certain characteristic (large amount of resources) that the tag represents. How would a pipeline writer indicate when a tagged step MUST only run on tagged workers and when it's okay to fallback to any untagged worker?

The pipeline we use here has this exact case, where we only want this step to run on a worker with the bosh tag: https://github.com/concourse/ci/blob/d0c33276223dac65799851d46c96628a06862650/pipelines/concourse.yml#L966-L967

I guess most of my thoughts/questions come down to how the implementation of this idea would work. Would it be a new container placement strategy? But then that wouldn't allow for the original use-case of tags for a cluster that uses that placement strategy.

I think there's quite a few things to work through here. I suggest submitting an RFC to explore this idea further: https://github.com/concourse/rfcs/

[haydonryan]

Thanks - will do - after seeing the container placement strategy I get how this is a lot more complicated than I originally thought. Creating RFC...