Any ideas for secure, server-controlled worker selection?

[moqmar]

Hi there! I'm a member of the Codeberg e. V., a non-profit git hosting service, and we want to offer our users a CI service, using a "bring-your-own-worker" approach, like you may know it from GitLab CI.

I think that Concourse would be a good choice for that, but there's some feature that's missing for a use-case like this, just like in all alternatives, and we will probably have to build it ourselves: how to securely let the CI server choose a worker that can then access the secrets and build a binary that the user trusts?

As the first approach that comes into my mind, we could use Concourse's tags as a secret, but this would require (if I haven't forgotten anything) that...

• ...the workers can't see each other's worker tags (is that the case?)
• ...build secrets are only sent to the worker chosen for a build (I guess that's the case?)
• ...the server can set tags independently from the configuration (I guess we can do that in JobConfig.StepConfig() and db.Resource.Tags())
• ...if organizations and repos can get their own secret tag, and both organization-bound workers and repo-bound workers shall be usable by a job, the configuration sent to the worker must only contain the tag meant for this worker (I haven't found yet where I'd change the configuration sent to the worker)

Would that be a viable way, have I understood the architecture of Concourse correctly, or is there a simpler way to choose the exact worker node (or a set of possible nodes) programmatically that I have overlooked so far?

On another note, is that generally a feature we should think about as an upstream change & PR for here, or is it out of scope for Concourse and we should only implement it in a fork?

[taylorsilva]

One thing I didn't see you mention is team workers:

concourse/worker/workercmd/worker_config.go

Line 12 in 9bd016f

TeamName string `long:"team" description:"The name of the team that this worker will be assigned to."`

That allows you to assign one or more worker to a single team. No need for tags! We currently do this for our our hush-house deployment (ci-aas that we offer internally to our org) where teams can choose to bring their own workers that other teams can't use:
https://github.com/concourse/hush-house/blob/f323dcb2dbc097fa10ae61f859fa1d60037faa98/deployments/with-creds/hush-house/values.yaml#L9-L33
Haven't tested this but I'm pretty sure you can have a Concourse that only has team workers and zero tagged/team workers. Everything should Just Work ™️ and users don't have to tag every step/job to run in a group of workers.

Further more, you can have also have team workers that are tagged. So if a team wants to isolate some of their jobs/steps, they can deploy a team worker with a tag.

that can then access the secrets

the web node is the component that does all secret fetching currently and sends it to the worker as part of the step plan. Secrets are cached in-memory. Not sure if you knew that so stating it just in-case :)

There are some features on the horizon that may be of interest to you, specifically var_sources (var_sources rfs) which allow users to bring their own vault (for now) and eventually their own secret storage. Or this could possibly give you a more secure way of offering secret storage to your users in the future.

Hope that helps you with your decision a bit!

[taylorsilva]

A bit outdated but still quite true, this may be worth a read: https://concourse-ci.org/internals.html

Here's a talk I did doing a deep dive into the internals of Concourse as well: https://youtu.be/H-4pvC7t2AI

[taylorsilva]

Was looking at Forgejo and remembered this thread.

So you can't lock down what tags a worker declares for itself. You can lock down what team(s) a worker can declare itself for. We used to do this for a shared Concourse instance, where each team would generate a key and share the public key with us. We'd then assign the public key to their team here: https://github.com/concourse/hush-house/blob/afad5222eb4dde2324ea5be35689281d5c88b5a3/deployments/with-creds/hush-house/values.yaml#L8-L27
This uses the CONCOURSE_TSA_TEAM_AUTHORIZED_KEYS/--tsa-team-authorized-keys=NAME:PATH env var/flag. This ensures that only workers with the right private key can join under a certain team name. Only workloads from the specified teams pipeline can be sent to the worker.

[alanzanattadev]

Hi,

I think the question mentioned in the title is still legit though. There is no way to ensure the tags are signed and trusted. Let's say we want to have a set of workers in an untrusted environments, used for testing, and a set of workers running in a more trusted environment for automated deployments purposes. We are not able to ensure the tags are really what they are supposed to be, and a compromised testing worker could spoof the "deployment" tag to gain access to more sensitive workload, which includes secrets.

Any plan to add "signed tags" in Concourse ? Either through authorized_keys (adding some metadata) or including it inside SSH keys