Substitute vars then lookup secrets

[esauser]

What challenge are you facing?

We frequently duplicate a decent amount of code in our pipeline yaml files due to many environments. For example, if we have dev, qa, and prod we might have

- task: flyway-info-dev
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  output_mapping:
    flyway-info: flyway-info-dev
  params:
    FLYWAY_USER: ((dev-database.user))
    FLYWAY_PASSWORD: ((dev-database.password))

- task: flyway-info-qa
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  output_mapping:
    flyway-info: flyway-info-qa
  params:
    FLYWAY_USER: ((qa-database.user))
    FLYWAY_PASSWORD: ((qa-database.password))

- task: flyway-info-prod
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  output_mapping:
    flyway-info: flyway-info-prod
  params:
    FLYWAY_USER: ((prod-database.user))
    FLYWAY_PASSWORD: ((prod-database.password))

What would make this better?

With the relatively new vars on tasks we were hoping we'd be able to pass dev, qa, and prod as an ENV var. We can do this, but then the secret isn't evaluated. It would be better if it was. If it was, our pipeline would look like this

- task: flyway-info-dev
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  vars:
    ENV: dev

- task: flyway-info-qa
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  vars:
    ENV: qa

- task: flyway-info-prod
  file: ci/tasks/flyway-info.yml
  image: flyway-info
  vars:
    ENV: prod

and our task would look like this

platform: linux
inputs:
  - name: pull-request
  - name: task-docker-images
outputs:
  - name: flyway-info-((ENV))
run:
  dir: pull-request
  path: ../task-docker-images/flyway-info/flyway-info.sh
  user: root
params:
  FLYWAY_USER: ((((ENV))-database.user))
  FLYWAY_PASSWORD: ((((ENV))-database.password))

Are you interested in implementing this yourself?

No

[evanchaoli]

This looks like a cool feature. To implement this feature, I think change should be not big. If Concourse team confirm by label "help-wanted", I can help when I have some bandwidth.

[stale[bot]]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[esauser]

@vito what do you think about this?

[willejs]

I am after something similar too...

[vito]

Hrm, I don't know, I can see your point but I'm just not really sure if it's worth the complexity and potential confusion just to DRY things up a little bit more. 🤔

I'm really hesitant to add features that are centered completely around templating. At the end of the day if your templating system is sufficiently powerful you can probably accomplish anything you want, but now you're maintaining a lot of meta-logic alongside your actual logic, and for new folks coming onto a project that can be very confusing.

I would prefer to think more about the workflow you're trying to model and see where Concourse's abstractions are at their limit - especially because we're in the midst of putting together a long-term roadmap which might already have items to address your underlying needs. With more context we might be able to find a different way to go about this. Here's a question: what is the next thing that's done with each task? i.e. how are the outputs used?

In general this strikes me as something that could be modeled as a set of independent pipelines, dev/qa/prod. If these were separate pipelines instead, the var names wouldn't have to be explicitly "namespaced" like that. But you may have a valid reason for doing it all in one pipeline. 🙂

[esauser]

@vito the example above is from our database pipelines where we test the migrations in dev, qa, and prod during pr-verify. The only thing different about those tasks are the connection strings and credentials, which come from vault.

Another example where this would be nice is our pipelines for terraform and apps hosted in the cloud. There are separate jobs for each environment (lab, dev, qa, and prod) and each of them have apply tasks. The only difference in said tasks is the environment and the credentials.

The connection strings and environments can be handled with params since they aren't secrets, but the credentials come from vault, so they cannot be templated as-is.

This is a very common pattern for us. As you noted, it's purely for templating, simplifying the pipelines into more reusable sections. In our experience, more templating makes things easier to understand, maintain, etc. For example, we already use centralized tasks, yaml pointers, etc. as much as the system allows. This feature would allow us to take it to a whole new level.

[jmccann]

Just wanted to say I've had the same usecase for pretty much the same reasons.

Additionally, another usecase I've ran into is to have a template for implementing a pipeline in an non-prod environment to test the pipeline itself before "promoting" to to a prod environment by setting it appropriately with update secret/paths.

Would be cool to have dev/test/release strategy for pipelines like we typically see for applications. Where you deploy same application code but provide different config/secrets for applicaiton in NPE/prod it would be neat to be able to do the same with pipelines. 😄

[stale[bot]]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[esauser]

Stop it stale.