across step var interpolation doesn't work with resource names

[spiegela]

I was trying out the new across step for a pipeline, and found that references to resource names don't seem to support the variable interpolation.

In this example, the task parameters work as expected, but the input_mapping block does not.

  - name: kube-templates
    plan:
      - in_parallel:
          - get: pipelines
          - get: kube-template-v1.18.8
            trigger: true
          - get: kube-template-v1.19.1
            trigger: true
      - across:
          - var: kube_version
            values:
              - v1.18.8
              - v1.19.1
        do:
          - task: import-ova-spec
            file: pipelines/tasks/govc-cmd/task.yml
            input_mapping:
              files: kube-template-((.:kube_version))
            output_mapping:
              output: ova-spec
            params:
              <<: *govc_env
              COMMAND: import.spec
              ARGS: files/kube-((.:kube_version)).ova

I also found that get and put resource names are not interpolated with these variables, but that was easier to work around. Is there a way around this limitation that I'm not thinking of?

This may be expected behavior for this release of the feature. In that case, it would be really helpful to have resource names supported, as across works its way to GA.

[vito]

@aoldershaw and I have been discussing this - the advent of build-local vars is revealing many places where we don't support interpolation. In the past it was only used for creds, and those only go in a few places, so those were all that we needed to support.

Some things can't be interpolated at a fundamental level - get step resource/version for example cannot be interpolated because we need to know the set of inputs and versions before scheduling a build, i.e. it cannot be determined at runtime.

[aoldershaw]

Yeah, I ran into this exact problem when trying to utilize the across step in one of our pipelines. I've been putting some thought into how to support structured interpolation for most things in the build plan without giving up on static config validation and without adding too much complexity, but it's proving to be a bit challenging. I'll keep poking away at it!

[schmurfy]

I tried to use across on a set_pipeline step and it looks to be the same which is a shame as it prevents me to create pipelines for every (pull-request, service) pair for our monorepo.
I know there is work on this on the way but is there a way to do that currently ?

[aoldershaw]

There isn't any good way to do that now - this workflow is a little further down the line from just improving var interpolation, though.

Currently, the across step is pretty limited in what it can do - it can only work on a static list of values. What you need is the dynamic across step with var_source prototypes (see https://github.com/vito/rfcs/blob/spatial-resources/029-across-step/proposal.md#dynamic-var-values-from-a-var-source), neither of which is implemented yet. More accurately, you'd want something like https://github.com/vito/rfcs/blob/spatial-resources/029-across-step/proposal.md#running-across-a-matrix-of-var-values, where you'd set a pipeline across a dynamic list of PRs and across a static list of services.

All this to say, your use case is one of the primary motivating factors for the across step, but it unfortunately won't be able to handle your needs for a little while most likely.

[schmurfy]

I was afraid of that answer but thanks for the clarifications :)

[schmurfy]

I have an almost working workaround using the deprecated concourse-pipeline-resource, when using a put task with across I can spawn a pipeline and set its name.
It means that the old pipelines won't be archived but I can live with that, I just need to script it myself and this solution should work for us until all the features are ready :)

concourse is awesome :)

[schmurfy]

nevermind, I missed than across with dynamic values is not available yet...
This is very frustrating xD