I'm struggling to figure out how to properly order jobs, and keep one from running until the others are complete.

[PLColuccio]

We have this issue that's been plaguing us for a while. We have these instance pipelines, which are generated by a metadata file in S3, where the destroy step cleans up the metadata causing the pipeline to become archived. Our pipeline starts when triggering by an S3 file to build images based on our codebase. Once the build is complete, the deploy will deploy a helm chart with those images, and finally the destroy step will kick off based on another S3 file. The flow is pretty simple.

jobs:
  - name: build
    plan:
    - get: pr-ref-s3-file
      trigger: true

    - do some stuff

  - name: deploy
    plan:
    - get: pr-ref
      trigger: true
      passed: [build]

    - do some stuff

  - name: destroy
    plan:
    - get: destroy-s3-file
      trigger: true

    - do some stuff

This doesn't work well since developers tend to push multiple commits in succession, then close the PR fairly quickly. We run into resources that aren't being destroyed, due to the deploy running after the destroy is finished.

I have attempted to mitigate it with the concourse-pool-resource, but it doesn't seem to have the features we need. Since these are instance pipelines, and are short lived, we create a claimed lock in the build step, then unclaim the lock in the deploy step. This worked well when there was only one build required for a pull request. The destroy pipeline would wait for an unclaimed lock. The problem arises when there are multiple builds, you cannot create a claimed lock when an unclaimed lock exists (which makes sense). So the pipeline was modified to try: remove: lock before creating the claimed lock.

So the pipeline basically looks like this now:

jobs:
  - name: build
    plan:
    - get: pr-ref-s3-file
      trigger: true

    - task: lock-data
      config:
        platform: linux
        image_resource:
          type: registry-image
          source:
            repository: busybox
        outputs:
          - name: lock-data
        run:
          path: sh
          args:
            - -cx
            - |
              echo ((ctx.pr)) >lock-data/name
              echo ((.:ref)) >lock-data/metadata

    - try:
        put: locks
        params:
          remove: lock-data

    - put: locks
      params:
        add_claimed: lock-data

    - do some stuff

  - name: deploy
    plan:
    - get: pr-ref
      trigger: true
      passed: [build]

    - do some stuff

    ensure:
      try:
        put: locks
        no_get: true
        params: { release: locks }

  - name: destroy
    plan:
    - get: destroy-s3-file
      trigger: true

    - put: locks
      params:
        claim: ((ctx.pr))
  
    - do some stuff

    on_success:
      try:
        put: locks
        params: { remove: locks }

More frequent successful cleanups in this case, since the destroy cannot run while the deploy is running, but we still have the occasional issue where build is running while deploy is running, and the deploy will unclaim the wrong builds lock (since we are basically atomically creating it on each build run).

I thought about using a serial group on the build and deploy steps, but I fear the destroy will run while there is a build job queued up waiting for the deploy to finish, putting me in the same scenario.

I can't figure out either:
a) Make the jobs wait for other jobs to completely finish
b) Cancel in-progress jobs from inside a pipeline

Has anyone faced a similar issue like this, or maybe has a decent solution / work around?

[PLColuccio]

I've decided to write a resource that will pause a pipeline, then query Concourse to check for running jobs in the pipeline, wait for them, and continue when they are complete.