add DRA version bump pipeline

[ninalee12]

Release notes

[rn:skip]

What does this PR do?

Hi Team,

This PR is the first phase of the version-bump automation rollout as mentioned in #mission-control. It introduces a generalized Buildkite pipeline for service teams in the DRA process. The baseline pipeline includes a block step that waits for each team to unblock before polling for DRA artifacts using a buildkite plugin json-watcher-buildkite-plugin for the polling.

Why is it important/What is the impact to the user?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• https://github.com/elastic/observability-robots/issues/3404
• If you haven't already, please provide the Slack channel name for notifications via a comment here or on the issue listed above

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify]

This pull request does not have a backport label. Could you fix it @ninalee12? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[v1v]

As I mentioned in some other PRs, who is responsible for keeping .buildkite/version_bump_pipeline.yml up to date? Wonder if adding a CODEOWNERS could help here,

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: eab5d72

History

• 💚 Build #4393 succeeded 3c1d94d
• 💛 Build #4350 was flaky 69586c2
• 💚 Build #4348 succeeded 6afb2d2
• 💛 Build #4337 was flaky d7fee86

[ninalee12]

Hey Victor, could you please merge this PR as well, thank you!

For CODEOWNERS, as mentioned here. The pipeline in this PR will be owned by the service team

[v1v]

Hey Victor, could you please merge this PR as well, thank you!

I'm waiting for the rest of the Logstash team to review it.

[donoghuc]

I'm having trouble understanding what this is for? Why do we want to poll for artifacts? What consumes this?

[ninalee12]

Hello, let me try to answer your questions the best I can.

Why do we want to poll for artifacts?

We (Release-eng) poll for artifacts as the last step to ensure that the version for the artifacts have been bumped.

What consumes this?

We'll be using a centralized pipeline (owned by release-eng) which will be used alongside the pipeline in this PR. How it'll work is that the centralized pipeline will trigger the service team's version bump pipeline using a dependency graph.

[donoghuc]

Why is "prompt" used here? Will this just consistently be in a "waiting" state? How is that different than just running the pipeline when needed?

[ninalee12]

The entire block step is to be replaced by each team. This block step is more of a placeholder.

[donoghuc]

I would like to understand more the exact use case here for logstash...

Today, the process for version bumps (IE the day a release goes out):

1. Run the GHA to bump the logstash version https://github.com/elastic/logstash/actions/workflows/bump-logstash.yml
2. Wait for 1 to go through CI (exhaustive testing etc)
3. Kick off DRA staging/snapshot pipelines

Is the intent of this new proposed pipeline to orchestrate all three of those steps? Or is there something else I'm missing?

[ninalee12]

I would like to understand more the exact use case here for logstash...

Today, the process for version bumps (IE the day a release goes out):

1. Run the GHA to bump the logstash version https://github.com/elastic/logstash/actions/workflows/bump-logstash.yml
2. Wait for 1 to go through CI (exhaustive testing etc)
3. Kick off DRA staging/snapshot pipelines

Is the intent of this new proposed pipeline to orchestrate all three of those steps? Or is there something else I'm missing?

No, the new proposed pipeline will be used by the centralized pipeline. The version bump logic itself will still be handled by each service team.

There's more details in the PSI regarding the pipeline in section 3.Implement team version bump automation

Hope this helps!