[8.17](backport #8736) ci: pin elastic-agent version

[mergify]

What does this PR do?

This PR temporarily pins AGENT_VERSION and BEAT_VERSION (probably for historical reasons we need both, ty for the BEAT_VERSION hint @pchila ) to 9.1.0-SNAPSHOT in CI pipelines. This is an effort to mitigate breakages caused by the recent version bump to 9.2.0 which point to unavailable snapshot artifacts for 9.2.0-SNAPSHOT in the unified release.

In particular, this patch:

• Pins AGENT_VERSION, AGENT_PACKAGE_VERSION, and BEAT_VERSION across all affected pipelines.
• Skips upgrade_broken_package_test.go since it fails due to the version mismatch until the pinning is removed.
• Wraps the parsing logic of version.go in Bash and PowerShell scripts to only run when the AGENT_VERSION is not explicitly set.
• Updates cloud:push and cloud:load mage targets to respect the pinned AGENT_VERSION, replacing previously hardcoded fallback to version.go.

Why is it important?

Pinning the version ensures a consistent and known state for the CI runs until the 9.2.0 snapshot artifacts are fully available via DRA and unified stack release.

In addition, there's another issue introduced by #8352, which pins the ESS stack version used in CI to the latest stable version. This means that even if the 9.2.0 DRA snapshot was published, the CI would still be failing due to version incompatibility between the provisioned ESS stack and the elastic-agent binary under test, until the pinned ESS version point to a 9.2.0 stack.

Also, the cloud:push and cloud:load mage targets - introduced by #8035 - did not originally respect version overrides and required additional code changes to support environment-based overrides, similar to the rest of the codebase.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None expected. These changes only affect CI behavior and not end-user installations or interactions.

How to test this PR locally

This changes regard only CI pipelines

Related issues

N/A

---

This is an automatic backport of pull request #8736 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 5c5b174 has failed:

On branch mergify/bp/8.17/pr-8736
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit 5c5b174ae.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/integration.pipeline.yml
	modified:   .buildkite/scripts/buildkite-integration-tests.ps1
	modified:   .buildkite/scripts/common.sh
	modified:   testing/integration/upgrade_broken_package_test.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .buildkite/bk.integration-fips.pipeline.yml
	both modified:   .buildkite/bk.integration.pipeline.yml
	both modified:   .buildkite/scripts/buildkite-integration-tests.sh
	both modified:   magefile.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: feced0c

Failed CI Steps

• Unit tests - MacOS 13

History

• 💔 Build #23300 failed 3ab8d1f

cc @pkoutsovasilis