[9.1] (backport #18181) Remove redundant testing and circular dependency from docker acceptance testing

[mergify]

Release notes

[rn:skip]

What does this PR do?

Previously we would build an image (which would not actually be used), build dockerfiles, modify dockerfiles to curl from
https://snapshots.elastic.co/downloads/logstash' then build the image used for testing based on the modified dockerfile. This resulted in testing the last published image to snapshots. This presents two problems 1. The test is running against the last published image (not the tip of the branch being tested) 2. this carries a dependency on both a DRA and unified stack release having been run. Therefor acceptance tests will fail in between the time we bump logstash version and a successful run of unified release.

For docker acceptance tests three steps were performed:

1. Build container images (based on local artifacts)
2. Build "public" dockerfiles
3. Build container based on (a modified) file from step 2.

The ONLY difference between the dockerfile that ultimately is used to define an image between 1 and 2 is WHERE the logstash source is downloaded from. In acceptance testing we WANT to use the source at the current checkout of logstash (not a remote). Using remote causes a dependency issue when changes are not published. Publishing is tied to unified release and gated on tests so naturally this is a bad fit for that dependency.

This commit removes the redundancy by ONLY generating images for testing (step 1 from above). This also firms up our use of LOCAL_ARTIFACTS. Namely, the ONLY time we want that set to false is when we build a "public" dockerfile. We explicitly set that in the corresponding DRA script now. Similarly we explicitly set it to true when testing.

Why is it important/What is the impact to the user?

n/a

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

You can run the test locally with:

# Set arch appropriately (i've got a mac with arm)
ARCH="aarch64"
ci/docker_acceptance_tests.sh full

Related issues

• https://github.com/elastic/ingest-dev/issues/5763

---

This is an automatic backport of pull request #18181 done by [Mergify](https://mergify.com).

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

Cherry-pick of a994c7c has failed:

On branch mergify/bp/9.1/pr-18181
Your branch is up to date with 'origin/9.1'.

You are currently cherry-picking commit a994c7cb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/scripts/dra/build_docker.sh
	modified:   ci/docker_acceptance_tests.sh

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   docker/Makefile
	both modified:   rakelib/artifacts.rake

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 2900182

Failed CI Steps

• :rspec: Ruby unit tests
• :rspec: Ruby unit tests
• :rspec: Ruby unit tests
• :rspec: Ruby unit tests
• 🥼 Integration Tests / part 1-of-6
• 🥼 Integration Tests / part 1-of-6
• 🥼 Integration Tests / part 1-of-6
• 🥼 Integration Tests / part 1-of-6
• 🥼 Integration Tests / part 2-of-6
• 🥼 Integration Tests / part 2-of-6
• 🥼 Integration Tests / part 2-of-6
• 🥼 Integration Tests / part 2-of-6
• 🥼 Integration Tests / part 3-of-6
• 🥼 Integration Tests / part 3-of-6
• 🥼 Integration Tests / part 3-of-6
• 🥼 Integration Tests / part 3-of-6
• 🥼 Integration Tests / part 4-of-6
• 🥼 Integration Tests / part 4-of-6
• 🥼 Integration Tests / part 4-of-6
• 🥼 Integration Tests / part 4-of-6
• 🥼 Integration Tests / part 5-of-6
• 🥼 Integration Tests / part 5-of-6
• 🥼 Integration Tests / part 5-of-6
• 🥼 Integration Tests / part 5-of-6
• 🥼 Integration Tests / part 6-of-6
• 🥼 Integration Tests / part 6-of-6
• 🥼 Integration Tests / part 6-of-6
• 🥼 Integration Tests / part 6-of-6
• 🥼 IT Persistent Queues / part 1-of-6
• 🥼 IT Persistent Queues / part 1-of-6
• 🥼 IT Persistent Queues / part 1-of-6
• 🥼 IT Persistent Queues / part 1-of-6
• 🥼 IT Persistent Queues / part 2-of-6
• 🥼 IT Persistent Queues / part 2-of-6
• 🥼 IT Persistent Queues / part 2-of-6
• 🥼 IT Persistent Queues / part 2-of-6
• 🥼 IT Persistent Queues / part 3-of-6
• 🥼 IT Persistent Queues / part 3-of-6
• 🥼 IT Persistent Queues / part 3-of-6
• 🥼 IT Persistent Queues / part 3-of-6
• 🥼 IT Persistent Queues / part 4-of-6
• 🥼 IT Persistent Queues / part 4-of-6
• 🥼 IT Persistent Queues / part 4-of-6
• 🥼 IT Persistent Queues / part 4-of-6
• 🥼 IT Persistent Queues / part 5-of-6
• 🥼 IT Persistent Queues / part 5-of-6
• 🥼 IT Persistent Queues / part 5-of-6
• 🥼 IT Persistent Queues / part 5-of-6
• 🥼 IT Persistent Queues / part 6-of-6
• 🥼 IT Persistent Queues / part 6-of-6
• 🥼 IT Persistent Queues / part 6-of-6
• 🥼 IT Persistent Queues / part 6-of-6
• 🥼 x-pack unit tests
• 🥼 x-pack unit tests
• 🥼 x-pack unit tests
• 🥼 x-pack unit tests
• 🥼 x-pack integration
• 🥼 x-pack integration
• 🥼 x-pack integration
• 🥼 x-pack integration

History

cc @donoghuc

[mergify]

This pull request has not been merged yet. Could you please review and merge it @donoghuc? 🙏