Remove redundant testing and circular dependency from docker acceptance testing (#18181)

* Use locally build artifact to build container from public dockerfile

Previously we would build an image (which would not actually be used), build
dockerfiles, modify dockerfiles to curl from
`https://snapshots.elastic.co/downloads/logstash'` then build the image used for
testing based on the modified dockerfile. This resulted in testing the last
published image to `snapshots`. This presents two problems 1. The test is
running against the last published image (not the tip of the branch being
tested) 2. this carries a dependency on both a DRA and unified stack release
having been run. Therefor acceptance tests will fail in between the time we bump
logstash version and a successful run of unified release.

This commit modifies the dockerfile to use the artifact prepared in the first
step instead of curling the last published one. This solves both issues as the
tests run against the code from the tip fo the branch being tested and there is
no dependency on an artifact existing as a result of a unified release pipeline.

* Remove redundant docker steps from workflows

Previously for docker acceptance tests three steps were performed: 1. Build
container images (based on local artifacts) 2. Build "public" dockerfiles 3.
Build container based on (a modified) file from step 2. The ONLY difference
between the dockerfile that ultimately is used to define an image between 1 and
2 is WHERE the logstash source is downloaded from. In acceptance testing we WANT
to use the source at the current checkout of logstash (not a remote). Using
remote causes a dependency issue when changes are not published. Publishing is
tied to unified release and gated on tests so naturally this is a bad fit for
that dependency.

This commit removes the redundancy by ONLY generating images for testing (step 1
from above). This also firms up our use of LOCAL_ARTIFACTS. Namely, the ONLY
time we want that set to `false` is when we build a "public" dockerfile. We
explicitly set that in the corresponding DRA script now. Similarly we explicitly
set it to `true` when testing.

* Remove unused function and argument

This commit removes the unused function for building from dockerfiles.
It also removes an unused argument for the make task for build_docker.

(cherry picked from commit a994c7c)

# Conflicts:
#	docker/Makefile
#	rakelib/artifacts.rake

Author: donoghuc

.buildkite/scripts/dra/build_docker.sh

@@ -24,6 +24,9 @@ esac
 rake artifact:docker || error "artifact:docker build failed."
 rake artifact:docker_oss || error "artifact:docker_oss build failed."
 rake artifact:docker_wolfi || error "artifact:docker_wolfi build failed."
+
+# Generating public dockerfiles is the primary use case for NOT using local artifacts
+export LOCAL_ARTIFACTS=false
 rake artifact:dockerfiles || error "artifact:dockerfiles build failed."
 
 if [[ "$ARCH" != "aarch64" ]]; then

ci/docker_acceptance_tests.sh

@@ -8,6 +8,9 @@ set -x
 export JRUBY_OPTS="-J-Xmx1g"
 export GRADLE_OPTS="-Xmx4g -Dorg.gradle.console=plain -Dorg.gradle.daemon=false -Dorg.gradle.logging.level=info -Dfile.encoding=UTF-8"
 
+# Use local artifacts for acceptance test Docker builds
+export LOCAL_ARTIFACTS=true
+
 if [ -n "$BUILD_JAVA_HOME" ]; then
   GRADLE_OPTS="$GRADLE_OPTS -Dorg.gradle.java.home=$BUILD_JAVA_HOME"
 fi

docker/Makefile

@@ -128,7 +128,7 @@ public-dockerfiles_full: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 		version_tag="${VERSION_TAG}" \
 	  release="${RELEASE}" \
 		image_flavor="full" \
-		local_artifacts="false" \
+		local_artifacts="$(or $(LOCAL_ARTIFACTS),false)" \
 		templates/Dockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-full" && \
 	cd $(ARTIFACTS_DIR)/docker && \
 	cp $(ARTIFACTS_DIR)/Dockerfile-full Dockerfile && \
@@ -142,12 +142,13 @@ public-dockerfiles_oss: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 		version_tag="${VERSION_TAG}" \
 	  release="${RELEASE}" \
 		image_flavor="oss" \
-		local_artifacts="false" \
+		local_artifacts="$(or $(LOCAL_ARTIFACTS),false)" \
 		templates/Dockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-oss" && \
 	cd $(ARTIFACTS_DIR)/docker && \
 	cp $(ARTIFACTS_DIR)/Dockerfile-oss Dockerfile && \
 	tar -zcf ../logstash-oss-$(VERSION_TAG)-docker-build-context.tar.gz Dockerfile bin config env2yaml pipeline
 
+<<<<<<< HEAD
 public-dockerfiles_ubi8: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 	../vendor/jruby/bin/jruby -S erb -T "-"\
 		created_date="${BUILD_DATE}" \
@@ -162,6 +163,8 @@ public-dockerfiles_ubi8: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 	cp $(ARTIFACTS_DIR)/Dockerfile-ubi8 Dockerfile && \
 	tar -zcf ../logstash-ubi8-$(VERSION_TAG)-docker-build-context.tar.gz Dockerfile bin config env2yaml pipeline
 
+=======
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
 public-dockerfiles_wolfi: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 	../vendor/jruby/bin/jruby -S erb -T "-"\
 		created_date="${BUILD_DATE}" \
@@ -170,13 +173,31 @@ public-dockerfiles_wolfi: templates/Dockerfile.erb docker_paths $(COPY_FILES)
 		version_tag="${VERSION_TAG}" \
 	  release="${RELEASE}" \
 		image_flavor="wolfi" \
-		local_artifacts="false" \
+		local_artifacts="$(or $(LOCAL_ARTIFACTS),false)" \
 		templates/Dockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-wolfi" && \
 	cd $(ARTIFACTS_DIR)/docker && \
 	cp $(ARTIFACTS_DIR)/Dockerfile-wolfi Dockerfile && \
 	tar -zcf ../logstash-wolfi-$(VERSION_TAG)-docker-build-context.tar.gz Dockerfile bin config env2yaml pipeline
 
+<<<<<<< HEAD
 public-dockerfiles_ironbank: templates/hardening_manifest.yaml.erb templates/Dockerfile.erb ironbank_docker_paths $(COPY_IRONBANK_FILES)
+=======
+public-dockerfiles_observability-sre: templates/Dockerfile.erb docker_paths $(COPY_FILES)
+	../vendor/jruby/bin/jruby -S erb -T "-"\
+		created_date="${BUILD_DATE}" \
+		elastic_version="${ELASTIC_VERSION}" \
+		arch="${ARCHITECTURE}" \
+		version_tag="${VERSION_TAG}" \
+		release="${RELEASE}" \
+		image_flavor="observability-sre" \
+		local_artifacts="$(or $(LOCAL_ARTIFACTS),false)" \
+		templates/Dockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-observability-sre" && \
+	cd $(ARTIFACTS_DIR)/docker && \
+	cp $(ARTIFACTS_DIR)/Dockerfile-observability-sre Dockerfile && \
+	tar -zcf ../logstash-observability-sre-$(VERSION_TAG)-docker-build-context.tar.gz Dockerfile bin config env2yaml pipeline
+
+public-dockerfiles_ironbank: templates/hardening_manifest.yaml.erb templates/IronbankDockerfile.erb ironbank_docker_paths $(COPY_IRONBANK_FILES)
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
 	../vendor/jruby/bin/jruby -S erb -T "-"\
 	  elastic_version="${ELASTIC_VERSION}" \
 	  templates/hardening_manifest.yaml.erb > $(ARTIFACTS_DIR)/ironbank/hardening_manifest.yaml && \
@@ -187,8 +208,13 @@ public-dockerfiles_ironbank: templates/hardening_manifest.yaml.erb templates/Doc
 		version_tag="${VERSION_TAG}" \
 	  release="${RELEASE}" \
 		image_flavor="ironbank" \
+<<<<<<< HEAD
 		local_artifacts="false" \
 		templates/Dockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-ironbank" && \
+=======
+		local_artifacts="$(or $(LOCAL_ARTIFACTS),false)" \
+		templates/IronbankDockerfile.erb > "${ARTIFACTS_DIR}/Dockerfile-ironbank" && \
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
 	cd $(ARTIFACTS_DIR)/ironbank && \
 	cp $(ARTIFACTS_DIR)/Dockerfile-ironbank Dockerfile && \
 	tar -zcf ../logstash-ironbank-$(VERSION_TAG)-docker-build-context.tar.gz scripts Dockerfile hardening_manifest.yaml LICENSE README.md

rakelib/artifacts.rake

@@ -171,7 +171,11 @@ namespace "artifact" do
 
   desc "Generate rpm, deb, tar and zip artifacts"
   task "all" => ["prepare", "build"]
+<<<<<<< HEAD
   task "docker_only" => ["prepare", "build_docker_full", "build_docker_oss", "build_docker_ubi8", "build_docker_wolfi"]
+=======
+  task "docker_only" => ["prepare", "docker", "docker_oss", "docker_wolfi", "docker_observabilitySRE"]
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
 
   desc "Build all (jdk bundled and not) tar.gz and zip of default logstash plugins with all dependencies"
   task "archives" => ["prepare", "generate_build_metadata"] do
@@ -380,18 +384,30 @@ namespace "artifact" do
     build_dockerfile('oss')
   end
 
+<<<<<<< HEAD
+=======
+  desc "Generate Dockerfile for observability-sre images"
+  task "dockerfile_observabilitySRE" => ["prepare-observabilitySRE", "generate_build_metadata"] do
+    puts("[dockerfiles] Building observability-sre Dockerfile")
+    build_dockerfile('observability-sre')
+  end
+
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
   desc "Generate Dockerfile for full images"
   task "dockerfile_full" => ["prepare", "generate_build_metadata"] do
     puts("[dockerfiles] Building full Dockerfiles")
     build_dockerfile('full')
   end
 
+<<<<<<< HEAD
   desc "Generate Dockerfile for UBI8 images"
   task "dockerfile_ubi8" => ["prepare", "generate_build_metadata"] do
     puts("[dockerfiles] Building ubi8 Dockerfiles")
     build_dockerfile('ubi8')
   end
 
+=======
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
   desc "Generate Dockerfile for wolfi images"
   task "dockerfile_wolfi" => ["prepare", "generate_build_metadata"] do
     puts("[dockerfiles] Building wolfi Dockerfiles")
@@ -424,6 +440,7 @@ namespace "artifact" do
     Rake::Task["artifact:archives_oss"].invoke
   end
 
+<<<<<<< HEAD
   task "build_docker_full" => [:generate_build_metadata] do
     Rake::Task["artifact:docker"].invoke
     Rake::Task["artifact:dockerfile_full"].invoke
@@ -444,6 +461,8 @@ namespace "artifact" do
     Rake::Task["artifact:dockerfile_wolfi"].invoke
   end
 
+=======
+>>>>>>> a994c7cb (Remove redundant testing and circular dependency from docker acceptance testing (#18181))
   task "generate_build_metadata" do
     require 'time'
     require 'tempfile'