Forwardport observability-sre internal distro support from 8.19 to main (#17785)

* forward-port observabilitySRE image creation into `main`

This is the CLEAN subset of a cherry-pick of the merge-commit from the
observabilitySRE feature branch into 8.x in PR #17541 (0b1d299),
OMITTING changes to `docker/*` and `rakelib/artifacts.rake` that would
conflict due to substantial refactorings on `main`.

* forward-port observabilitySRE image creation into `main` (re-implament)

This is a forward-port of _functionality_ from the observabilitySRE feature
branch into 8.x in PR #17541 (0b1d299),
wholly re-implementing the changes in `docker/*` and `rakelib/artifacts.rake`
from the 8.x-style docker structure to the refactored structure present
on `main`.

* Fix pull request pipeline definition for buildkite (#17552)

When the fedramp high feature branch was merged into 8.x the PR pipeline
accidentally duplicated the top level `steps` key. This was a mistake and is
causing issues generating exhaustive test pipeline definition. This commit fixes
the bug by ensuring there is a single `steps` key that defines all the steps in
the pipeline.

* Ensure observabilitySRE image is pushed on DRA staging (#17569)

The `artifactDockerObservabilitySRE` gradle task *always* produces a tag with a
`SNAPSHOT` postfix. In the staging pipeline we use the shared
`qualified-version` script for determining the LS version. That script correctly
handles conditionally adding a `SNAPSHOT` postfix which is important for the
tagging scheme for pushing to our container registry. Given the intermediate tag
produced by the gradle task is never pushed anywhere we can update the build
script to ensure the "local" artifact is always referenced with the `SNAPSHOT`
postfix.

* Use dedicated elasticsearch image for observabilitySRE smoke testing (#17627)

* Use dedicated elasticsearch image for observabilitySRE smoke testing

The ES team has started publishing a purpose built image for the fedramp high
project. Update our smoke test stack to use this container.

* Override default entrypoint into elasticsearch container

The new image does not provide the stub `/app/elasticsearch.sh` file
https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/README.md?plain=1#L16-L19
previously available. This commit overrides the entrypoint to avoid needing that
file. See: https://github.com/elastic/elasticsearch/blob/1a1763c591c4c32bf66f0df3bce2040e8f19a1a2/distribution/docker/src/docker/Dockerfile.ess#L38C5-L40C37

* Remove entrypoint workaround due to fix landing upstream

* Restore code review changes (#17539)

* Comment to clarify why FIPS flag is not needed for smoke tests

* Use full versions of docker commands for readability

* Simplify grock pattern match

The grok pattern is unanchored-by-default, we don't need the leading and trailing
wildcards.

* Add a step to exhaustive tests for observabilitySRE accetpance testing (#17623)

* Add a step to exhaustive tests for observabilitySRE accetpance testing

This commit shows the proposed pattern for adding acceptance testing for the
observability SRE image. This will run when exhaustive tests run. A new gradle
task will hook in to rspec similar to how it is done for the smoke tests. The
main difference is that instead of building a container, the latest is pulled
from the container registry and run on a fips configured host VM.

* WIP: Idea for how to handle multipe container configs for acceptance tests

This commit shows the rough structure for how I am planning on handling docker
compose networks for acceptance tests. The main idea is to use interpolation in
the docker compose file to point to different configuration files for
filebeat/logstash/elasticsearch. This is mainly due to the nature of these tests
showing behavior when the system is and is not configured properly for FIPS. The
breakdown in responsibility is:

1. Gradle handles cert generation (similar to smoke test, this avoids checking
in PKI)
2. Rspec handles stopping/starting docker compose and managing environment vars
for intperolation in docker compose manifests (different from smoke tests where
a single static docker compose is started in gradle)
3. Rspec handles deciding when containers are ready and querying state about
data flowing through the system
4. Gradle cleans up certs

THis is just a rough sketch, there are still bugs to be worked out but before i
get too far in to it I want to get the idea out there.

* Add tests describing behavior of LS -> ES with non-fips config

This commit adds a test to show that data will not flow from LS to ES
when weak non fips config is used.

* Use latest ES image

This will be handled separately in a separate PR, but taking this
commit for now on this branch.

* Remove custom entrypoint from new container

The latest ES images do not require this workaround.

* Take up code review suggestions

1. Remove rogue character from test file causing interpreter failure
2. Split out helpers for docker compose orchestration
3. Only send a single message instead of infinite through to ES

* Add full prefix name for new image

* Test filebeat -> LS -> ES using fips config

As described in elastic/ingest-dev#5471 this commit
adds a test for filebeat sending data through logstash to elasticsearch using
fips config.

* Test LS wont accept input from non fips configured filebeat

This test ensures logstash will not accept data from filebeat when using weak
tls configuration.

See elastic/ingest-dev#5472

* Fix a funny typo.

Crytpo is actually kind of a funny.

* Ensure we are using the purpose build ES image in testing

Similar to #17627

* Ensure JAVA_HOME is set etc

Use the same buildkite agent script for setting up a vm based runner as other pipes

---------

Co-authored-by: Cas Donoghue <[email protected]>

Author: yaauie

.buildkite/pull_request_pipeline.yml

@@ -19,6 +19,26 @@ steps:
       export GRADLE_OPTS="-Xmx2g -Dorg.gradle.daemon=false -Dorg.gradle.logging.level=info"
       ci/license_check.sh -m 4G
 
+  - label: ":passport_control: License check - Fedramp High Mode"
+    key: "license-check-fedramp-high"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      source .buildkite/scripts/common/container-agent.sh
+      export JRUBY_OPTS="-J-Xmx1g"
+      export GRADLE_OPTS="-Xmx2g -Dorg.gradle.daemon=false -Dorg.gradle.logging.level=info"
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e ORG_GRADLE_PROJECT_fedrampHighMode=true test-runner-image ci/license_check.sh -m 4G
+
   - label: ":rspec: Ruby unit tests"
     key: "ruby-unit-tests"
     agents:
@@ -39,6 +59,25 @@ steps:
     artifact_paths:
       - "coverage/coverage.json"
 
+  - label: ":rspec: Ruby unit tests - FIPS mode"
+    key: "ruby-unit-tests-fips"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run test-runner-image ./gradlew --info --stacktrace -PfedrampHighMode=true rubyTests
+    artifact_paths:
+      - "coverage/coverage.json"
+
   - label: ":java: Java unit tests"
     key: "java-unit-tests"
     agents:
@@ -60,6 +99,29 @@ steps:
       - "**/jacocoTestReport.xml"
       - "**/build/classes/**/*.*"
 
+  - label: ":java: Java unit tests - FIPS mode"
+    key: "java-unit-tests-fips"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    env:
+      ENABLE_SONARQUBE: true
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run test-runner-image ./gradlew --info --stacktrace -PfedrampHighMode=true javaTests
+    artifact_paths:
+      - "**/build/test-results/javaTests/TEST-*.xml"
+      - "**/jacocoTestReport.xml"
+      - "**/build/classes/**/*.*"
+
   - label: ":sonarqube: Continuous Code Inspection"
     if: | 
       build.pull_request.id != null || 
@@ -81,6 +143,84 @@ steps:
       manual:
         allowed: true
 
+  - label: "Observability SRE container smoke test"
+    key: "observability-sre-container-smoke-test"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+      source .buildkite/scripts/common/vm-agent.sh
+      QUALIFIED_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
+      # Build the image locally with the gradle task
+      ./gradlew --stacktrace artifactDockerObservabilitySRE -PfedrampHighMode=true
+      # Ensure it can at least start logstash
+      docker run docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
+        logstash -e 'input { generator { count => 3 } } output { stdout { codec => rubydebug } }'
+      # Run the smoke tests on the PR code
+      docker tag docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
+        pr-built-observability-sre-image
+      # observabilitySREsmokeTests orchestrates FIPS-mode docker images
+      # and validates assertions separately, so it does not need FIPS flag.
+      ./gradlew observabilitySREsmokeTests --stacktrace
+
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 1-of-3"
+    key: "integration-tests-fips-part-1-of-3"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split 0 3
+
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 2-of-3"
+    key: "integration-tests-fips-part-2-of-3"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split 1 3
+
+  - label: ":lab_coat: Integration Tests - FIPS mode / part 3-of-3"
+    key: "integration-tests-fips-part-3-of-3"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image ci/integration_tests.sh split  2 3
+
   - label: ":lab_coat: Integration Tests / part 1-of-3"
     key: "integration-tests-part-1-of-3"
     agents:
@@ -228,6 +368,40 @@ steps:
       source .buildkite/scripts/common/container-agent.sh
       x-pack/ci/integration_tests.sh
 
+  - label: ":lab_coat: x-pack unit tests - FIPS mode"
+    key: "x-pack-unit-tests-fips"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image x-pack/ci/unit_tests.sh
+
+  - label: ":lab_coat: x-pack integration - FIPS mode"
+    key: "integration-tests-x-pack-fips"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+
+      docker build -t test-runner-image -f x-pack/distributions/internal/observabilitySRE/docker/Dockerfile .
+      docker run -e FEDRAMP_HIGH_MODE=true test-runner-image x-pack/ci/integration_tests.sh
+
   - wait: ~
     continue_on_failure: true
 

.buildkite/scripts/common/qualified-version.sh

@@ -17,6 +17,11 @@ export QUALIFIED_VERSION="$(
   # e.g: 8.0.0-alpha1
   printf '%s' "${VERSION_QUALIFIER:+-${VERSION_QUALIFIER}}"
 
+  # Include git SHA if requested
+  if [[ -n "${INCLUDE_COMMIT_ID:+x}" ]]; then
+    printf '%s' "-$(git rev-parse --short HEAD)"
+  fi
+
   # add the SNAPSHOT tag unless WORKFLOW_TYPE=="staging" or RELEASE=="1"
   if [[ ! ( "${WORKFLOW_TYPE:-}" == "staging" || "${RELEASE:+$RELEASE}" == "1" ) ]]; then
     printf '%s' "-SNAPSHOT"

.buildkite/scripts/dra/build-and-push-observability-sre.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Script to build and publish ObservabilitySRE container
+# Currently this is built on a host with the target architecture.
+# This allows us to utilize the make file for building the container and
+# to ensure the best compatability with the host architecture.
+# A later step in CI will take care of pushing a tag that references the right
+# image using `docker manifest` commands.
+
+echo "Setting up environment"
+source .buildkite/scripts/common/vm-agent.sh
+source .buildkite/scripts/dra/docker-env-setup.sh
+
+echo "Building ObservabilitySRE container"
+./gradlew --stacktrace artifactDockerObservabilitySRE -PfedrampHighMode=true
+
+echo "Pushing ObservabilitySRE container to Docker repository"
+docker_login
+
+# Get qualified version without SHA (this is what the gradle task will produce)
+# Note that the gradle task always produces a version with -SNAPSHOT so if the
+# workflow type is staging we need to append -SNAPSHOT to the version.
+QUALIFIED_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
+if [[ "${WORKFLOW_TYPE:-}" == "staging" && "${QUALIFIED_VERSION}" != *-SNAPSHOT ]]; then
+  QUALIFIED_VERSION="${QUALIFIED_VERSION}-SNAPSHOT"
+fi
+
+# Set environment variable to include SHA and get version with SHA
+QUALIFIED_VERSION_WITH_SHA="$(INCLUDE_COMMIT_ID=1 .buildkite/scripts/common/qualified-version.sh)"
+
+REGISTRY_PATH=docker.elastic.co/logstash/logstash-observability-sre
+
+# Current architecture
+ARCH="${ARCH:-x86_64}"  # Default to x86_64 if ARCH is not set
+echo "Architecture: ${ARCH}"
+
+# Create the full tag with SHA and architecture
+FULL_TAG="${QUALIFIED_VERSION_WITH_SHA}-${ARCH}"
+echo "Tagging and pushing: ${REGISTRY_PATH}:${QUALIFIED_VERSION} as ${REGISTRY_PATH}:${FULL_TAG}"
+docker tag ${REGISTRY_PATH}:${QUALIFIED_VERSION} ${REGISTRY_PATH}:${FULL_TAG}
+docker push ${REGISTRY_PATH}:${FULL_TAG}
+
+# Teardown Docker environment
+source .buildkite/scripts/dra/docker-env-teardown.sh

.buildkite/scripts/dra/generatesteps.py

@@ -79,6 +79,61 @@ def package_aarch64_docker_step(branch, workflow_type):
 
     return step
 
+def ship_observability_sre_image_steps(branch, workflow_type):
+    step = f'''
+- label: ":package: Build & Ship aarch64 ObservabilitySRE container / {branch}-{workflow_type.upper()}"
+  key: "logstash_build_and_ship_observability_sre_aarch64"
+  soft_fail: true
+  depends_on: logstash_publish_dra
+  agents:
+    provider: aws
+    imagePrefix: platform-ingest-logstash-ubuntu-2204-aarch64
+    instanceType: "m6g.4xlarge"
+    diskSizeGb: 200
+  artifact_paths:
+    - "**/*.hprof"
+  command: |
+    export WORKFLOW_TYPE="{workflow_type}"
+    export PATH="/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH"
+    export ARCH="aarch64"
+    eval "$(rbenv init -)"
+    .buildkite/scripts/dra/build-and-push-observability-sre.sh
+- label: ":package: Build & Ship x86_64 ObservabilitySRE container / {branch}-{workflow_type.upper()}"
+  key: "logstash_build_and_ship_observability_sre_x86_64"
+  soft_fail: true
+  depends_on: logstash_publish_dra
+  agents:
+    provider: gcp
+    imageProject: elastic-images-prod
+    image: family/platform-ingest-logstash-ubuntu-2204
+    machineType: "n2-standard-16"
+    diskSizeGb: 200
+  artifact_paths:
+    - "**/*.hprof"
+  command: |
+    export WORKFLOW_TYPE="{workflow_type}"
+    export PATH="/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH"
+    export ARCH="x86_64"
+    eval "$(rbenv init -)"
+    .buildkite/scripts/dra/build-and-push-observability-sre.sh
+- label: ":docker: Create & Push ObservabilitySRE Multi-Arch Manifest / {branch}-{workflow_type.upper()}"
+  key: "logstash_create_observability_sre_manifest"
+  depends_on:
+    - "logstash_build_and_ship_observability_sre_aarch64"
+    - "logstash_build_and_ship_observability_sre_x86_64"
+  agents:
+    provider: gcp
+    imageProject: elastic-images-prod
+    image: family/platform-ingest-logstash-ubuntu-2204
+    machineType: "n2-standard-8"
+  command: |
+    export WORKFLOW_TYPE="{workflow_type}"
+    export PATH="/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH"
+    eval "$(rbenv init -)"
+    .buildkite/scripts/dra/multi-architecture-observability-sre.sh
+'''
+    return step
+
 def publish_dra_step(branch, workflow_type, depends_on):
     step = f'''
 - label: ":elastic-stack: Publish  / {branch}-{workflow_type.upper()} DRA artifacts"
@@ -139,9 +194,14 @@ def build_steps_to_yaml(branch, workflow_type):
             "steps": build_steps_to_yaml(branch, workflow_type),
         })
 
-        # Final step: pull artifacts built above and publish them via the release-manager
+        # Pull artifacts built above and publish them via the release-manager
         structure["steps"].extend(
             yaml.safe_load(publish_dra_step(branch, workflow_type, depends_on=group_key)),
         )
 
+        # Once published, do the same for observabilitySRE image
+        structure["steps"].extend(
+            yaml.safe_load(ship_observability_sre_image_steps(branch, workflow_type)),
+        )
+
     print(YAML_HEADER + yaml.dump(structure, Dumper=yaml.Dumper, sort_keys=False))

.buildkite/scripts/dra/multi-architecture-observability-sre.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+# Script to create and push Docker manifest for multi-architecture support
+# This MUST be run after build-and-push-observabilty-sre.sh!
+
+source .buildkite/scripts/common/vm-agent.sh
+source .buildkite/scripts/dra/docker-env-setup.sh
+
+docker_login
+
+# Set INCLUDE_COMMIT_ID to include git SHA in version
+QUALIFIED_VERSION="$(INCLUDE_COMMIT_ID=1 .buildkite/scripts/common/qualified-version.sh)"
+REGISTRY_PATH=docker.elastic.co/logstash/logstash-observability-sre
+
+# Architecture-specific tags (created by the build steps)
+X86_64_TAG="${QUALIFIED_VERSION}-x86_64"
+AARCH64_TAG="${QUALIFIED_VERSION}-aarch64"
+
+# Target manifest tags - already has SHA from QUALIFIED_VERSION
+VERSION_MANIFEST_TAG="${QUALIFIED_VERSION}"
+
+# Create and push manifest with version (which already includes SHA)
+echo "Creating manifest list for: ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}"
+docker manifest create ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
+  ${REGISTRY_PATH}:${X86_64_TAG} \
+  ${REGISTRY_PATH}:${AARCH64_TAG}
+
+docker manifest annotate ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
+  ${REGISTRY_PATH}:${X86_64_TAG} --os linux --arch amd64
+
+docker manifest annotate ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
+  ${REGISTRY_PATH}:${AARCH64_TAG} --os linux --arch arm64
+
+echo "Pushing manifest: ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}"
+docker manifest push ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}
+
+# Also create version without SHA for effective "latest" tag
+BASE_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
+echo "Creating manifest list for: ${REGISTRY_PATH}:${BASE_VERSION}"
+docker manifest create ${REGISTRY_PATH}:${BASE_VERSION} \
+  ${REGISTRY_PATH}:${X86_64_TAG} \
+  ${REGISTRY_PATH}:${AARCH64_TAG}
+
+docker manifest annotate ${REGISTRY_PATH}:${BASE_VERSION} \
+  ${REGISTRY_PATH}:${X86_64_TAG} --os linux --arch amd64
+
+docker manifest annotate ${REGISTRY_PATH}:${BASE_VERSION} \
+  ${REGISTRY_PATH}:${AARCH64_TAG} --os linux --arch arm64
+
+echo "Pushing manifest: ${REGISTRY_PATH}:${BASE_VERSION}"
+docker manifest push ${REGISTRY_PATH}:${BASE_VERSION}
+
+# Teardown Docker environment
+source .buildkite/scripts/dra/docker-env-teardown.sh