[On-Week] Automate chromium linux builds for reporting in Kibana (elastic#212674)

## Summary

Closes elastic#38277

This PR ~attempts to take a stab at automating~ automates the process
for building the linux headless chromium used for reporting within
Kibana.

The Idea here is that one would simply only need create an issue within
the Kibana project that will then trigger the build for chromium, we'll
be leveraging GitHub actions for this, the Github workflow defined only
kicks in when an issue is created with the label
`trigger-chromium-build`, when such an issue is created, the body of the
issue is analysed; there's an expectation that the body would contain a
fenced codeblock of type json that specifies the property
`puppeteer_version`, if this is found this value is read and, we then
trigger a build on buildkite passing along this value, in the case where
there are multiple fenced code blocks specifying the expected property
`puppeteer_version` we leave a message that the issue body be modified,
also if the expected label was added but the expected property is not
found we also leave a message prompting that the expected property be
provided.

Once the build has commenced a message containing the link to the build
would be provided as initial feedback to the user, also on completion
another message is provided to the user, that provides a link to the PR
containing all the required changes to update puppeteer to the version
specified.

~It's also worth pointing out that this PR also, modifies the source for
`win`, `mac` chromium binaries to leverage the [JSON API
endpoint](https://github.com/GoogleChromeLabs/chrome-for-testing#json-api-endpoints)
provided by Google to get the required binaries for chromium headless in
a deterministic way, which in turns is part of what makes this
automation possible.~

## How to test this

- If you'd love to test the Github action too without the PR being
merging in just yet, you should consider setting up
[act](https://github.com/nektos/act), alongside it's companion [vscode
extension](https://sanjulaganepola.github.io/github-local-actions-docs/)
, we'll then want to create a payload file providing similar data that
github would return for our workflow trigger, more info about setting
this up
[here](https://sanjulaganepola.github.io/github-local-actions-docs/usage/settings/#payloads).
The payload file we'd want would be something along the following lines;

```json
{
    "action": "labeled",
    "label":{
        "name": "trigger-chromium-build"
    },
    "issue": {
        "number": 1,
        "title": "Issue 1",
        "author_association": "MEMBER",
        "labels": [
            {
                "name": "trigger-chromium-build"
            }
        ],
        "body": "\n## Random text \n\n ~~~json\n{\n  \"puppeteer_version\": \"24.6.1\" \n}\n~~~\n~~~json\n{\n  \"some_random_value\": \"23.0.1\" \n}\n~~~"
    }
}
```

- To test the actual build process it can be initiated through this
specific pipeline
https://buildkite.com/elastic/kibana-migration-pipeline-staging by
creating a custom build on `pull/212674/head` (this pull request) with
the env variable similar to this

```
TESTED_PIPELINE_PATH=.buildkite/pipelines/chromium_linux_build/build_chromium.yml
PUPPETEER_VERSION=24.6.1
GITHUB_ISSUE_NUMBER=212732
GITHUB_ISSUE_BASE_OWNER=elastic
GITHUB_ISSUE_BASE_REPO=kibana
GITHUB_ISSUE_TRIGGER_USER=eokoneyo
```

PS: Issue elastic#212732 is an issue that's been created to test providing
feedback to the user about the build process, re-triggering a build on
an existing issue updates the comments in place.

<!-- ### Where to go from here?

- Ideas and thoughts welcome
-->
<!--
### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...


-->

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Author: eokoneyo

.buildkite/package-lock.json

@@ -11,6 +11,7 @@
   },
   "dependencies": {
     "@octokit/rest": "^18.10.0",
+    "adm-zip": "^0.5.16",
     "axios": "^1.8.3",
     "globby": "^11.1.0",
     "js-yaml": "^4.1.0",
@@ -21,11 +22,14 @@
   "devDependencies": {
     "@types/chai": "^4.3.3",
     "@types/js-yaml": "^4.0.9",
+    "@types/jscodeshift": "^0.12.0",
     "@types/minimatch": "^3.0.5",
     "@types/minimist": "^1.2.5",
     "@types/mocha": "^10.0.1",
     "@types/node": "^15.12.2",
     "chai": "^4.3.10",
+    "chai-snapshot-tests": "^0.6.0",
+    "jscodeshift": "^17.1.2",
     "mocha": "^11.0.1",
     "nock": "^12.0.2",
     "ts-node": "^10.9.2",

.buildkite/package.json

@@ -0,0 +1,40 @@
+# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
+apiVersion: backstage.io/v1alpha1
+kind: Resource
+metadata:
+  name: bk-kibana-chromium-linux-build
+  description: Create chromium linux builds for the Kibana Reporting feature
+  links:
+    - title: Pipeline link
+      url: https://buildkite.com/elastic/kibana-chromium-linux-build
+spec:
+  type: buildkite-pipeline
+  owner: group:response-ops
+  system: buildkite
+  implementation:
+    apiVersion: buildkite.elastic.dev/v1
+    kind: Pipeline
+    metadata:
+      name: kibana / Chromium Linux Build
+      description: Creates headless Chromium Linux builds
+    spec:
+      env:
+        SLACK_NOTIFICATIONS_CHANNEL: "#kibana-alerting"
+        ELASTIC_SLACK_NOTIFICATIONS_ENABLED: "false"
+      repository: elastic/kibana
+      branch_configuration: main
+      default_branch: main
+      pipeline_file: ".buildkite/pipelines/chromium_linux_build/build_chromium.yml"
+      provider_settings:
+        trigger_mode: none
+      teams:
+        kibana-operations:
+          access_level: MANAGE_BUILD_AND_READ
+        appex-sharedux:
+          access_level: MANAGE_BUILD_AND_READ
+        response-ops:
+          access_level: MANAGE_BUILD_AND_READ
+        everyone:
+          access_level: READ_ONLY
+      tags:
+        - kibana

.buildkite/pipeline-resource-definitions/kibana-chromium-linux-build.yml

@@ -14,6 +14,7 @@ spec:
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-artifacts-staging.yml
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-artifacts-trigger.yml
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-chrome-forward-testing.yml
+    - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-chromium-linux-build.yml
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-codeql.yml
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-console-definitions-sync.yml
     - https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-coverage-daily.yml

.buildkite/pipeline-resource-definitions/locations.yml

@@ -0,0 +1,59 @@
+env:
+  GITHUB_PR_TARGET_BRANCH: 'main' # common utils needs this value to be set
+
+steps:
+  - command: |
+      ts-node .buildkite/scripts/lifecycle/comment_on_pr.ts \
+        --message "Linux headless chromium build started at: $BUILDKITE_BUILD_URL" \
+        --context "chromium-linux-build-job-start" \
+        --issue-number $GITHUB_ISSUE_NUMBER \
+        --repository $GITHUB_ISSUE_BASE_REPO \
+        --repository-owner $GITHUB_ISSUE_BASE_OWNER \
+        --clear-previous
+    label: Comment with job URL on issue
+    agents:
+      provider: gcp
+      image: family/kibana-ubuntu-2004
+      imageProject: elastic-images-prod
+      machineType: n2-standard-2
+    timeout_in_minutes: 5
+
+  - command: |
+      .buildkite/scripts/pipelines/chromium_linux_build/chromium_version.sh
+    label: Infer chromium version for puppeteer version
+    key: infer_chromium_version
+    agents:
+      provider: gcp
+      image: family/kibana-ubuntu-2004
+      imageProject: elastic-images-prod
+      machineType: n2-standard-2
+    timeout_in_minutes: 5
+
+  - command: "PLATFORM_VARIANT={{matrix}} .buildkite/scripts/pipelines/chromium_linux_build/build.sh"
+    label: "Build {{matrix}} linux headless chromium"
+    depends_on: "infer_chromium_version"
+    key: build_chromium
+    env:
+      DEBIAN_FRONTEND: noninteractive
+    agents:
+      image: family/kibana-ubuntu-2004
+      imageProject: elastic-images-prod
+      provider: gcp
+      machineType: c2d-highcpu-112
+      diskSizeGb: 275
+    matrix:
+      - "x64"
+      - "arm64"
+
+  - command: node .buildkite/scripts/pipelines/chromium_linux_build/issue_feedback/entry.js
+    label: provided feedback to issue created to update puppeteer
+    depends_on: "build_chromium"
+    env:
+      KIBANA_MACHINE_USERNAME: kibanamachine
+      KIBANA_MACHINE_EMAIL: 42973632+kibanamachine@users.noreply.github.com
+    agents:
+      provider: gcp
+      image: family/kibana-ubuntu-2004
+      imageProject: elastic-images-prod
+      machineType: n2-standard-2
+    timeout_in_minutes: 30

.buildkite/pipelines/chromium_linux_build/build_chromium.yml

@@ -76,6 +76,9 @@ if [[ -z "$EMAIL" ]]; then
     "ci-artifacts.kibana.dev")
       EMAIL="kibana-ci-access-artifacts@$GCLOUD_EMAIL_POSTFIX"
       ;;
+    "kibana-ci-access-chromium-blds")
+      EMAIL="kibana-ci-access-chromium-blds@$GCLOUD_EMAIL_POSTFIX"
+      ;;
     *)
       EMAIL="$BUCKET_NAME@$GCLOUD_EMAIL_POSTFIX"
       ;;

.buildkite/scripts/common/activate_service_account.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+CHROMIUM_COMMIT_HASH=$(buildkite-agent meta-data get "chromium_commit_hash")
+
+echo "---Preparing to build Chromium of commit hash: $CHROMIUM_COMMIT_HASH"
+
+BUILD_ROOT_DIR="$HOME/chromium"
+
+KIBANA_CHECKOUT_DIR="$(pwd)"
+
+BUILD_SCRIPT="$KIBANA_CHECKOUT_DIR/x-pack/build_chromium"
+
+# Create a dedicated working directory outside of the default buildkite working directory.
+mkdir "$BUILD_ROOT_DIR" && cd "$BUILD_ROOT_DIR"
+
+ARTIFACT_STAGING_STORAGE_BUCKET="gs://headless_shell_staging"
+ARTIFACT_PROD_STORAGE_BUCKET="gs://headless_shell"
+
+ARTIFACT_QUERY="chromium-${CHROMIUM_COMMIT_HASH:0:7}-.*_$PLATFORM_VARIANT"
+
+## impersonate service account that has access to our storage bucket 
+"$KIBANA_CHECKOUT_DIR/.buildkite/scripts/common/activate_service_account.sh" "kibana-ci-access-chromium-blds"
+
+# Query to determine if expected build artifact from a prior build exists, 
+# the build.py script uploads the build artifact to the staging bucket
+artifacts=$(gsutil ls "$ARTIFACT_STAGING_STORAGE_BUCKET" | grep "$ARTIFACT_QUERY" || true)
+
+if [[ -z "$artifacts" ]]; then
+  echo "No files found matching the query: $ARTIFACT_QUERY"
+
+   echo "---Chromium build does not exist in the bucket, proceeding with the build"
+
+    # Install the required packages for building chromium
+    sudo apt-get update && \
+    sudo apt-get install -y curl git build-essential pkg-config gcc gperf python3 && \
+    sudo rm -rf /var/lib/apt/lists/*
+
+    BUILD_SCRIPT_SYMLINK="$BUILD_ROOT_DIR/build_scripts"
+
+    # link existing build_chromium directory from kibana
+    ln -s "$BUILD_SCRIPT" "$BUILD_SCRIPT_SYMLINK"
+
+    # Allow our scripts to use depot_tools commands
+    export PATH=$BUILD_ROOT_DIR/depot_tools:$PATH
+
+    # Install the OS packages, configure the environment, download the chromium source (56GB)
+    python3 "$BUILD_SCRIPT_SYMLINK/init.py"
+
+    echo "---Building $PLATFORM_VARIANT Chromium of commit hash: $CHROMIUM_COMMIT_HASH"
+
+    # Run the build script with the path to the chromium src directory, the git commit hash
+    python3 "$BUILD_SCRIPT_SYMLINK/build.py" "$CHROMIUM_COMMIT_HASH" "$PLATFORM_VARIANT"
+
+    echo "---Upload build artefact to prod storage bucket"
+
+    gsutil cp "$BUILD_ROOT_DIR/chromium/src/out/headless/chromium-*" "$ARTIFACT_PROD_STORAGE_BUCKET"
+
+    echo "---Persisting build artefact to buildkite for following steps"
+
+    buildkite-agent artifact upload "$BUILD_ROOT_DIR/chromium/src/out/headless/chromium-*"
+
+else
+  echo "$artifacts" | while read -r file; do
+    gsutil cp "$file" .
+  done
+
+  shopt -s nullglob
+  files=("chromium-*")
+  shopt -u nullglob
+  
+  if [ ${#files[@]} -gt 0 ]; then
+    echo "---Chromium build already exists in the bucket, skipping build"
+    # Upload the existing build artifact to buildkite so it can be used in the next steps, 
+    # and accessible without necessarily having access to the storage bucket itself
+    buildkite-agent artifact upload "chromium-*"
+  fi
+fi
+
+echo "---Build completed"

.buildkite/scripts/pipelines/chromium_linux_build/build.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+source .buildkite/scripts/common/util.sh
+.buildkite/scripts/bootstrap.sh
+
+echo "---Attempting to compute chromium version for provided puppeteer version"
+
+CHROMIUM_VERSION_OUTPUT=$(node scripts/chromium_version $PUPPETEER_VERSION)
+
+echo "$CHROMIUM_VERSION_OUTPUT" | grep -i "chromium commit" | awk '{print $5}' | buildkite-agent meta-data set "chromium_commit_hash"
+echo "$CHROMIUM_VERSION_OUTPUT" | grep -i "chrome version" | awk '{print $5}' | buildkite-agent meta-data set "chromium_version"
+echo "$CHROMIUM_VERSION_OUTPUT" | grep -i "chromium revision" | awk '{print $5}' | buildkite-agent meta-data set "chromium_revision"