[REQUEST]: Document air-gapped list of architectures / package types / artifact names

[lucabelluccini]

Description

The page https://www.elastic.co/guide/en/fleet/current/air-gapped.html#air-gapped-diy-epr shows the example of artifacts only for linux.
It would be nice to mention the possible values for <artifact_name>-<version>-<arch>-<package_type> or even develop a short script to pull all the values from EPR or Artifact API and generate the command to be ran.

Resources

Reach out privately as those are questions shared internally

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @lucabelluccini @111andre111

Stakeholders:

[lucabelluccini]

Examples:

https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.0-linux-arm64.tar.gz
https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.0-windows-x86_64.zip
https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.0-darwin-x86_64.tar.gz
https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.0-darwin-aarch64.tar.gz

The artifacts could be derived from https://www.elastic.co/api/product_versions (but we're missing some of them.
Alternatively, it could be just be used to get the versions.

[nimarezainia]

@lucabelluccini is the ask here to document/pull all the assets in the EPR?
If you could elaborate the use case or what the customer was trying to achieve that this would help with. If it's the content of the EPR, I wonder if there's a better way of doing it rather than documentation.

is this the list you are referring to: https://www.elastic.co/guide/en/fleet/current/air-gapped.html#host-artifact-registry

[lucabelluccini]

Sorry Nima I have to admit the issue description is not clear.

The ask is: Artifact Repository (not EPR) docs contain only assets for Linux, while we need a sort of script to to generate the content so that people can automatically pull assets for all platforms and architectures.

Let me see if I can contribute with a script. I'll update the issue within today.

[lucabelluccini]

I've written a very minimal Go program to download stuff from our https://www.elastic.co/api/product_versions
The main problems are this API doesn't provide:

• osquerybeat
• endpoint-dev
• cloudbeat
• prodfiler
• fleet-server

I've then followed the KISS and I've gone with a brute force script we might provide in the docs (downloads all binaries 8.x and 7.17.x):

#!/bin/bash

download_if_not_exists() {
  local FILE_URL="$1"     # URL to download from
  local DEST_PATH="$2"    # Directory where the file should be saved

  # Extract the filename from the URL
  local FILENAME=$(basename "$FILE_URL")

  # Full path where the file will be saved
  local LOCAL_FILE="$DEST_PATH/$FILENAME"

  # Create the destination directory if it does not exist
  mkdir -p "$DEST_PATH"

  # Check if the file exists
  if [ ! -f "$LOCAL_FILE" ]; then
    echo "File does not exist. Downloading to $LOCAL_FILE..."
    if curl -s -f -o "$LOCAL_FILE" "$FILE_URL"; then
        echo "Downloaded."
    else
        echo "Download failed due to missing file or other reasons. For some assets it is normal."
    fi
  else
    echo "File already exists at $LOCAL_FILE. Skipping download."
  fi
}

# Function to generate artifact URLs
generate_artifacts() {
    local version=("$@")
    
    binaries=(
        "apm-server/apm-server"
        "beats/auditbeat/auditbeat"
        "beats/elastic-agent/elastic-agent"
        "beats/filebeat/filebeat"
        "beats/heartbeat/heartbeat"
        "beats/metricbeat/metricbeat"
        "beats/osquerybeat/osquerybeat"
        "beats/packetbeat/packetbeat"
        "cloudbeat/cloudbeat"
        "endpoint-dev/endpoint-security"
        "fleet-server/fleet-server"
        "prodfiler/pf-host-agent"
        "prodfiler/pf-elastic-collector"
        "prodfiler/pf-elastic-symbolizer"
    )
    
    files=(
        "linux-arm64.tar.gz"
        "linux-x86_64.tar.gz"
        "darwin-x86_64.tar.gz"
        "windows-x86_64.zip"
        "darwin-x86_64.tar.gz"
        "darwin-aarch64.tar.gz"
    )
    
    artifact_extensions=(
        ""
        ".sha512"
        ".asc"
    )

    for binary in "${binaries[@]}"; do
        for file in "${files[@]}"; do
            for ext in "${artifact_extensions[@]}"; do
                asset_url="https://artifacts.elastic.co/downloads/${binary}-${version}-${file}${ext}"
                echo "$asset_url"
            done
        done
    done
}

DESTINATION_PATH=/tmp/assets

# Parse JSON data and process each version
curl -s https://www.elastic.co/api/product_versions | jq -c '.[] | map(select(.version_number | test("^(7\\.17\\.[0-9]+)|(8\\.[0-9]+\\.[0-9]+)$")) | .version_number)[]' -r  | sort -r -V | uniq | while IFS= read -r version; do
    generate_artifacts "$version" | while IFS= read -r fileUrl; do
        download_if_not_exists "$fileUrl" "$DESTINATION_PATH"
    done
done

[nimarezainia]

@lucabelluccini thanks for this. @davidkyle and I discussed this issue and again reviewed the documentation.
We have decided to address this request this way:

the document does include a sample for all linux related artifacts. In step 1b, we also describe how the artifacts are named and organized. We will enhance that description and also include a few tables showing all the possible options for: "artifact_type", "artiffact_name", "architecture" & "package type". the user would then be able to build the full artifact name they desire to download.

We can't include a script here as that would become another item to manage and support. We also can't rely on a static copy and paste of the content (like you see for the Linux options) as they tend to go stale also, not to mention the sheer number of options that need to be covered.

Hope this is ok to address this issue. We can track customer usage of these docs and make enhancements in the future if needed.

[lucabelluccini]

We will enhance that description and also include a few tables showing all the possible options for: "artifact_type", "artiffact_name", "architecture" & "package type". the user would then be able to build the full artifact name they desire to download.

I think it is a good tradeoff.

We can't include a script here as that would become another item to manage and support. We also can't rely on a static copy and paste of the content (like you see for the Linux options) as they tend to go stale also, not to mention the sheer number of options that need to be covered.

Makes sense - I can put this in a KB for our own reference and provided as-is with a best effort attempt to provide something automatic.

[kilfoyle]

Thanks for the suggestion @lucabelluccini and for the draft @nimarezainia. Here's a docs PR: #1305 with a preview of the change.

[kilfoyle]

I've merged the docs PR. If we need any further changes please let me know.