✨ Make package sources flexible; support GH releases (#8)

Daniel Ruthardt · web-flow · commit 681f7882bd1d · 2024-12-20T14:11:35.000+01:00
* ✨ Add support for Basher installation Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * ✨ Make package sources flexible; support ghreleases Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🚸 Add binaries so Basher picks up the right files Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 📝 Remove bin directory I assumed installation option #2, but even then, it is safe to assume that someone would install the binaries or add the `bin` path to the PATH. * 🚚 Rename ghrelease:// to gh:// * 🚚 Rename ghrelease:// to gh:// * 🚚 Rename ghrelease:// to gh:// * 🚚 Rename ghrelease:// to gh:// * 🚸 Be XDG Base Directory Specification compliant * 🚸 Be XDG Base Directory Specification compliant * ♻️ Use another, "non-recursive" but common pattern Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🩹 Add missing file Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 📝 gh has been replaced with curl * 📝 Remove removed installation method Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🩹 Forward another relevant variable Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 📝 Update documentation blocks Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🔥 Remove accidentally committed files Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🩹 Make users and linters happy Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🐛 Ensure output isn't accidentally used as code Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🩹 Properly check for required env vars Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🧑‍💻 Exit early on failure Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🐛 Properly parse the URL of the correct asset Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> * 🐛 Create missing directory structures Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org> --------- Signed-off-by: Daniel Ruthardt <druthardt@linuxfoundation.org>
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -22,3 +22,5 @@ jobs:
             bashp
             bashpc
             bashpp
+            bashpp-gh
+            bashpp-oci
diff --git a/README.md b/README.md
@@ -13,23 +13,25 @@ functionalities with ease.
 
 - **Modular Design**: Easily include and manage external libraries and packages in your Bash
   scripts.
-- **Package Management**: Leverage reusable components and packages, potentially hosted in external
-  repositories or available as Docker images.
+- **Package Management**: Leverage reusable components and packages from multiple sources:
+  - OCI registries (Docker/other OCI clients)
+  - GitHub releases
+  - Local directories
 - **Enhanced Scripting Capabilities**: Make your Bash scripts more powerful and maintainable with
   advanced features.
 - **Preprocessing**: A preprocessing step that allows for dynamic inclusion of scripts and
-  packages. Supports including functions from local directories or Docker repositories, enhancing
-  modularity and ease of use.
-- **Git Integration**: Automatically adds included functions to `.gitignore` when pulling from OCI
-  registries, preventing unintended commits of external functions.
+  packages, supporting multiple package sources and enhancing modularity.
+- **Git Integration**: Automatically adds included functions to `.gitignore` when pulling from
+  external sources, preventing unintended commits.
 
 ## Getting Started
 
 You can install Bash+ in two ways:
 
 ### Option 1: Using Basher (Recommended)
 
-The easiest way to install Bash+ is using [Basher](https://www.basher.it/), a package manager for Bash scripts:
+The easiest way to install Bash+ is using [Basher](https://www.basher.it/), a package manager for
+Bash scripts:
 
 ```bash
 # Install basher if you haven't already
@@ -41,7 +43,10 @@ basher install lf-certification/bashp
 
 ### Option 2: Manual Installation
 
-Alternatively, you can clone this repository and ensure you have Docker installed, as it's required for fetching some of the packages. Ensure Git is configured correctly if you're using Bash+ in a Git-managed project. Bash+ intelligently adds dynamically included functions to `.gitignore`, keeping your repository clean.
+Alternatively, you can clone this repository and ensure you have Docker and `curl` installed, as
+they are required for fetching some of the packages. Ensure Git is configured correctly if you're
+using Bash+ in a Git-managed project. Bash+ intelligently adds dynamically included functions to
+`.gitignore`, keeping your repository clean.
 
 ### Usage
 
@@ -70,7 +75,8 @@ test::hello
 test::hello_world
 ```
 
-This script demonstrates two ways to include and use functions from external libraries or packages in your Bash+ scripts.
+This script demonstrates two ways to include and use functions from external libraries or packages
+in your Bash+ scripts.
 
 ## Contributing
 
@@ -79,4 +85,5 @@ features.
 
 ## License
 
-This project is licensed under the BSD-3-Clause license - see the [LICENSE](LICENSE) file for details.
+This project is licensed under the BSD-3-Clause license - see the [LICENSE](LICENSE) file for
+details.
diff --git a/bashpp b/bashpp
@@ -6,13 +6,11 @@ set -e
 #
 # Description:
 #   bashpp is a Bash script preprocessor that allows for the inclusion of external functions into
-#   Bash scripts. It supports including functions from local directories specified by
-#   $BASHP_INCLUDE_DIRS or from Docker repositories listed in $BASHP_INCLUDE_REPOS. This enables
-#   modular script development by allowing scripts to dynamically include and use functions from
-#   external packages. The script processes input files line by line, identifying and handling
-#   include directives for external functions. If a function is not already included, bashpp
-#   attempts to find it locally or pull it from a Docker repository. Each included function is then
-#   preprocessed for further include directives.
+#   Bash scripts. It supports including functions from:
+#   - Local directories specified by $BASHP_INCLUDE_DIRS
+#   - External sources via extension handlers (e.g., bashpp-oci, bashpp-gh)
+#   This enables modular script development by allowing scripts to dynamically include and use
+#   functions from external packages.
 #
 # Usage:
 #   bashpp [file]
@@ -25,20 +23,33 @@ set -e
 #   BASHP_INCLUDE_DIRS - Colon-separated list of directories to search for functions. If not set,
 #                        defaults to the value of $INCLUDE_DIR or 'libs' if $INCLUDE_DIR is also
 #                        unset.
-#   BASHP_INCLUDE_REPOS - Colon-separated list of Docker repositories to search for functions.
+#   BASHP_INCLUDE_REPOS - Comma-separated list of repositories to search for functions.
+#                         Supports the following formats:
+#                         - OCI registries (default prefix or oci://)
+#                         - GitHub releases (gh://)
 #                         Defaults to 'ghcr.io/lf-certification' if unset.
 #   INCLUDE_DIR - Directory to search for functions. Used if BASHP_INCLUDE_DIRS is not set.
+#                 Defaults to 'libs' if unset.
+#   OCI_CLIENT - OCI client to use for pulling packages (defaults to 'docker')
+#   BASHP_CACHE_DIR - Cache directory for GitHub release assets (defaults to XDG_CACHE_HOME/bashp
+#                     or ~/.cache/bashp)
+#   GH_TOKEN/GITHUB_TOKEN - GitHub authentication token for accessing releases (optional)
 #
 # Dependencies:
-#   - Docker must be installed and available in the PATH if external functions are to be pulled 
-#     from Docker repositories.
+#   - Docker (or alternative OCI client specified by OCI_CLIENT) for pulling from OCI registries
+#   - curl for pulling from GitHub releases
 #
 # Notes:
-#   - Functions to be included should be placed in directories or Docker repositories following the
-#     structure <package>/<function>, where <package> is the name of the package containing the
-#     function, and <function> is the name of the function file.
-#   - The script sets the 'set -e' option to exit immediately if a command exits with a non-zero
-#     status, ensuring that errors in function inclusion or preprocessing are not silently ignored.
+#   - Functions to be included should be placed in directories or repositories following the
+#     structure <package>/<function>
+#   - For OCI registries, packages should be named 'bashp-<package>:latest'
+#   - For GitHub releases, assets should be named 'bashp-<package>.tar.gz'
+#   - The script automatically adds included functions to .gitignore when pulled from external
+#     sources to prevent accidental commits
+#   - The script sets 'set -e' to exit immediately on errors
+#   - GitHub releases are cached in $BASHP_CACHE_DIR to avoid repeated downloads
+#   - Repository URLs can include explicit protocol handlers (oci://, gh://)
+#   - If no protocol is specified in repository URL, OCI is assumed
 
 # include
 # Attempts to include a specified function from a given package.
@@ -68,26 +79,12 @@ function include() {
     [[ -f "$path" ]] && break
   done
 
-  if [[ ! -f "$path" ]] && command -v docker &>/dev/null; then
-    local repository
-    IFS=:; for repository in $BASHP_INCLUDE_REPOS; do
-      local package=${include%%/*}
-      local image="$repository/bashp-$package:latest"
-      if docker manifest inspect "$image" &>/dev/null; then
-        local id
-        id=$(docker create "$image" none)
-        mkdir -p "$INCLUDE_DIR/$package"
-        docker cp "$id:/$include" "$INCLUDE_DIR/$include"
-        docker rm "$id"
-
-        if command -v git &>/dev/null && git rev-parse --is-inside-work-tree &>/dev/null; then
-          local function=${include##*/}
-          local gitignore="$INCLUDE_DIR/$package/.gitignore"
-          [[ -f "$gitignore" ]] && grep -q "^$function\$" "$gitignore" || \
-              echo "$function" >> "$gitignore"
-        fi
-        break
-      fi
+  if [[ ! -f "$path" ]]; then
+    IFS=,; for repository in $BASHP_INCLUDE_REPOS; do
+      local handler=${repository%%://*}
+      [[ "$handler" != "$repository" ]] || handler=oci
+      INCLUDE_DIR=$INCLUDE_DIR BASHP_CACHE_DIR="$BASHP_CACHE_DIR" \
+        "bashpp-$handler" "${repository#*://}" "$include" >/dev/null
     done
   fi
 
@@ -130,13 +127,13 @@ function include_once() {
 #   preprocess <file>
 #
 # Parameters:
-#   file - The path to the file to be processed.
+#   file - The path to the file to be processed. If not specified, reads from /dev/stdin.
 #
 # Description:
 #   This function reads the specified file line by line. For lines that contain a "::" or
 #   "#include" directive, it calls the include function to handle the specified function. Each line
 #   is echoed back, with modifications applied to lines containing directives for function
-#   inclusion.
+#   inclusion. A newline is added at the end of the file if the last line is not empty.
 function preprocess() {
   local file=$1
 
diff --git a/bashpp-gh b/bashpp-gh
@@ -0,0 +1,65 @@
+# shellcheck shell=bash
+
+# Script: bashpp-gh
+# Extension handler for downloading packages from GitHub releases.
+#
+# Description:
+#   This script serves as an extension handler for bashpp to pull packages from GitHub releases.
+#   When bashpp encounters a GitHub repository URL (gh:// prefix), it delegates the package
+#   retrieval to this handler. The handler downloads the latest release asset, extracts the
+#   specified include file, and installs it to the appropriate directory.
+#
+# Usage:
+#   bashpp-gh <repository> <include>
+#   Called automatically by bashpp when processing GitHub repository URLs
+#
+# Parameters:
+#   repository - The GitHub repository in the format "owner/repo"
+#   include    - The path to the file to include, in the format "package/function"
+#
+# Outputs:
+#   No direct output, but creates files in the cache directory and include directory.
+#   Updates .gitignore if running within a git repository.
+#
+# Note:
+#   Requires curl to be installed.
+#   Requires either GH_TOKEN or GITHUB_TOKEN environment variable for GitHub API access.
+#   Will create cache directories as needed at BASHP_CACHE_DIR, XDG_CACHE_HOME/bashp or
+#   ~/.cache/bashp.
+
+declare -r INCLUDE_DIR=${INCLUDE_DIR:-${INCLUDE_DIR:?must be set}}
+declare -r repository=${1:-${repository:?must be set}}
+declare -r include=${2:-${include:?must be set}}
+
+command -v curl &>/dev/null || exit 1
+
+declare -r package=${include%%/*}
+declare -r cache="${BASHP_CACHE_DIR:-${XDG_CACHE_HOME:-${HOME}/.cache}/bashp}/$repository/$package"
+
+if [[ ! -f "$cache/$include" ]]; then
+  declare -r asset="bashp-$package.tar.gz"
+
+  response=$(curl -H "Authorization: Bearer ${GH_TOKEN:-${GITHUB_TOKEN}}" -sSf \
+    "https://api.github.com/repos/$repository/releases/latest" | tr -d "\n")
+  declare -r response
+  [[ -z "$response" ]] && exit 1
+
+  url=$(echo "$response" | grep -Eo '"url":\s*"[^"]*"[^}]*"name":\s*"'"$asset"'"' |
+    grep -Eo '"url":\s*"[^"]*"' | cut -d'"' -f4)
+  declare -r url
+  [[ -z "$url" ]] && exit 1
+
+  mkdir -p "$cache"
+  curl -H "Authorization: Bearer ${GH_TOKEN:-${GITHUB_TOKEN}}" \
+        -H "Accept: application/octet-stream" -sSfL "$url" -o "$cache/$asset" || exit 1
+  tar -xzf "$cache/$asset" -C "$cache"
+fi
+
+mkdir -p "$INCLUDE_DIR/$package"
+cp "$cache/$include" "$INCLUDE_DIR/$include"
+
+if command -v git &>/dev/null && git rev-parse --is-inside-work-tree &>/dev/null; then
+  declare -r function=${include##*/}
+  declare -r gitignore="$INCLUDE_DIR/$package/.gitignore"
+  grep -q "^$function$" "$gitignore" 2>/dev/null || echo "$function" >> "$gitignore"
+fi
diff --git a/bashpp-oci b/bashpp-oci
@@ -0,0 +1,57 @@
+# shellcheck shell=bash
+
+# bashpp-oci
+# Extension handler for downloading packages from OCI container images.
+#
+# Description:
+#   This script serves as an extension handler for bashpp to pull packages from OCI container
+#   images.
+#   When bashpp encounters an OCI repository URL (default or oci:// prefix), it delegates the
+#   package retrieval to this handler. The handler pulls the package, extracts the specified
+#   include file, and installs it to the appropriate directory.
+#
+# Usage:
+#   bashpp-oci <repository> <include>
+#   Called automatically by bashpp when processing OCI repository URLs
+#
+# Parameters:
+#   repository - The OCI repository containing the package image
+#   include    - The path to the file to include, in the format "package/function"
+#
+# Outputs:
+#   No direct output, but creates files in the include directory and updates .gitignore
+#   if running within a git repository.
+#
+# Note:
+#   Requires an OCI client (docker by default) to be installed.
+#   The OCI client can be configured using the OCI_CLIENT environment variable.
+#   The image is expected to be tagged as 'latest' and follow the naming convention
+#   'repository/bashp-package:latest'.
+
+declare -r INCLUDE_DIR=${INCLUDE_DIR:-${INCLUDE_DIR:?must be set}}
+declare -r repository=${1:-${repository:?must be set}}
+declare -r include=${2:-${include:?must be set}}
+
+declare -r ociClient=${OCI_CLIENT:-docker}
+command -v "$ociClient" &>/dev/null || exit 1
+
+declare -r package=${include%%/*}
+declare -r image="$repository/bashp-$package:latest"
+if "$ociClient" manifest inspect "$image" &>/dev/null; then
+  mkdir -p "$INCLUDE_DIR/$package"
+
+  id=$("$ociClient" create "$image" none)
+  declare -r id
+
+  # shellcheck disable=SC2064
+  trap "'$ociClient' rm '$id' >/dev/null" EXIT
+
+  "$ociClient" cp "$id:/$include" "$INCLUDE_DIR/$include"
+
+  if command -v git &>/dev/null && git rev-parse --is-inside-work-tree &>/dev/null; then
+    declare -r function=${include##*/}
+    declare -r gitignore="$INCLUDE_DIR/$package/.gitignore"
+    grep -q "^$function$" "$INCLUDE_DIR/$package/.gitignore" 2>/dev/null ||
+      echo "$function" >> "$gitignore"
+  fi
+fi
