Skip to content

Commit 9c81369

Browse files
TechassiNickLarsenNZ
Techassi
and
NickLarsenNZ
authored
feat!: Split up CI workflow per product (#733)
* Add build and publish actions * Add newline to build action file * Fix yamllint errors * Add Airflow workflow file * Use env var for product name in some places * Use sharding * Remove leading dot from actionlint file * Just enumerate the product specific versions * Don't enumerate product versions when creating manifest * Extract the image version from bake-target-tags file * Also use correct manifest name for Harbor * Add separate workflows for each product * Delete old workflow file * Narrow workflow path selectors * Add an action to generate shards for the matrix * Use shard action in workflows * fix the shard action * debug: airflow * debug: airflow * fix action inputs config * fix: add new lines to github outputs for shard action * revert airflow changes for debugging * bash strict mode * split manifest publishing * replace cut with sed to strip architecture suffices * Add the product name and version to the job name * Collapse conf.py * Fix product names * Remove product name from the job name * Use product versions as the matrix * Include architecture in the job name * Update matrix * Update runs-on values * add schedule to run every 2 days * fix publish manifests job name * rename generate_matrix job * update trigger paths * remove pull_request action ahead of merge * move registry usernames to input with a default * make SBOM step names clearer * pin setup-python action version * remove expired comment * pin checkout action * keep yaml-lint happy * keep yaml-lint happy again --------- Co-authored-by: Nick Larsen <[email protected]>
1 parent 98fbaa9 commit 9c81369

31 files changed

+2360
-310
lines changed

‎.github/actions/build/action.yml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
name: Build Product Image
3+
description: This action builds a product Docker image with a specific version
4+
inputs:
5+
product-name:
6+
description: The name of the product to build via bake (directory name)
7+
required: true
8+
product-version:
9+
description: The version of the product to build via bake
10+
required: true
11+
image-tools-version:
12+
description: The Stackable image-tools version
13+
default: 0.0.8
14+
outputs:
15+
image-name:
16+
description: This is the full image name before the tag (left of ':')
17+
value: ${{ steps.image_info.outputs.IMAGE_NAME }}
18+
image-version:
19+
description: This is the container image end tag excluding the architecture information (right of ':')
20+
value: ${{ steps.image_info.outputs.IMAGE_VERSION }}
21+
runs:
22+
using: composite
23+
steps:
24+
- name: Setup Docker Buildx
25+
uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0
26+
27+
# NOTE (@Techassi): Why do we install python via apt and not the setup-python action?
28+
- name: Setup Python
29+
shell: bash
30+
run: |
31+
set -euo pipefail
32+
sudo apt update
33+
sudo apt install -y python3
34+
35+
- name: Building ${{ inputs.product-name }}
36+
shell: bash
37+
run: echo ${{ inputs.product-name }}
38+
39+
- name: Install image-tools-stackabletech
40+
shell: bash
41+
run: pip install image-tools-stackabletech==${{ inputs.image-tools-version }}
42+
43+
- name: Build image using bake
44+
shell: bash
45+
run: |
46+
set -euo pipefail
47+
ARCH="$(uname -m | sed -e 's#x86_64#amd64#' | sed -e 's#aarch64#arm64#')"
48+
bake \
49+
--product ${{ inputs.product-name }}=${{ inputs.product-version }} \
50+
--image-version "0.0.0-dev-${ARCH}" \
51+
--architecture "linux/${ARCH}" \
52+
--export-tags-file bake-target-tags
53+
54+
- name: Setup Environment Variables
55+
id: image_info
56+
shell: bash
57+
run: |
58+
set -euo pipefail
59+
echo "bake-target-tags: "$(< bake-target-tags)
60+
# Strip the architecture from the version tag
61+
IMAGE_VERSION=$(cat bake-target-tags | cut -d ":" -f 2 | sed -E 's/-(amd64|arm64)$//')
62+
IMAGE_NAME=$(cat bake-target-tags | cut -d ":" -f 1)
63+
64+
echo "IMAGE_VERSION=$IMAGE_VERSION" >> $GITHUB_OUTPUT
65+
echo "IMAGE_NAME=$IMAGE_NAME" >> $GITHUB_OUTPUT

‎.github/actions/publish-image/action.yml

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
name: Publish Product Image
3+
description: This action publishes a Docker image
4+
inputs:
5+
product:
6+
description: The name of the product to publish
7+
required: true
8+
image-name:
9+
description: This is the full image name before the tag (left of ':')
10+
required: true
11+
image-version:
12+
description: This is the container image end tag excluding the architecture information (right of ':')
13+
required: true
14+
nexus-username:
15+
description: The username to login to Nexus
16+
default: github
17+
nexus-password:
18+
description: The password to login to Nexus
19+
required: true
20+
harbor-username:
21+
description: The username to login to Harbor
22+
default: robot$sdp+github-action-build
23+
harbor-secret:
24+
description: The secret to login to Harbor
25+
required: true
26+
runs:
27+
using: composite
28+
steps:
29+
- name: Set up Cosign
30+
uses: sigstore/cosign-installer@59acb6260d9c0ba8f4a2f9d9b48431a222b68e20 # v3.5.0
31+
32+
- name: Set up syft
33+
uses: anchore/sbom-action/download-syft@e8d2a6937ecead383dfe75190d104edd1f9c5751 # v0.16.0
34+
35+
- name: Login to Stackable Nexus
36+
uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
37+
with:
38+
registry: docker.stackable.tech
39+
username: ${{ inputs.nexus-username }}
40+
password: ${{ inputs.nexus-password }}
41+
42+
- name: Login to Stackable Harbor
43+
uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
44+
with:
45+
registry: oci.stackable.tech
46+
username: ${{ inputs.harbor-username }}
47+
password: ${{ inputs.harbor-secret }}
48+
49+
- name: Setup Environment Variables
50+
shell: bash
51+
run: |
52+
set -euo pipefail
53+
IMAGE_VERSION=${{ inputs.image-version }}
54+
IMAGE_NAME=${{ inputs.image-name }}
55+
ARCH="$(uname -m | sed -e 's#x86_64#amd64#' | sed -e 's#aarch64#arm64#')"
56+
TAG_NAME="${IMAGE_VERSION}-${ARCH}"
57+
58+
echo "IMAGE_VERSION=$IMAGE_VERSION" >> $GITHUB_ENV
59+
echo "IMAGE_NAME=$IMAGE_NAME" >> $GITHUB_ENV
60+
echo "TAG_NAME=$TAG_NAME" >> $GITHUB_ENV
61+
62+
- name: Push Image to repo.stackable.tech and sign via cosign
63+
shell: bash
64+
run: |
65+
set -euo pipefail
66+
# Store the output of `docker image push` into a variable, so we can
67+
# parse it for the digest
68+
PUSH_OUTPUT=$(docker image push "$(< bake-target-tags)" 2>&1)
69+
echo "$PUSH_OUTPUT"
70+
# Obtain the digest of the pushed image from the output of `docker image
71+
# push`, because signing by tag is deprecated and will be removed from
72+
# cosign in the future
73+
DIGEST=$(echo "$PUSH_OUTPUT" | awk "/: digest: sha256:[a-f0-9]{64} size: [0-9]+$/ { print \$3 }")
74+
echo "DIGEST=$DIGEST" >> $GITHUB_ENV
75+
# Refer to image via its digest (docker.stackable.tech/stackable/airflow@sha256:0a1b2c...)
76+
# This generates a signature and publishes it to the registry, next to the image
77+
# Uses the keyless signing flow with Github Actions as identity provider
78+
cosign sign -y "$IMAGE_NAME@$DIGEST"
79+
80+
- name: Generate SBOM for the Nexus Image
81+
shell: bash
82+
run: |
83+
set -euo pipefail
84+
syft scan --output cyclonedx-json=sbom.json --select-catalogers "-cargo-auditable-binary-cataloger" --scope all-layers --source-name "${{ inputs.product }}" --source-version "$TAG_NAME" "$IMAGE_NAME@$DIGEST";
85+
# Determine the PURL for the image
86+
PURL="pkg:docker/stackable/${{ inputs.product }}@$DIGEST?repository_url=docker.stackable.tech";
87+
# Get metadata from the image
88+
IMAGE_METADATA_DESCRIPTION=$(docker inspect --format='{{.Config.Labels.description}}' "$IMAGE_NAME@$DIGEST");
89+
IMAGE_METADATA_NAME=$(docker inspect --format='{{.Config.Labels.name}}' "$IMAGE_NAME@$DIGEST");
90+
# Merge the SBOM with the metadata for the image
91+
jq -s '{"metadata":{"component":{"description":"'"$IMAGE_METADATA_NAME. $IMAGE_METADATA_DESCRIPTION"'","supplier":{"name":"Stackable GmbH","url":["https://stackable.tech/"]},"author":"Stackable GmbH","purl":"'"$PURL"'","publisher":"Stackable GmbH"}}} * .[0]' sbom.json > sbom.merged.json;
92+
# Attest the SBOM to the image
93+
cosign attest -y --predicate sbom.merged.json --type cyclonedx "$IMAGE_NAME@$DIGEST"
94+
95+
- name: Push Image to oci.stackable.tech and sign via cosign
96+
shell: bash
97+
run: |
98+
set -euo pipefail
99+
IMAGE_NAME=oci.stackable.tech/sdp/${{ inputs.product }}
100+
echo "image: $IMAGE_NAME"
101+
docker tag "$(< bake-target-tags)" "$IMAGE_NAME:$TAG_NAME"
102+
# Store the output of `docker image push` into a variable, so we can parse it for the digest
103+
PUSH_OUTPUT=$(docker image push "$IMAGE_NAME:$TAG_NAME" 2>&1)
104+
echo "$PUSH_OUTPUT"
105+
# Obtain the digest of the pushed image from the output of `docker image push`, because signing by tag is deprecated and will be removed from cosign in the future
106+
DIGEST=$(echo "$PUSH_OUTPUT" | awk "/: digest: sha256:[a-f0-9]{64} size: [0-9]+$/ { print \$3 }")
107+
echo "DIGEST=$DIGEST" >> $GITHUB_ENV
108+
# Refer to image via its digest (oci.stackable.tech/sdp/airflow@sha256:0a1b2c...)
109+
# This generates a signature and publishes it to the registry, next to the image
110+
# Uses the keyless signing flow with Github Actions as identity provider
111+
cosign sign -y "$IMAGE_NAME@$DIGEST"
112+
113+
- name: Generate SBOM for the Harbor Image
114+
shell: bash
115+
run: |
116+
set -euo pipefail
117+
syft scan --output cyclonedx-json=sbom.json --select-catalogers "-cargo-auditable-binary-cataloger" --scope all-layers --source-name "${{ inputs.product }}" --source-version "$TAG_NAME" "$IMAGE_NAME@$DIGEST";
118+
# Determine the PURL for the image
119+
PURL="pkg:docker/sdp/${{ inputs.product }}@$DIGEST?repository_url=oci.stackable.tech";
120+
# Get metadata from the image
121+
IMAGE_METADATA_DESCRIPTION=$(docker inspect --format='{{.Config.Labels.description}}' "$IMAGE_NAME@$DIGEST");
122+
IMAGE_METADATA_NAME=$(docker inspect --format='{{.Config.Labels.name}}' "$IMAGE_NAME@$DIGEST");
123+
# Merge the SBOM with the metadata for the image
124+
jq -s '{"metadata":{"component":{"description":"'"$IMAGE_METADATA_NAME. $IMAGE_METADATA_DESCRIPTION"'","supplier":{"name":"Stackable GmbH","url":["https://stackable.tech/"]},"author":"Stackable GmbH","purl":"'"$PURL"'","publisher":"Stackable GmbH"}}} * .[0]' sbom.json > sbom.merged.json;
125+
# Attest the SBOM to the image
126+
cosign attest -y --predicate sbom.merged.json --type cyclonedx "$IMAGE_NAME@$DIGEST"

‎.github/actions/publish-manifest/action.yml

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
name: Publish Product Image Manifest
3+
description: This action publishes a Docker image
4+
inputs:
5+
product:
6+
description: The name of the product to publish
7+
required: true
8+
image-version:
9+
description: This is the container image end tag excluding the architecture information (right of ':')
10+
required: true
11+
nexus-password:
12+
description: The password to login to Nexus
13+
required: true
14+
harbor-secret:
15+
description: The secret to login to Harbor
16+
required: true
17+
runs:
18+
using: composite
19+
steps:
20+
- name: Set up Cosign
21+
uses: sigstore/cosign-installer@59acb6260d9c0ba8f4a2f9d9b48431a222b68e20 # v3.5.0
22+
23+
- name: Login to Stackable Nexus
24+
uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
25+
with:
26+
registry: docker.stackable.tech
27+
username: github
28+
password: ${{ inputs.nexus-password }}
29+
30+
- name: Login to Stackable Harbor
31+
uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
32+
with:
33+
registry: oci.stackable.tech
34+
username: robot$sdp+github-action-build
35+
password: ${{ inputs.harbor-secret }}
36+
37+
- name: Setup Environment Variables
38+
shell: bash
39+
run: |
40+
set -euo pipefail
41+
IMAGE_VERSION=${{ inputs.image-version }}
42+
echo "IMAGE_VERSION=$IMAGE_VERSION" >> $GITHUB_ENV
43+
44+
- name: Create Manifest
45+
shell: bash
46+
run: |
47+
set -euo pipefail
48+
MANIFEST_NAME="docker.stackable.tech/stackable/${{ inputs.product }}:$IMAGE_VERSION"
49+
# Create and push to Stackable Nexus
50+
# `docker manifest push` directly returns the digest of the manifest list
51+
# As it is an experimental feature, this might change in the future
52+
# Further reading: https://docs.docker.com/reference/cli/docker/manifest/push/
53+
# --amend because the manifest list would be updated since we use the same tag: 0.0.0-dev
54+
docker manifest create "$MANIFEST_NAME" --amend "${MANIFEST_NAME}-amd64" --amend "${MANIFEST_NAME}-arm64"
55+
DIGEST=$(docker manifest push $MANIFEST_NAME)
56+
57+
# Refer to image via its digest (oci.stackable.tech/sdp/airflow@sha256:0a1b2c...)
58+
# This generates a signature and publishes it to the registry, next to the image
59+
# Uses the keyless signing flow with Github Actions as identity provider
60+
cosign sign -y "$MANIFEST_NAME@$DIGEST"
61+
62+
# Push to oci.stackable.tech as well
63+
MANIFEST_NAME="oci.stackable.tech/sdp/${{ inputs.product }}:$IMAGE_VERSION"
64+
docker manifest create "$MANIFEST_NAME" --amend "${MANIFEST_NAME}-amd64" --amend "${MANIFEST_NAME}-arm64"
65+
DIGEST=$(docker manifest push $MANIFEST_NAME)
66+
cosign sign -y "$MANIFEST_NAME@$DIGEST"

‎.github/actions/shard/action.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
name: Generate Shards
3+
description: This action builds list of shard indices for use in Github Actions Matrices
4+
inputs:
5+
product:
6+
description: The name of the product to build via bake (directory name)
7+
required: true
8+
outputs:
9+
versions:
10+
description: A list of product versions
11+
value: ${{ steps.generate_shards.outputs.VERSIONS }}
12+
runs:
13+
using: composite
14+
steps:
15+
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
16+
with:
17+
python-version: '3.12'
18+
- name: Generate Shards
19+
id: generate_shards
20+
shell: python
21+
env:
22+
PRODUCT_NAME: ${{ inputs.product }}
23+
run: |
24+
# Need to get the list of versions for the product
25+
import sys
26+
import os
27+
sys.path.append(str(os.getcwd()))
28+
import conf
29+
30+
product=os.environ['PRODUCT_NAME']
31+
print(f"Generating version list for {product}")
32+
33+
# get the product config
34+
product_conf = list(filter(lambda x: x["name"] == product, conf.products))[0]
35+
# list the versions, eg: [1.0, 1.1, 2.0]
36+
versions = [v["product"] for k,v in enumerate(product_conf["versions"])]
37+
output_versions = f"VERSIONS={versions}\n"
38+
39+
github_outputs_file = os.environ['GITHUB_OUTPUT']
40+
f = open(github_outputs_file, "w")
41+
print(f"Writing to $GITHUB_OUTPUT: {output_versions}")
42+
f.write(output_versions)
43+
f.close()
44+
- name: Print Shards
45+
shell: bash
46+
run: |
47+
set -euo pipefail
48+
echo versions=${{ steps.generate_shards.outputs.VERSIONS }}

0 commit comments

Comments
 (0)