Add workflow to rebuild all available old version and the latest one #2

OptimumCode · 2025-03-01T12:14:13Z

A workflow that rebuilds all available versions on demand. It uses tags to find previous versions plus it uses the main branch to build that latest available version

.github/workflows/build-all.yml

Julian · 2025-03-05T16:40:52Z

.github/workflows/build-all.yml

+      - name: Install qemu
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y qemu-user-static


May as well ask: do you know if Kotlin supports cross-compiling natively rather than needing to use QEMU? (some languages like rust, c# and golang I know do and we already use that functionality for some of those)

I don't know honestly. I can dig into that (I am not familiar with that topic - will learn something new 😃 ).

But Kotlin here is used as a JVM language. If Java supports cross-compiling so does Kotlin (I think so at least)

Actually, I think the answer would be yes because we compile kotlin into java classes which don't depend on the platform. So, we can remove this step I think

Or not...something went wrong once I removed the qemu
https://github.com/bowtie-json-schema/kotlin-kmp-json-schema-validator/actions/runs/13722121362/job/38379776771?pr=2

[1/2] STEP 6/8: RUN gradle --no-daemon --parallel dependencies > /dev/null
exec container process /bin/sh: Exec format error
Error: building at STEP "RUN gradle --no-daemon --parallel dependencies > /dev/null": while running runtime: exit status 1

I will try to debug it locally

Okay, so it looks like the problem is not in Java/Kotlin but in Gradle which uses sh to execute scripts. And apparently, this is not possible without emulation which is provided by QEMU

Huh, that's... strange. Shell scripts are text, so they shouldn't be arch-specific?? And I have to assume gradle isn't installing its own sh binary... is it????? If I can ask for one more time could you try to find someone who's maybe written a blog post on compiling cross-arch JVM images and maybe just confirm they ended up doing so via qemu? If so it's all fine to leave, but I'm just honestly curious.

No, it doesn't not install its own shell. I was looking into something like that but had no luck. I will try doing it one more time. Something doesn't add up here...

I have not found an article about creating multiarch image with Java app. However, I did a few experiments and everything works just fine without any modifications with docker (I don't believe it installs QEMU automatically)
https://github.com/bowtie-json-schema/kotlin-kmp-json-schema-validator/actions/runs/13725912881/job/38392187019

But with podman and buildah we are getting the same error...
Will continue looking.

CI is green without QEMU, however I am not yet sure how it happened (exactly the same configuration failed before...). Will dig deeper tomorrow

I have figured out why the same changes did not work before)
I used a build-all workflow to test them - it collects tags, checkouts them and builds images. And this is the part I was missing - Dockerfile on the tag does not contain the changes I did 😅 And the build failed

Re changes:

The only important thing is --platform=$BUILDPLATFORM in the Dockerfile. It checks out an image that is compatible with build platform and compiles Kotlin/Java classes. After that, the resulting files are moved to the container with target platform which can use those classes

.github/workflows/build-all.yml

OptimumCode · 2025-03-05T16:48:59Z

.github/workflows/build-all.yml

+      - name: Install qemu
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y qemu-user-static


I don't know honestly. I can dig into that (I am not familiar with that topic - will learn something new 😃 ).

But Kotlin here is used as a JVM language. If Java supports cross-compiling so does Kotlin (I think so at least)

OptimumCode · 2025-03-05T16:57:42Z

.github/workflows/build-all.yml

+      - name: Install qemu
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y qemu-user-static


Actually, I think the answer would be yes because we compile kotlin into java classes which don't depend on the platform. So, we can remove this step I think

Julian · 2025-03-07T15:30:31Z

.github/workflows/build-all.yml

+      matrix:
+        revision: ${{ fromJson(needs.versions.outputs.revisions) }}
+    permissions:
+      id-token: write


We really need all 3 of these write perms? (Another thing I'd like to do going forward is make sure we run zizmor on all workflows, which will complain about this, but if it's needed it's needed)

As I understand we need them to publish provenance. But I will double check.

Re zizmor: I will look at it (probably in a separate PR). Never heard about that tool before

id-token and attestations permissions are required by actions/attest-build-provenance@v2
https://github.com/actions/attest-build-provenance?tab=readme-ov-file#usage

packages required to publish the image

Not sure about contents though. According to this link contents: read are default permission for github token. So, probably we can remove it

But maybe it explicitly restricts permissions to read, because there is a column Default access (permissive) which has read/write value for that permission type

Reg zizmor: did I get you right that you want to run zizmor as a part of CI (whenever workflow files are changed)?

It actually also integrates with pre-commit hook:

https://woodruffw.github.io/zizmor/usage/#use-with-pre-commit

Yeah I forget if I've added it to bowtie itself but I've definitely added that workflow to a bunch of other repos I maintain

I didn't find zizmor in bowtie. Have you added it as a pre-commit hook or as a service workflow in other repos? Just to make it consistent across them

Julian · 2025-03-07T15:33:57Z

I left some last comments, and pre-commit is probably complaining about the missing end-of-line at the end of the file (or other stuff, I didn't click through), but after this I think feel free to merge whenever!

OptimumCode · 2025-03-07T15:50:41Z

pre-commit is probably complaining about the missing end-of-line at the end of the file

It actually complains about missing pre-commit config) it is in another branch. I will merge them together to make sure everything is formatted correctly

… arm64

…well

for more information, see https://pre-commit.ci

This reverts commit f976eb6.

This reverts commit 29e624a.

OptimumCode · 2025-03-08T07:53:11Z

@Julian I will merge this PR into prep-ci branch and will apply changes to make zizmor happy in a separate PR

…ication (#1) * Add pre-commit config file * Add dependabot config * Add license * Add build workflow * Add automerge for dependabot PRs * Add missing bracket * Correct quotes * Remove dotnet condition * Remove --from when installing bowtie * Use correct package to install bowtie * Version is always set in the matrix * Try extract version from latest and current version and prepare a the tag when needed * Move part of tags to build image step * Print collected versions * Execute pre-commit hook * Use raw value in jq to get rid of quotes * Trigger image build only on a push to main branch * Correct which version is used for a tag * Add tag with version to the final image * Use gh cli to create tag and release * Add missing GH_TOKEN env variable * Use personal access token to create a tag and release * Use default github token for release creation * Use bowtie action * Add workflow to rebuild all available old version and the latest one (#2) This commit also adds `--platform=$BUILDPLATFORM` into Dockerfile to enable multi-arch builds without QEMU --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add zizmor pre-commit hook and update workflow to comply with zizmor rules (#3) * Cleanup workflows * Use user.login to identify whether the PR is from dependabot * Remove manual tag creation. Use release API to create a tag --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

OptimumCode requested a review from Julian March 1, 2025 12:14

Julian reviewed Mar 5, 2025

View reviewed changes

.github/workflows/build-all.yml Outdated Show resolved Hide resolved

Julian reviewed Mar 5, 2025

View reviewed changes

OptimumCode commented Mar 5, 2025

View reviewed changes

OptimumCode mentioned this pull request Mar 6, 2025

Add common and harness specific configuration for deployment and publication #1

Merged

OptimumCode requested a review from Julian March 7, 2025 15:25

Julian reviewed Mar 7, 2025

View reviewed changes

Oleg added 12 commits March 7, 2025 17:26

Add workflow to rebuild all available old version and the latest one

29feca1

Remove qemu installation - java/kotlin can be compiled without it for…

6d65bf9

… arm64

Use bowtie action instead of manual installation

09e73f4

Tmp cheanges to test workflow

92c2d2a

Add GH_TOKEN when reading tags

58e11f8

Pass a parameter with build platform to builder

ce68718

Try target platform as input for builder image

192c26b

Go back to BUILDPLATFORM as input to builder image. Add it as ARG as …

dd520bc

…well

Return back the qemu installation

8eecef3

Use only git to collect tags

d280a35

Fetch full history

36343bd

Remove tmp changes for testing

658ef51

OptimumCode changed the base branch from bowtie-main to prep-ci March 7, 2025 16:26

OptimumCode force-pushed the rebuild-all branch from 5229f18 to 658ef51 Compare March 7, 2025 16:26

pre-commit-ci bot and others added 7 commits March 7, 2025 16:27

[pre-commit.ci] auto fixes from pre-commit.com hooks

cb11da9

for more information, see https://pre-commit.ci

Try using universal base image for build

20da7ec

Use image with jdk 21. There is no ubi image for jdk 17

c63fe64

Disable qemu installation

9da2dc1

Try docker out of curiosity

5712070

Remove docker actions

22079ca

Try using platforms instead of archs

d40f100

Oleg added 11 commits March 7, 2025 19:13

Try raw podman command

3807527

Try passing BUILDPLATFORM to builder image

cddf842

Remove step with podman

c392a46

Retrun back the original builder image

ff2a2cc

Retrun arhs back to see if built will fail

61038f3

Make changes in files to invalidate docker cache

f976eb6

Revert "Make changes in files to invalidate docker cache"

c7d5cdf

This reverts commit f976eb6.

Remove platform parameter from docker image

29e624a

Revert "Remove platform parameter from docker image"

ce12a04

This reverts commit 29e624a.

Disable qemu in build all workflow

cac3b88

Remove qemu installation from workflows

a674dc3

OptimumCode merged commit 401d87b into prep-ci Mar 8, 2025
9 checks passed

OptimumCode deleted the rebuild-all branch March 8, 2025 07:54

OptimumCode mentioned this pull request Mar 8, 2025

Add zizmor pre-commit hook and update workflow to comply with zizmor rules #3

Merged

Add workflow to rebuild all available old version and the latest one #2

Add workflow to rebuild all available old version and the latest one #2

Uh oh!

Conversation

OptimumCode commented Mar 1, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OptimumCode Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OptimumCode Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OptimumCode Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Julian commented Mar 7, 2025

Uh oh!

OptimumCode commented Mar 7, 2025

Uh oh!

OptimumCode commented Mar 8, 2025

Uh oh!

Uh oh!

Uh oh!

OptimumCode Mar 5, 2025 •

edited

Loading

OptimumCode Mar 5, 2025 •

edited

Loading

OptimumCode Mar 7, 2025 •

edited

Loading