CI/CD status

[prabhu]

Discussion to track our build status, downtime etc.

self-hosted agents

We have two Mac Minis (64GB RAM each) and one Mini PC (32 GB RAM) powering cdxgen.

Todo

• Move to BDC1 (Upcoming Bude data centre) and invest in a physical lock.
• Publish OBOMs for the compute units.
• Harden the OS a bit based on the OBOM and lynis scans.
• Investigate multi-arch builds with docker rootless

Learnings

• Docker rootless and containerd-based builds have low reliability. Running docker as root magically makes things better. This is because most official docker steps, including the build-and-push action, expect sole root access.
• Macs continue to suck in headless mode. After the recent software update, tailscale got disabled, requiring physical access to toggle the switch. Why?
• Docker builds from an ARM64 host machine usually work only on sunny days. I don't think QEMU is that well tested on ARM64. So we need AMD64 machines with working KVM to reliably build for ARM64.
• Multi-arch builds with Docker are possible only due to mere luck. Ubuntu Server, images such as tonistiigi/binfmt, multiarch/qemu-user-static, moby/buildkit:buildx-stable-1 wrap a very old version of QEMU 8.2.2 with several known bugs. Upgrading the entire stack to use QEMU 10 is not for the faint-hearted. Even with QEMU 10, you are met with regression bugs with the only solution being to downgrade to QEMU v8.1.5!

[prabhu]

Thursday 3 Apr BST 23:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Created and released blint 2.4.1 to fix. Re-ran the container builds to make the master branch green. Total DevOps time spent: 4 hours. This is, however, non-chargeable since I am also responsible for the bug originating from the blint 2.4.0 release.

[prabhu]

Tuesday 8 Apr BST 11:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Have created this upstream ticket to investigate the problem.

[prabhu]

pip install has started working by simply re-running the workflows a few times. If I come across any RCA from the PyPI team, I will link it here.

[prabhu]

Thursday 10 Apr BST 10:20 (UTC+1)

Noticed timeouts in the upload base images workflow with the macos-hosted agent. Suspected the issue to be disk space, so re-ran the workflow after doing docker system prune -a.

[prabhu]

docker system prune did the trick. A couple of failures with sle .NET builds, which is like the one we have seen before.

#5 29.73 To install missing framework, download:
#5 29.73 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

My attempts to build sle dotnet images without performing zypper refresh and update did not work.

148.8 Running post-transaction scripts [....done]
150.4 8.0.407 [/usr/share/dotnet/sdk]
151.7
156.1 Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Value cannot be null or an empty string. (Parameter 'value').
156.1 Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Value cannot be null or an empty string. (Parameter 'value').
156.1 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maccatalyst doesn't exist.
156.1 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net6 doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.current doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.sdk.tvos: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.sdk.tvos doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net7 doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net7 doesn't exist.
156.4 Failed to update the advertising manifest microsoft.net.sdk.maui: Value cannot be null or an empty string. (Parameter 'value').
156.4 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maui doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.current doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.sdk.android: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.sdk.android doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.macos: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.macos doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.aspire: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.aspire doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.sdk.ios doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net6 doesn't exist.
159.3 Installing pack Microsoft.Android.Sdk.Linux version 34.0.43...
159.5 Workload installation failed. Rolling back installed packs...
159.5 Rolling back pack Microsoft.Android.Sdk.Linux installation...
159.5 Workload installation failed: Value cannot be null or an empty string. (Parameter 'value')

SLE issues usually resolve themselves in a day or two. Let's check again tomorrow without wasting any more DevOps time.

[prabhu]

SLE dotnet 8 and 9 images are still broken. The error message lists two conflicting versions, 8.0.14 and 8.0.15, indicating upstream issues. Raised a query here.

#5 35.12 8.0.408 [/usr/share/dotnet/sdk]
#5 35.12 You must install or update .NET to run this application.
#5 35.12 
#5 35.12 App: /usr/share/dotnet/sdk/8.0.408/dotnet.dll
#5 35.12 Architecture: x64
#5 35.12 Framework: 'Microsoft.NETCore.App', version '8.0.15' (x64)
#5 35.12 .NET location: /usr/share/dotnet/
#5 35.12 
#5 35.12 The following frameworks were found:
#5 35.12   8.0.14 at [/usr/share/dotnet/shared/Microsoft.NETCore.App]
#5 35.12 
#5 35.12 Learn more:
#5 35.12 https://aka.ms/dotnet/app-launch-failed
#5 35.12 
#5 35.12 To install missing framework, download:
#5 35.12 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

Have removed the dotnet workload install commands with this commit, since it's been days. Recommendation to deprecate the SLE dotnet images in favour of the Debian ones, which have been stable as well as support multiple architectures.

[prabhu]

To vent my frustration with the reliability of SLE Docker builds, below is an error from failed Python 3.11 builds. I have to now wait a while for all other builds to finish before restarting the failed ones.

16.22 Retrieving: bzip2-1.0.8-150400.1.122.aarch64 (SLE_BCI) (1/46),  45.8 KiB    
16.27 Retrieving: bzip2-1.0.8-150400.1.122.aarch64.rpm [....error]
17.50 Download (curl) error for 'https://public-dl.suse.com/SUSE/Products/SLE-BCI/15-SP6/aarch64/product/aarch64/bzip2-1.0.8-150400.1.122.aarch64.rpm':
17.50 Error code: Curl error 35
17.50 Error message: OpenSSL/3.1.4: error:0A000438:SSL routines::tlsv1 alert internal error
17.50 
17.50 Abort, retry, ignore? [a/r/i/...? shows all options] (a): a
17.52 Problem occurred during or after installation or removal of packages:
17.52 Installation has been aborted as directed.
17.52 Please see the above error message for a hint.
------
Dockerfile.python311:21

[prabhu]

Wednesday 16 Apr BST 11:00 (UTC+1)

Power is down, so the self-hosted agents are unavailable.

(Power must be back in six hours. The good news is that we have an EcoFlow 1KWh battery backup. The bad news is that I am travelling in the opposite direction, so I couldn't physically deliver the battery to the co-location centre. Note to self: we need to distribute compute units across locations.)

[prabhu]

Power is back at 4 p.m. Have to revisit my plans to scale entirely with in-house computes.

[prabhu]

Thursday 17 Apr BST 17:00 (UTC+1)

Noticed that the "Upload base images" had errors. Apparently, this is a known issue with colima which requires a workaround after every macOS update.

brew uninstall colima qemu lima
rm -rf ~/.colima
brew install colima
brew services restart colima

[prabhu]

For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts.

[prabhu]

Could have tried this simpler workaround too.

rm -rf ~/.colima/_lima/_networks/user-v2

[malice00]

FYI: keychain can be interacted with from CLI. Not sure what you need, but I use it in CI all the time!

…

On Thu, Apr 17, 2025, 18:41 prabhu ***@***.***> wrote: For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIPPXZPRAUMWKWXEBMT2Z7KUZAVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBWHE4DQOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[prabhu]

Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues.

[malice00]

I believe the correct command would be `keychain unlock`. It needs credentials, but these can be given on the cli.

…

On Thu, Apr 17, 2025, 18:59 prabhu ***@***.***> wrote: Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIJKXQIQR45ZMJ3ELWL2Z7MV3AVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBXGAYDKMI> . You are receiving this because you commented.Message ID: ***@***.***>

[prabhu]

Will keep in mind. So far we are ok with colima.

[prabhu]

Saturday 26 Apr BST 17:00 (UTC+1)

swift-actions/setup-swift step has stopped working due to this issue. This step is quite flaky. Need to prioritise #1711 in case this GPG issue is not resolved in a day or two.

[prabhu]

Our workflows worked after multiple restarts. It is frustrating that setup-swift is not a first-party step similar to setup-java, owned and distributed by GitHub. Relying on the free Ubuntu keyserver for verifying the signature of Swift keys can only scale so much. Open-source devs can also do only so much free support for the world.

[prabhu]

Thursday 1 May BST 15:00 (UTC+1)

Happy Labour Day (International Workers' Day)!

Today a brand new server asin joined our infra. It is currently at a highly-secure and undisclosed location (under my desk 10 minutes from GCHQ in Bude).

I would love to share an HBOM and OBOM. Sadly, there are no HBOM tools I could find. (cc: @malice00). So, just some output from a couple of commands and a decent OBOM.

Output from lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   48
  On-line CPU(s) list:    0-47
Vendor ID:                AuthenticAMD
  Model name:             AMD Ryzen Threadripper 7960X 24-Cores
    CPU family:           25
    Model:                24
    Thread(s) per core:   2
    Core(s) per socket:   24
    Socket(s):            1
    Stepping:             1
    CPU(s) scaling MHz:   12%
    CPU max MHz:          5665.0000
    CPU min MHz:          545.0000
    BogoMIPS:             8386.78
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulq
                          dq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwait
                          x cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveop
                          t xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassis
                          ts pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization features:
  Virtualization:         AMD-V
Caches (sum of all):
  L1d:                    768 KiB (24 instances)
  L1i:                    768 KiB (24 instances)
  L2:                     24 MiB (24 instances)
  L3:                     128 MiB (4 instances)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-47
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Mitigation; Safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

lshw.txt

OBOM

asin-obom.cdx.json

Bugs identified

• [obom] track third-party PPA in ubuntu/debian #1767

[prabhu]

Thursday 8 May BST 00:00 (UTC+1)

Self-hosted agents ran out of disk space. It turned out, due to some known bugs in cdxgen, the /tmp folder was full of docker-images-*, atom-usages-*, and atom-reachables-* directories. Manually cleared the /tmp folder for now. Don't know which one is easier - fixing cdxgen to properly clear the temp directories or simply periodically checking the self-hosted agents.

[malice00]

Idea: setup a cron to clean temp dir every so often

[prabhu]

Added this line in the repotests workflow. BTW, I had an old Intel MacBook Pro with 32GB RAM lying around. I removed macOS and installed Ubuntu Server on this. This server is called simran. It ran the repotests successfully last night.

Meanwhile, I upgraded the asin server to 256GB RAM. Initially, one 64GB is reserved for cdxgen, while the rest can be used for any experiments, testing, etc. Thank you PC specialist for the generous discount on the upgrade RAM pricing!

[prabhu]

Ok, looks like it wasn't that successful. Taking a look.

[prabhu]

Adding || true to the various pip install commands did the trick. I think this hybrid setup of self-hosted and github-hosted will make our workflows and tests more generic and portable. Something other projects could try and copy to reduce our reliance of certain large enterprises.

[prabhu]

Thursday 8 May BST 13:00 (UTC+1)

Despite the new self-hosted machines, our CI usage is going up well beyond the 50K minutes that GitHub offers for Enterprise users. We are now beginning to consume minutes from other OWASP orgs as well (CycloneDX is a sub-org under OWASP).

Thanks to the excellent work from @malice00, we have a new matrix-based workflow for building base images. However, building Ruby 3.4.3 steps still takes too much time, so we are at risk of exceeding 50K minutes this month as well.

I think the next step is to look into multi-stage builds and layer caching, so that $HOME/.rbenv gets mounted and copied over from a cache for various architectures. Other ideas to reduce usage are welcome.

[prabhu]

Wednesday 14 May BST 12:00 (UTC+1)

New Raspberry Pi 5 build agent. If it works well for our needs, we can set up a mini-cluster for testing and get it to a stage where users could run cdx at industrial scale and reduce cloud costs.

OBOM

pi5-obom.cdx.json

[prabhu]

Thursday 15 May BST 12:00 (UTC+1)

Currently, fighting the CI due to multiple failures (all of them with github-hosted).

Installing dotnet on debian isn't reliable

#10 [linux/arm64 2/5] COPY ci/base-images/debian/install.sh /tmp/
#10 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://mcr.microsoft.com/v2/dotnet/sdk/blobs/sha256:1df4174e1641c24bac819a993f5ed3d2ae6facbada6f7a943bed15d22e64851b: 503 Service Unavailable

Node 20 builds continue to fail for mysterious reasons

102.7 update-alternatives: using /usr/bin/g++-13 to provide /usr/bin/g++ (g++) in auto mode
112.6 npm error code EEXIST
112.6 npm error path /usr/lib64/node_modules/npm20/bin/npm
112.6 npm error EEXIST: file already exists
112.6 npm error File exists: /usr/lib64/node_modules/npm20/bin/npm
112.6 npm error Remove the existing file and try again, or run npm
112.6 npm error with --force to overwrite files recklessly.
112.6 npm error A complete log of this run can be found in: /root/.npm/_logs/2025-05-15T11_00_43_489Z-debug-0.log

I am also seeing random qemu crashes.

corepack command not found errors

A number of java builds are failing, despite us installing corepack explicitly!

0.127 /bin/sh: corepack: command not found
------
Dockerfile.java17-slim:20
--------------------
  19 |     
  20 | >>> RUN cd /opt/cdxgen && corepack enable && corepack pnpm install --config.strict-dep-builds=true --prod --package-import-method copy --frozen-lockfile && corepack pnpm cache delete \
  21 | >>>     && mkdir -p /opt/cdxgen-node-cache \
  22 | >>>     && node /opt/cdxgen/bin/cdxgen.js --help \
  23 | >>>     && rm -rf /root/.cache/node \
  24 | >>>     && chmod a-w -R /opt
  25 |     WORKDIR /app

The rolling image is looking for corepack-default. WHY?

> [linux/arm64 3/4] RUN cd /opt/cdxgen && corepack enable && corepack pnpm install --config.strict-dep-builds=true --prod --package-import-method copy --frozen-lockfile && corepack pnpm cache delete     && mkdir -p /opt/cdxgen-node-cache     && node /opt/cdxgen/bin/cdxgen.js --help     && pip install --upgrade --no-cache-dir blint atom-tools --target /opt/pypi     && rm -rf /root/.cache/node     && chmod a-w -R /opt:
0.183 /usr/bin/corepack-default: No such file or directory

opensuse continues to do opensuse things

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: registry.opensuse.org/opensuse/bci/python:3.10: failed to resolve source metadata for registry.opensuse.org/opensuse/bci/python:3.10: failed to do request: Head "https://registry.opensuse.org/v2/opensuse/bci/python/manifests/3.10": dial tcp 195.135.223.221:443: i/o timeout

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: registry.opensuse.org/opensuse/bci/python:3.9: failed to resolve source metadata for registry.opensuse.org/opensuse/bci/python:3.9: failed to do request: Head "https://registry.opensuse.org/v2/opensuse/bci/python/manifests/3.9": dial tcp 195.135.223.221:443: i/o timeout

[prabhu]

Like most things with software and IT, a mere restart is making all these builds pass again. Guess we will never know the reason. Probably there is a way to make a job retry once if the previous job failed.

[malice00]

The node image was failing because we were trying to install npm although it was already there. Fixed in #1801

[prabhu]

Very good catch!

[prabhu]

Wednesday 21 May BST 10:00 (UTC+1)

Multiple build failures since macos-hosted was out of disk space. Retrying the builds after running the command docker system prune -a

Noticed another issue with attaching sboms to the nightly images.

node bin/cdxgen.js -t docker -o sbom-oci-base-image.cdx.json ghcr.io/cyclonedx/debian-dotnet8:nightly
node bin/verify.js -i sbom-oci-base-image.cdx.json --public-key contrib/bom-signer/public.key

Retrying with sha256:6dd7e99d793d70ff2ce73260d0cdd66ed32cbaebf468163457cfdfcbe5da1d89
Manifest file /home/runner/work/_temp/cdxgen-sboms/docker-images-Ees8wm/manifest.json was not found after export at /home/runner/work/_temp/cdxgen-sboms/docker-images-Ees8wm
OCI BOM generation has failed due to problems with exporting the image ghcr.io/cyclonedx/debian-dotnet8:nightly.
sbom-oci-base-image.cdx.json is invalid!

Possible bug in cdxgen?

[prabhu]

@malice00 I think we need docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes and the docker/setup-buildx-action@v3 for macos-hosted as well. Also I think the qemu workaround step needs to be above the buildx step to make the buildx agents use the correct qemu static binaries. Finally we need load: true in the docker/build-push-action@v5 for cdxgen sbom generation to work correctly.

[malice00]

load doesn't work on multi-platform builds, but I've been working on a solution in a branch. Hope to finish it this weekend.

[malice00]

Also, about the disk space: A lot of that is probably our build caches. We should remove those periodically -- I'll see if I can add that to the branch as well!

[prabhu]

Please rebase from master since have merged a change to split the ruby build per architecture.

[prabhu]

colima on prabhus-mini seems to be running with just 100GB disk. The template, however, shows 500GB indicating this VM probably got created with default settings before the template was set correctly.

prabhu@colima:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        96G   85G   12G  89% /
tmpfs            20G     0   20G   0% /dev/shm
tmpfs           7.9G  972K  7.9G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs         56K  2.6K   54K   5% /sys/firmware/efi/efivars
vz-rosetta      927G  589G  339G  64% /mnt/lima-rosetta
/dev/vda16      891M   32M  797M   4% /boot
/dev/vda15       98M  6.4M   92M   7% /boot/efi
tmpfs           4.0G  4.0K  4.0G   1% /run/user/501
/dev/vdb         47M   47M     0 100% /mnt/lima-cidata
:/Users/prabhu  927G  589G  339G  64% /Users/prabhu
:/tmp/colima    927G  589G  339G  64% /tmp/colima

[prabhu]

Thursday 22 May BST 10:00 (UTC+1)

load: true doesn't support multi-arch builds in build-push-action. The error was ERROR: Docker exporter does not currently support exporting manifest lists. When a built image is not loaded, attempting to generate an SBOM is resulting in cdxgen downloading the image again from the registry!

Related: #1606

[prabhu]

Friday 23 May BST 09:00 (UTC+1)

Happy Friday! mini-dev-1 (that gray Beelink ESR8 one since we don't have the HBOM!) required some servicing today. When I set up this machine a while ago, my thinking was to set enough UMA Frame Buffer size so that we could invoke cdx1 and other mini ML models with llama.cpp as part of CI pipelines. For some reason, I went with 16GB (out of 32GB RAM). This not only reduced the amount of memory available for CI, but also reduced the system performance a bit as per AMD support document.

Changing this setting to auto required physically rebooting to BIOS, etc. With auto, Ubuntu Server can now see and access all 32GB RAM! However, as you can imagine, anything with the word auto never really works on Linux. So, llama.cpp and others can no longer access the GPU memory!

The plan is to run this as a CPU-only build agent for a bit more, till we reach a point where we need some GPU.

[prabhu]

Note to self: can a HBOM tool capture the BIOS settings?

[prabhu]

Friday 23 May BST 17:00 (UTC+1)

After a lot of testing and research, our macos-hosted agents can now support faster multi-arch builds with Docker (via colima). The secret lies in how the VMs are created. We need qemu only for AMD64, while ARM64 could be of vz type. This works quite well since QEMU is highly optimised for AMD64 virtualisation.

colima start -c 6 -d 250 -m 16 -t qemu -r docker --dns 1.1.1.1 --dns 1.0.0.1 --mount-type sshfs -a x86_64 --profile colima-amd64
colima start -c 6 -d 250 -m 16 -t vz -r docker --dns 1.1.1.1 --dns 1.0.0.1 --mount-type virtiofs -a aarch64 --profile colima-aarch64

docker buildx create --use --name multi-builder colima-amd64
docker buildx create --append --name multi-builder colima-aarch64

There is no need for any binfmt hacks. The workflows that use macos-hosted agents do not need setup-qemu and setup-buildx steps.

[prabhu]

Note for self:

If the default buildx gets changed or colima gets restarted, we need to make the multi-builder as the default in the macos-hosted agents. Otherwise, the multi-arch builds will fail due to docker driver limitations.

docker buildx use multi-builder

docker buildx ls
NAME/NODE            DRIVER/ENDPOINT      STATUS    BUILDKIT   PLATFORMS
multi-builder*       docker-container
 \_ multi-builder0    \_ colima-amd64     running   v0.21.1    linux/amd64 (+3), linux/386
 \_ multi-builder1    \_ colima-aarch64   running   v0.21.1    linux/amd64 (+2), linux/arm64, linux/386
colima-aarch64       docker
 \_ colima-aarch64    \_ colima-aarch64   running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386
colima-amd64         docker
 \_ colima-amd64      \_ colima-amd64     running   v0.17.3    linux/amd64 (+3), linux/386
default                                   error

[malice00]

In case colima start gives an error like:

WARN[0000] Failed to resolve the guest agent binary for "x86_64" error="guest agent binary could not be found for Linux-x86_64 (Hint: try installing lima-additional-guestagents package)"

install that package with brew:

brew install lima-additional-guestagents

and start the VM again:

colima start <vm that had issues>

[prabhu]

Monday 2 June BST 13:00 (UTC+1)

Multiple build failures with the macos-hosted agents. Suspecting the issue could be due to misconfigured buildx.

[prabhu]

docker buildx ls
NAME/NODE            DRIVER/ENDPOINT      STATUS    BUILDKIT   PLATFORMS
multi-builder        docker-container
 \_ multi-builder0    \_ colima-amd64     error
 \_ multi-builder1    \_ colima-aarch64   error
colima*              docker
 \_ colima            \_ colima           running   v0.21.0    linux/amd64 (+2), linux/arm64, linux/386
colima-aarch64                            error
colima-amd64                              error
default                                   error

Failed to get status for multi-builder (multi-builder0): Cannot connect to the Docker daemon at unix:///Users/prabhu/.colima/amd64/docker.sock. Is the docker daemon running?
Failed to get status for multi-builder (multi-builder1): Cannot connect to the Docker daemon at unix:///Users/prabhu/.colima/aarch64/docker.sock. Is the docker daemon running?
Cannot load builder colima-aarch64: Cannot connect to the Docker daemon at unix:///Users/prabhu/.colima/aarch64/docker.sock. Is the docker daemon running?
Cannot load builder colima-amd64: Cannot connect to the Docker daemon at unix:///Users/prabhu/.colima/amd64/docker.sock. Is the docker daemon running?
Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[prabhu]

To fix the issue, had to start colima-amd64 and colima-aarch64 and kill colima.

colima start -c 6 -d 250 -m 16 -t qemu -r docker --dns 1.1.1.1 --dns 1.0.0.1 --mount-type sshfs -a x86_64 --profile colima-amd64
colima start -c 6 -d 250 -m 16 -t vz -r docker --dns 1.1.1.1 --dns 1.0.0.1 --mount-type virtiofs -a aarch64 --profile colima-aarch64

colima stop colima

Make buildx use multi-builder.

docker buildx use multi-builder

Check for the default buildx instance.

docker buildx ls
NAME/NODE            DRIVER/ENDPOINT      STATUS    BUILDKIT   PLATFORMS
multi-builder*       docker-container
 \_ multi-builder0    \_ colima-amd64     running   v0.21.1    linux/amd64 (+3), linux/386
 \_ multi-builder1    \_ colima-aarch64   running   v0.21.1    linux/amd64 (+2), linux/arm64, linux/386
colima-aarch64       docker
 \_ colima-aarch64    \_ colima-aarch64   running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386
colima-amd64         docker
 \_ colima-amd64      \_ colima-amd64     running   v0.17.3    linux/amd64 (+3), linux/386
default                                   error

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[malice00]

Builds where still failing. After restart it is probably a good idea to prune the cache and start fresh:

docker buildx prune -a

[prabhu]

Tuesday 3 June BST 11:00 (UTC+1)

Opensuse reliability issues. These errors often disappear with a manual restart, but all these take up time.

#12 [linux/amd64 3/4] RUN zypper --non-interactive install --allow-downgrade -l --no-recommends readline-devel clang15 llvm15 llvm15-devel libjpeg62-devel libmariadb-devel         postgresql16-devel postgresql16-server-devel libopenssl-devel libopenblas_pthreads-devel lapacke-devel graphviz-devel     && cd /opt/cdxgen && corepack enable && corepack pnpm install --config.strict-dep-builds=true --prod --package-import-method copy --frozen-lockfile && corepack pnpm cache delete     && mkdir -p /opt/cdxgen-node-cache     && node /opt/cdxgen/bin/cdxgen.js --help     && pip install --upgrade --no-cache-dir atom-tools --target /opt/pypi     && chmod a-w -R /opt     && rm -rf /root/.cache/node     && zypper clean -a
#12 98.83 Warning: Failed to parse/receive mirror information for URL: http://download.opensuse.org/tumbleweed/repo/non-oss/?mirrorlist
#12 98.83 History:
#12 98.83  - Location 'http://download.opensuse.org/tumbleweed/repo/non-oss/?mirrorlist' is temporarily unaccessible.
#12 98.83    Location 'http://download.opensuse.org/tumbleweed/repo/non-oss/?mirrorlist' is temporarily unaccessible. Curl error (22)
#12 105.1 Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata [.error]
#12 106.4 Repository 'openSUSE-Tumbleweed-Non-Oss' is invalid.
#12 106.4 [repo-non-oss|http://download.opensuse.org/tumbleweed/repo/non-oss/] Failed to retrieve new repository metadata.
#12 106.4 History:
#12 106.4  - [|] Error trying to read from 'http://download.opensuse.org/tumbleweed/repo/non-oss/'
#12 106.4  - Location 'http://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml' is temporarily unaccessible.
#12 106.4    Location 'http://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml' is temporarily unaccessible. Curl error (22)
#12 106.4 Please check if the URIs defined for this repository are pointing to a valid repository.
#12 106.4 Warning: Skipping repository 'openSUSE-Tumbleweed-Non-Oss' because of the above error.
#12 106.4 Retrieving repository 'Open H.264 Codec (openSUSE Tumbleweed)' metadata [.error]
#12 106.9 Repository 'Open H.264 Codec (openSUSE Tumbleweed)' is invalid.
#12 106.9 [repo-openh264|http://codecs.opensuse.org/openh264/openSUSE_Tumbleweed] Failed to retrieve new repository metadata.
#12 106.9 History:
#12 106.9  - [|] Error trying to read from 'http://codecs.opensuse.org/openh264/openSUSE_Tumbleweed'
#12 106.9  - Location 'http://codecs.opensuse.org/openh264/openSUSE_Tumbleweed/content' is temporarily unaccessible.
#12 106.9    Location 'http://codecs.opensuse.org/openh264/openSUSE_Tumbleweed/content' is temporarily unaccessible. Curl error (22)

#12 81.19 Package 'readline-devel' is not available in your repositories. Cannot reinstall, upgrade, or downgrade.
#12 81.19 'clang15' not found in package names. Trying capabilities.
#12 81.19 No provider of 'clang15' found.
#12 81.19 'graphviz-devel' not found in package names. Trying capabilities.
#12 81.19 No provider of 'graphviz-devel' found.
#12 81.19 'lapacke-devel' not found in package names. Trying capabilities.
#12 81.19 No provider of 'lapacke-devel' found.
#12 81.19 'libjpeg62-devel' not found in package names. Trying capabilities.
#12 81.19 No provider of 'libjpeg62-devel' found.
#12 81.19 'libmariadb-devel' not found in package names. Trying capabilities.
#12 81.19 No provider of 'libmariadb-devel' found.
#12 81.19 'libopenblas_pthreads-devel' not found in package names. Trying capabilities.
#12 81.19 No provider of 'libopenblas_pthreads-devel' found.
#12 81.19 'libopenssl-devel' not found in package names. Trying capabilities.

[prabhu]

Tuesday 17 June BST 11:00 (UTC+1)

We have lost mini-dev-1 due to a lack of disk space (and human errors). Even though it has a 1 TB disk, due to existing cdxgen bugs wrt temp directories cleanup, we lost this space overnight (after < 10 builds). There is a cron job that does docker system prune -a -f at midnight, so there is some hope it might come back on its own in a few hours. Our servers have Tailscale access to log in remotely. Unfortunately, as part of another hardening task, I stopped Tailscale in mini-dev-1, but forgot to open up SSH in the firewall, so we have lost remote access completely :( Getting physical access to this machine would require an hour's drive and is unlikely to happen before mid-day tomorrow.

Plan

• Take a deep breath.
• Move containers and containers-secure builds back to github-hosted agents. Yes, I know we are completely out of minutes yadda. Branch.
• Prioritise temp directory cleanup project. It will be super cool if cdxgen completely removes all its temp directories post the BOM generation.

[prabhu]

Oh, the irony. While containers and containers-secure are going strong with GitHub-hosted, containers-deno is failing with an out-of-space error. This is because we get only 33GB free space even after trimming the hosted agents, which is barely enough to hold two images in buildx cache and generate two SBOMs by extracting the multi-arch images.

It actually reminds us as to why we went with self-hosted agents in the first place!

[malice00]

Are you using only our own clean-up script? I added an extra one in the image builds that worked magic for some of the larger builds (eg dotnet) -- maybe this can help here as well!

[prabhu]

oh nice! worth trying that one too.

[prabhu]

Wednesday 18 June BST 12:00 (UTC+1)

mini-dev-1 arrived today (Thanks, Bernie!) for maintenance. When I executed the df -h command, the output shocked me. The root partition only had 400GB of space, with the remaining 600GB under another partition called /scratch. I must have set it up this way while installing Ubuntu Server, but completely forgot to configure GitHub Actions and Docker to use /scratch as the mount point for storage.

Thanks to ChatGPT's help, I managed to delete /scratch and resize the root /.

sudo lvdisplay
sudo umount /dev/ubuntu-vg/lv-to-delete
# Manually remove from /etc/fstab
sudo lvremove /dev/ubuntu-vg/lv-to-delete
sudo lvextend -l +100%FREE /dev/ubuntu-vg/target-lv
sudo btrfs filesystem resize max /

With tailscale back on,

sudo df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              3.1G   10M  3.1G   1% /run
efivarfs                           128K   43K   81K  35% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv  929G  296G  630G  32% /
tmpfs                               16G   92K   16G   1% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               16G     0   16G   0% /run/qemu
/dev/nvme0n1p2                     2.0G  317M  1.5G  18% /boot
/dev/nvme0n1p1                     1.1G  6.2M  1.1G   1% /boot/efi
tmpfs                              3.1G   16K  3.1G   1% /run/user/1000
overlay                            929G  296G  630G  32% /var/lib/docker/rootfs/overlayfs/35fd7f7783db3920697d529dce17daf4f2563262c9bef49c6c5c53bbcf0faa75
overlay                            929G  296G  630G  32% /var/lib/docker/rootfs/overlayfs/45a05b792f98bfda809375bde04d5b2425ac35a2fdaf0fc659c7034692baed1e
overlay                            929G  296G  630G  32% /var/lib/docker/rootfs/overlayfs/a33f86060ad9ba76964aa1efe8d96041d86e5bbcaeebf95a7f5de37d400f0480

What about OBOM

We generate OBOM for all our servers. While, the document has details about the packages, services, mount points, etc, it currently lacks information about partitions.

For interested people, our osquery queries are here. We need to add some queries to read the device_partitions table based on this document.

Bonus points, if the annotation section can be enhanced to describe the partitions, networks, and services in plain text. It currently reads as shown:

This Operations Bill-of-Materials (OBOM) document was created on Wednesday, June 18, 2025 with cdxgen. The lifecycles phases represented are: pre-build and operations. The document describes an operating system named 'Ubuntu' with the build name '24.04.2 LTS (Noble Numbat)'. There are 2743 components. In addition, there are 1193 applications installed on the system.

mini-dev-1 obom

[prabhu]

Tuesday 22nd July BST 14:00 (UTC+1)

Thanks to some brilliant work from @malice00 and @bandhan-majumder we have reached the stage where our in-house compute is fully automated requiring limited maintenance. Roland has also kindly setup a local Nexus server to improve build performance and resilience.

All of our release artefacts and container images (including multi-arch) include a detailed SBOM. The next step is to look into generating the licence notices for all release artefacts.

[malice00]

Sunday 3rd August CEST 03:30 (UTC+2)

After changing our image builds to make use of our local Nexus, I finally figured out how to get Docker mirroring to work with buildkit! I created a toml configuration for buildkit:

# We must define all repositories that we want mirrored.

[registry."docker.io"]
  mirrors = ["100.73.146.80:5000"]

[registry."ghcr.io"]
  mirrors = ["100.73.146.80:5000"]

[registry."mcr.microsoft.com"]
  mirrors = ["100.73.146.80:5000"]

[registry."registry.opensuse.org"]
  mirrors = ["100.73.146.80:5000"]

[registry."registry.suse.com"]
  mirrors = ["100.73.146.80:5000"]

# We need to inform buildkit that our mirror only accepts HTTP.

[registry."100.73.146.80:5000"]
  http = true

Then, I removed the existing multi-builder and recreated it using the toml configuration:

docker rm multi-builder
docker buildx create --use --name multi-builder colima-amd64 --buildkitd-config /Users/prabhu/buildkit/buildkitd.toml
docker buildx create --append --name multi-builder colima-aarch64 --buildkitd-config /Users/prabhu/buildkit/buildkitd.toml

I had to use the IP address of our server here, because for some reason the colima instances that are running our docker engines can't consistently resolve names in our network.

Things to remember

• Any new registries that we might start using should be added -- since the config is read on startup, this means recreating the builder again
• Not sure if the config is necessary in both calls, but since it's working, I guess at least it doesn't hurt that it's there twice -- will investigate further...

[prabhu]

This is some incredible work that deserves a technical blog and an OWASP talk on its own! I really wish other open-source projects seek inspiration from this and set up their own infra with caching and mirroring to improve security and performance!