Update ffmpeg and change static build provider #64

matthewjmarangoni · 2025-09-10T15:27:31Z

Please merge PR #60 prior to merging this PR. The basis of this PR is PR #60 and once merged this diff should clear up.

This PR changes the provider for static ffmpeg builds to a Github repository source linked from https://ffmpeg.org/download.html.

The associated checksum work is updated as necessitated by the change from MD5 to SHA256.

The build for ffmpeg is set to the latest version and will continually update as new versions are released. Updating without some form of pinning may or may not be an acceptable approach here.

The old provider could be added as a backup. Issues could arise in cases where the fallback is used as builds between providers may not be versioned 1:1 (not in a strict reproducibility sense).

Note

Replaces FFmpeg static builds with BtbN GitHub releases using SHA256 verification and adds platform-scoped BuildKit caches across Dockerfiles for faster, reproducible builds.

FFmpeg:
- Add dedicated ffmpeg-downloader stage (Ubuntu 22.04) that fetches from BtbN/FFmpeg-Builds with SHA256 verification and arch selection; copy ffmpeg/ffprobe into final images.
- Remove old inline downloads and MD5 checks in images/chromium-headful/Dockerfile and images/chromium-headless/image/Dockerfile.
Build performance (BuildKit caching):
- Introduce platform-aware cache prefix CACHEIDPREFIX using TARGETOS/TARGETARCH.
- Add cache mounts for apt, npm, and go in chromium-headful, chromium-headless, client, xorg-deps, and xf86-input-neko Dockerfiles.
Misc Dockerfile tweaks:
- Standardize ARG TARGETOS/TARGETARCH across stages; keep downloaded APT packages and use locked cache sharing; minor cleanup (e.g., switch global npm installs to cached).

^{Written by Cursor Bugbot for commit 2f19ef4. This will update automatically on new commits. Configure here.}

TL;DR

Switched the static ffmpeg build provider to a more reliable, official source and significantly optimized Docker image builds across the project by leveraging BuildKit caching.

Why we made these changes

The previous ffmpeg provider was unreliable, causing potential build failures. The new provider is linked directly from the official ffmpeg.org website, ensuring a trusted and up-to-date source. Additionally, our Docker builds were inefficient; implementing widespread BuildKit caching dramatically speeds up build times and reduces final image sizes by avoiding redundant package downloads.

What changed?

FFmpeg Provider:
- Changed the download source for static ffmpeg builds to the BtbN/FFmpeg-Builds GitHub repository.
- Updated the checksum verification from MD5 to SHA256 to match the new provider.
- Pinned the build to the latest release for continuous updates.
Build Performance & Image Size:
- Optimized all major Dockerfiles (chromium-headful, chromium-headless, xorg-deps, etc.) to use BuildKit cache mounts for apt, npm, and go package management.
- Consistently used --no-install-recommends during apt-get install across all relevant Dockerfiles to create leaner final images.
Configuration:
- Added .mise.toml to the project's .gitignore to ignore local mise tool version configurations.

Validation

Verify that all Docker images build successfully.
Confirm that ffmpeg is correctly installed and functional inside the final images.
Check that subsequent builds are noticeably faster due to the new caching mechanisms.

^{Description generated by Mesa. Update settings}

mesa-dot-dev

Performed full review of 330685e...031d166

Tip

⚡ Quick Actions

This review was generated by Mesa.

Actions:

Configure your agents

Slash Commands:

/review - Request a full code review
/review latest - Review only changes since the last review
/describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
/help - Get help with Mesa commands and configuration options

^{6 files reviewed | 4 comments | Review on Mesa | Edit Reviewer Settings}

images/chromium-headful/Dockerfile

images/chromium-headless/image/Dockerfile

matthewjmarangoni · 2025-09-10T18:03:44Z

While purported behavior couldn't be replicated the base principle seemed fair and the usage of sed was not required. A commit has been made to address the suggested potential by substituting the usage of sed with a fixed pattern grep.

mesa-dot-dev

Performed full review of 2d73dc8...435b400

Analysis

FFmpeg Download Script Fragility: The shell script using sed and grep to manipulate checksum files is brittle and could fail silently if the upstream format changes. Error handling is incomplete for unexpected file formats.
Reproducibility Risk: Using "latest" FFmpeg releases without version pinning compromises build reproducibility over time. Consider if this trade-off is acceptable for your use case.
Fixed Temporary File Names: The script uses fixed temporary file names that could potentially cause conflicts in concurrent build scenarios, despite cache mount mitigations.
Future Maintenance Concerns: The FFmpeg download logic may need future hardening as the upstream repository evolves, potentially increasing maintenance costs.

Tip

⚡ Quick Actions

This review was generated by Mesa.

Actions:

Configure your agents

Slash Commands:

/review - Request a full code review
/review latest - Review only changes since the last review
/describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
/help - Get help with Mesa commands and configuration options

^{6 files reviewed | 0 comments | Review on Mesa | Edit Reviewer Settings}

Cache packages, remove unused packages, and prevent installation of apt recommended packages.

mesa-dot-dev · 2025-10-31T18:12:38Z

Mesa Description

Please merge PR #60 prior to merging this PR. The basis of this PR is PR #60 and once merged this diff should clear up.

This PR changes the provider for static ffmpeg builds to a Github repository source linked from https://ffmpeg.org/download.html.

The associated checksum work is updated as necessitated by the change from MD5 to SHA256.

The build for ffmpeg is set to the latest version and will continually update as new versions are released. Updating without some form of pinning may or may not be an acceptable approach here.

The old provider could be added as a backup. Issues could arise in cases where the fallback is used as builds between providers may not be versioned 1:1 (not in a strict reproducibility sense).

Note

Switches static FFmpeg to BtbN GitHub builds with SHA256 verification and introduces BuildKit caching across Dockerfiles, improving build speed and reliability.

Dockerfiles:
- FFmpeg:
  - Introduce ffmpeg-downloader stage (Ubuntu 22.04) that fetches latest static FFmpeg/FFprobe from BtbN/FFmpeg-Builds with SHA256 validation and arch handling; copy binaries into final images.
  - Remove prior inline download using johnvansickle.com and MD5 checks.
- Build performance:
  - Add BuildKit cache mounts for Go builds (/root/.cache/go-build, /go/pkg/mod).
  - Add npm cache mounts for installs (including global typescript, playwright-core, tsx).
  - Use apt layer caching and keep-downloaded-packages configuration.
- Chromium headful (images/chromium-headful/Dockerfile):
  - Apply caching to Go and client builds; swap FFmpeg install to copy from downloader; minor apt cleanup change (drop apt-get clean).
- Chromium headless (images/chromium-headless/image/Dockerfile):
  - Apply caching to Go builds and global npm install; add FFmpeg downloader stage and copy binaries; keep Chromium policy and Node 22 setup as before.

^{Description generated by Mesa. Update settings}

Sayan-

Overall looks good! Instead of latest could we keep 7.x? From my understanding the jumps between major versions (as infrequent as they are) like 7.x -> 8.y could introduce breaking changes and right now I don't think we have anything in CI that would detect breakage!

Confirmed container + unikernel images building + running against these changes

images/chromium-headful/Dockerfile

images/chromium-headless/image/Dockerfile

matthewjmarangoni · 2025-11-03T19:20:23Z

As requested @Sayan- the ffmpeg build version now uses the providers latest build of 7.1.x.

The procedure remains the same but the implementation was clarified a bit.

Sayan-

Thanks for making these changes! I noticed issues when running the images after the latest changes. Suggestions are what I used to get it working locally!

Sayan- · 2025-11-03T19:43:53Z

images/chromium-headful/Dockerfile

    make -j$(nproc); \
    make install;

+FROM docker.io/ubuntu:22.04 AS ffmpeg-downloader


Suggested change

FROM docker.io/ubuntu:22.04 AS ffmpeg-downloader

FROM docker.io/ubuntu:22.04 AS ffmpeg-downloader

ARG TARGETARCH

For context the failure I was seeing:

[kernel-images-api] panic: ffmpeg not found or not executable: signal: trace/breakpoint trap [kernel-images-api] [kernel-images-api] goroutine 1 [running]: [kernel-images-api] main.mustFFmpeg() [kernel-images-api] /workspace/server/cmd/api/main.go:212 +0x90 [kernel-images-api] main.main() [kernel-images-api] /workspace/server/cmd/api/main.go:48 +0x2b8 [kernel-images-api] time=2025-11-03T11:27:53.645-08:00 level=INFO msg="server configuration" config="&{Port:10001 FrameRate:10 DisplayNum:1 MaxSizeInMB:500 OutputDir:/recordings PathToFFmpeg:ffmpeg LogCDPMessages:false}" [kernel-images-api] panic: ffmpeg not found or not executable: signal: trace/breakpoint trap [kernel-images-api] [kernel-images-api] goroutine 1 [running]: [kernel-images-api] main.mustFFmpeg() [kernel-images-api] /workspace/server/cmd/api/main.go:212 +0x90 [kernel-images-api] main.main() [kernel-images-api] /workspace/server/cmd/api/main.go:48 +0x2b8 kernel-images-api: ERROR (spawn error)

under the hood when running build-docker.sh on my macbook I ended up with the wrong arch

root@5c30d0e00714:/# which ffmpeg /usr/local/bin/ffmpeg root@5c30d0e00714:/# ls -la /usr/local/bin/ffmpeg -rwxr-xr-x 1 root root 137227080 Nov 3 19:23 /usr/local/bin/ffmpeg root@5c30d0e00714:/# /usr/local/bin/ffmpeg -version rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2 Trace/breakpoint trap root@5c30d0e00714:/# ldd /usr/local/bin/ffmpeg not a dynamic executable root@5c30d0e00714:/# uname -m aarch64 root@5c30d0e00714:/# dpkg --print-architecture arm64

images/chromium-headless/image/Dockerfile

images/chromium-headful/Dockerfile

mesa-dot-dev bot reviewed Sep 10, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

matthewjmarangoni marked this pull request as draft September 30, 2025 18:13

matthewjmarangoni marked this pull request as ready for review September 30, 2025 18:13

This comment was marked as outdated.

Sign in to view

mesa-dot-dev bot reviewed Sep 30, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

matthewjmarangoni and others added 12 commits October 30, 2025 19:26

upd(build): reduce image size and build time

68a1256

Cache packages, remove unused packages, and prevent installation of apt recommended packages.

upd(build): use cached ffmpeg when host is unreachable

4575a56

upd(build): erase straggling metadata removal

56c8c09

upd(build): improve cache writer parallelism

5f938da

upd(build): install recommended chromium packages

2692ad5

fix(build): whitespace and command order consistency

05a63a1

upd(build): upgrade ffmpeg and change provider

a724b78

fix(build): avoid potential of pattern evaluation

f460cf2

upd(build): stage ffmpeg download

f06f24d

upd(build): add extra space for checksum format

95ab4be

upd(build): change ffmpeg cache access

0133b6f

upd(build): add cache to go and npm invocations

fe3e0e5

matthewjmarangoni force-pushed the upd-change-ffmpeg-provider branch from b0cb372 to fe3e0e5 Compare October 31, 2025 18:12

Merge branch 'main' into upd-change-ffmpeg-provider

d4db75e

Sayan- requested changes Oct 31, 2025

View reviewed changes

images/chromium-headful/Dockerfile Outdated Show resolved Hide resolved

matthewjmarangoni added 3 commits November 1, 2025 00:43

fix(build): limit ffmpeg to 7.1 latest releases

3cc12de

upd(build): handle arm64 and amd64 ffmpeg architectures

c360d31

upd(build): simplify implementation

9d94b9a

cursor bot reviewed Nov 3, 2025

View reviewed changes

images/chromium-headless/image/Dockerfile Show resolved Hide resolved

matthewjmarangoni requested a review from Sayan- November 3, 2025 19:20

Sayan- reviewed Nov 3, 2025

View reviewed changes

upd(build): support multi-platform builds, refine caching

2f19ef4

cursor bot reviewed Nov 4, 2025

View reviewed changes

images/chromium-headful/Dockerfile Show resolved Hide resolved

Sayan- approved these changes Nov 4, 2025

View reviewed changes

Sayan- merged commit ce46982 into kernel:main Nov 4, 2025
4 of 6 checks passed

	FROM docker.io/ubuntu:22.04 AS ffmpeg-downloader
	FROM docker.io/ubuntu:22.04 AS ffmpeg-downloader
	ARG TARGETARCH

Update ffmpeg and change static build provider #64

Update ffmpeg and change static build provider #64

Uh oh!

Conversation

matthewjmarangoni commented Sep 10, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Why we made these changes

What changed?

Validation

Uh oh!

mesa-dot-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

matthewjmarangoni commented Sep 10, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

mesa-dot-dev bot left a comment

Choose a reason for hiding this comment

Analysis

Uh oh!

This comment was marked as outdated.

Uh oh!

mesa-dot-dev bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mesa Description

Uh oh!

Sayan- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

matthewjmarangoni commented Nov 3, 2025

Uh oh!

Sayan- left a comment

Choose a reason for hiding this comment

Uh oh!

Sayan- Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Sayan- Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthewjmarangoni commented Sep 10, 2025 •

edited by cursor bot

Loading

mesa-dot-dev bot commented Oct 31, 2025 •

edited

Loading