Space and Time Reduction #60
Conversation
matthewjmarangoni
force-pushed
the
upd-space-and-time
branch
from
94603a7 to
18ae79c
Compare
rgarcia
left a comment
rgarcia
left a comment
There was a problem hiding this comment.
one question otherwise looks awesome. Thanks @matthewjmarangoni!
|
Caching can be used to help builds continue to run if a resource is offline and having just experienced an outage of the Also cleaned up a straggler that was removing index metadata. |
matthewjmarangoni
force-pushed
the
upd-space-and-time
branch
2 times, most recently
from
1e8213e to
d66de84
Compare
|
Changing the locking mechanism for apt caches supports multiple writers and yields some time reduction. |
matthewjmarangoni
force-pushed
the
upd-space-and-time
branch
from
d4a4831 to
0a438f7
Compare
![mesa-dot-dev[bot]](https://avatars.githubusercontent.com/in/1050077?s=60&v=4){kind=small}
There was a problem hiding this comment.
Performed full review of 330685e...0a438f7
6 files reviewed | 0 comments | Review on Mesa | Edit Reviewer Settings
matthewjmarangoni
force-pushed
the
upd-space-and-time
branch
from
d91a31b to
2d40710
Compare
There was a problem hiding this comment.
Performed full review of 330685e...2d40710
Tip
⚡ Quick Actions
This review was generated by Mesa.
Actions:
Slash Commands:
/review- Request a full code review/review latest- Review only changes since the last review/describe- Generate PR description. This will update the PR body or issue comment depending on your configuration/help- Get help with Mesa commands and configuration options
6 files reviewed | 6 comments | Review on Mesa | Edit Reviewer Settings
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@matthewjmarangoni let me know when you want another review! Would like to get this one over the finish line |
There was a problem hiding this comment.
Performed full review of 330685e...992f8fd
Tip
⚡ Quick Actions
This review was generated by Mesa.
Actions:
Slash Commands:
/review- Request a full code review/review latest- Review only changes since the last review/describe- Generate PR description. This will update the PR body or issue comment depending on your configuration/help- Get help with Mesa commands and configuration options
6 files reviewed | 6 comments | Review on Mesa | Edit Reviewer Settings
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
@rgarcia as long as that answer for the previous question you had was sufficient, it should be ready now that other issues have been resolved! |
There was a problem hiding this comment.
Performed full review of 330685e...992f8fd
Tip
⚡ Quick Actions
This review was generated by Mesa.
Actions:
Slash Commands:
/review- Request a full code review/review latest- Review only changes since the last review/describe- Generate PR description. This will update the PR body or issue comment depending on your configuration/help- Get help with Mesa commands and configuration options
6 files reviewed | 7 comments | Review on Mesa | Edit Reviewer Settings
Cache packages, remove unused packages, and prevent installation of apt recommended packages.
matthewjmarangoni
force-pushed
the
upd-space-and-time
branch
from
4ba6b75 to
05a63a1
Compare
Mesa DescriptionThis is a pass to reduce build times and image sizes. The impact varies but the general expectation is improvement. Image size reductions of 18% - 30% and build time reductions of 7% - 35% were found. Some results were greater yet less consistent and should not be expected. Changes to achieve this include:
Manual testing was done. Further testing is requested to verify impact and functionality. Particularly of interest are differing and constrained environments such as CI/CD pipelines. TL;DRReduced Docker image sizes and build times by implementing widespread caching for Why we made these changesTo reduce build times and image sizes for faster, more efficient CI/CD pipelines and local development. Image size reductions of 18% - 30% and build time reductions of 7% - 35% were observed in testing. What changed?
ValidationManual testing was done. Further testing is requested to verify impact and functionality, particularly in differing and constrained environments such as CI/CD pipelines. Description generated by Mesa. Update settings |
Sayan-
left a comment
Sayan-
left a comment
There was a problem hiding this comment.
I built container and unikernel images off of main@a0dca70 and this branch@05a63a1
confirmed container images were smaller as expected:
confirmed unikernel images were also smaller as expected:
onkernel/chromium-headful-test a0dca70 2.8 GB
onkernel/chromium-headful-test 05a63a1 2.1 GB
onkernel/chromium-headless-test a0dca70 1.8 GB
onkernel/chromium-headless-test 05a63a1 1.7 GB
Larger improvements for the headful image as expected.
Do note, I didn't compare build speeds as part of this review. We currently don't really optimize for it and the script run times felt good enough or better (100% vibes based).
Thank you for this contribution!
Sayan-
dismissed
rgarcia’s stale review
Confirming feedback is addressed and I'm taking over this PR!
Please merge PR #60 prior to merging this PR. The basis of this PR is PR #60 and once merged this diff should clear up. This PR changes the provider for static ffmpeg builds to a Github repository source linked from [https://ffmpeg.org/download.html](https://ffmpeg.org/download.html). The associated checksum work is updated as necessitated by the change from MD5 to SHA256. The build for ffmpeg is set to the latest version and will continually update as new versions are released. Updating without some form of pinning may or may not be an acceptable approach here. The old provider could be added as a backup. Issues could arise in cases where the fallback is used as builds between providers may not be versioned 1:1 (not in a strict reproducibility sense). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Replaces FFmpeg static builds with BtbN GitHub releases using SHA256 verification and adds platform-scoped BuildKit caches across Dockerfiles for faster, reproducible builds. > > - **FFmpeg**: > - Add dedicated `ffmpeg-downloader` stage (Ubuntu 22.04) that fetches from `BtbN/FFmpeg-Builds` with SHA256 verification and arch selection; copy `ffmpeg`/`ffprobe` into final images. > - Remove old inline downloads and MD5 checks in `images/chromium-headful/Dockerfile` and `images/chromium-headless/image/Dockerfile`. > - **Build performance (BuildKit caching)**: > - Introduce platform-aware cache prefix `CACHEIDPREFIX` using `TARGETOS`/`TARGETARCH`. > - Add cache mounts for `apt`, `npm`, and `go` in `chromium-headful`, `chromium-headless`, `client`, `xorg-deps`, and `xf86-input-neko` Dockerfiles. > - **Misc Dockerfile tweaks**: > - Standardize `ARG TARGETOS/TARGETARCH` across stages; keep downloaded APT packages and use locked cache sharing; minor cleanup (e.g., switch global npm installs to cached). > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 2f19ef4. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY --> --- <!-- mesa-description-start --> ## TL;DR Switched the static `ffmpeg` build provider to a more reliable, official source and significantly optimized Docker image builds across the project by leveraging BuildKit caching. ## Why we made these changes The previous `ffmpeg` provider was unreliable, causing potential build failures. The new provider is linked directly from the official `ffmpeg.org` website, ensuring a trusted and up-to-date source. Additionally, our Docker builds were inefficient; implementing widespread BuildKit caching dramatically speeds up build times and reduces final image sizes by avoiding redundant package downloads. ## What changed? - **FFmpeg Provider:** - Changed the download source for static `ffmpeg` builds to the `BtbN/FFmpeg-Builds` GitHub repository. - Updated the checksum verification from MD5 to SHA256 to match the new provider. - Pinned the build to the `latest` release for continuous updates. - **Build Performance & Image Size:** - Optimized all major Dockerfiles (`chromium-headful`, `chromium-headless`, `xorg-deps`, etc.) to use BuildKit cache mounts for `apt`, `npm`, and `go` package management. - Consistently used `--no-install-recommends` during `apt-get install` across all relevant Dockerfiles to create leaner final images. - **Configuration:** - Added `.mise.toml` to the project's `.gitignore` to ignore local `mise` tool version configurations. ## Validation - [ ] Verify that all Docker images build successfully. - [ ] Confirm that `ffmpeg` is correctly installed and functional inside the final images. - [ ] Check that subsequent builds are noticeably faster due to the new caching mechanisms. <sup>_Description generated by Mesa. [Update settings](https://app.mesa.dev/onkernel/settings/pull-requests)_</sup> <!-- mesa-description-end --> --------- Co-authored-by: Matt Marangoni <[email protected]>
3 participants
This is a pass to reduce build times and image sizes. The impact varies but the general expectation is improvement.
Image size reductions of 18% - 30% and build time reductions of 7% - 35% were found. Some results were greater yet less consistent and should not be expected.
Changes to achieve this include:
apt,npm,go, staticffmpegbuildsaptconfiguration changed to prevent automatic cache removal and to preserve downloadsaptpackages are no longer installedManual testing was done. Further testing is requested to verify impact and functionality. Particularly of interest are differing and constrained environments such as CI/CD pipelines.
TL;DR
Reduced Docker image sizes and build times by implementing widespread caching for
apt,npm,go, andffmpegbuilds.Why we made these changes
To reduce build times and image sizes for faster, more efficient CI/CD pipelines and local development. Image size reductions of 18% - 30% and build time reductions of 7% - 35% were observed in testing.
What changed?
--mount=type=cache) across multiple Dockerfiles to cache dependencies forapt,npm,go, and staticffmpegbuilds.aptUsage: Configuredaptto retain its cache and consistently used the--no-install-recommendsflag to prevent installing unnecessary packages, minimizing image size.gpg-agentto ensure build reliability..mise.tomlto.gitignore.Validation
Manual testing was done. Further testing is requested to verify impact and functionality, particularly in differing and constrained environments such as CI/CD pipelines.
Description generated by Mesa. Update settings
Note
Introduces BuildKit caching and apt optimizations across Dockerfiles, adds FFmpeg checksum/caching, trims packages, and ignores .mise.toml to reduce build times and image sizes.
go mod download,go build,npm install/build, andapt.ffmpegdownloads with MD5 verification and reuse.--no-install-recommendsconsistently; addgpg-agentwhere needed.sqlite3).--no-install-recommendsforxorg-depsandxf86-input-nekobuilds..gitignoreto ignore.mise.toml.Written by Cursor Bugbot for commit 05a63a1. This will update automatically on new commits. Configure here.