Container Vulnerabilities

[cf-sewe]

Piraeus is by far the container with most vulnerabilities currently in my environment. I have recently introduced Trivy Operator to scan for vulnerabilities.

Is this known, how could I contribute to improving this?
Who is responsible for piraeus-server and piraeus-csi (the ones with the most vulns)?

Piraeus Operator: 2.8.1 (as of 7/7/25):

Piraeus Operator: 2.9.0 (as of 7/7/25):

Note: The grafana dashboard is a bit misleading (I think numbers are too high), because it seemingly aggregates the number of Vulnerabilities across the deployment components (3 replicas etc.)

[WanzenBug]

This is known. The issue is simply that we do not have the necessary setup to rebuild the base images for fixed vulnerabilities. We use Github Actions to build the images, and for most components images are build on component release. If months later a new vulnerability in one of the base image components is found: bad luck, our CI pipeline is not set up to do a rebuild.

The images from here are bit different, as there we rebuild on every push to master, but currently this also only happens on actual changes.

Ideally, we would have a setup that automatically rebuild the latest image for every component once there is a fix. However, I am currently not aware of a CI setup that would allow that. We are open for contributions.

PS: You are not using the latest Piraeus release, so please first update to 2.9.0. It probably won't fix everything, but at least the images are a bit newer....

[cf-sewe]

Thanks for the clarification!

I’ve seen similar issues in other projects and wanted to share a potential approach to reduce exposure to CVEs from outdated base images — even when there’s no new upstream release (e.g., LINSTOR or CSI).

A few thoughts and questions:

• Would it be feasible to add a scheduled GitHub Actions workflow (e.g., weekly) that rebuilds and republishes container images using --pull to ensure the latest base layers are included?
• Does GitHub Actions execution create any cost or quota limitations that would prevent setting up such recurring jobs?
• If we trigger a rebuild due to base image updates only, what’s your preferred tagging strategy? I’d recommend a patch bump (v1.31.1) or a suffix like v1.31.0-rev1, to keep tags immutable and traceable.
• Would you be open to using Dependabot or Renovate to monitor base image changes (e.g., Dockerfile digests) and trigger rebuilds?
• For changelog/versioning automation, I’d propose using Release Please. For example, when a base image gets updated (via Renovate/Dependabot or a scheduled job), a small commit like chore: rebuild due to base update could trigger Release Please to bump the patch version and tag a release — which in turn would trigger the image build & push pipeline. That keeps everything reproducible and discoverable without requiring upstream changes.

One thing I'm unsure about: how complex is the Piraeus Operator ecosystem in terms of ownership? I assume some important containers (e.g. LINSTOR core components) are not maintained directly within your organization. Can you shed some light on which images are under your team's control, and which aren't?

Also, from what I’ve seen so far: each image is built from its own GitHub repo with its own Dockerfile and CI. So if rebuild automation is introduced, I assume it would need to be added individually in each repo. Would you consider this a realistic goal, or do you see major obstacles in doing that (e.g., ownership boundaries, CI quotas, coordination across teams)?

Happy to assist or contribute if this direction makes sense for you!

[WanzenBug]

Thanks for sharing your experience!

Would it be feasible to add a scheduled GitHub Actions workflow (e.g., weekly) that rebuilds and republishes container images using --pull to ensure the latest base layers are included?

Could certainly be done. It would need some thought on how to select the release to rebuild, but that should not be that hard.

Does GitHub Actions execution create any cost or quota limitations that would prevent setting up such recurring jobs?

I believe there is a limit, but I expect we are well below the limit on "free" CI resources.

If we trigger a rebuild due to base image updates only, what’s your preferred tagging strategy? I’d recommend a patch bump (v1.31.1) or a suffix like v1.31.0-rev1, to keep tags immutable and traceable.

This is actually a hard problem. If we update the tag, we lose traceability, if we don't update the tag, how would users get the new versions. For Piraeus Operator, we have the image configuration in a ConfigMap that is part of the release. To update the images, users would need to update that ConfigMap (replacing v1.31.0 with v1.31.0-rebuild20250708 or similar). That also does not seem ideal unless it could be automated some way 🤔

Would you be open to using Dependabot or Renovate to monitor base image changes (e.g., Dockerfile digests) and trigger rebuilds?

Sure.

One thing I'm unsure about: how complex is the Piraeus Operator ecosystem in terms of ownership? I assume some important containers (e.g. LINSTOR core components) are not maintained directly within your organization. Can you shed some light on which images are under your team's control, and which aren't?

Those components are not maintained here, that is correct, we only consume the provided public .deb files for those. So in a sense we are in control of all the images, with the caveat that for the piraeus-server and drbd9-* images we are not in control of the specific release process, i.e. if there was some kind of CVE in a .jar distributed by LINSTOR, we cannot simply rebuild the base image.

Another big exception are the CSI sidecar containers, which are maintained by the Kubernetes sig-storage team. Those generally only consist of a single Go binary, so they are not really prone to needing updates.

Also, from what I’ve seen so far: each image is built from its own GitHub repo with its own Dockerfile and CI. So if rebuild automation is introduced, I assume it would need to be added individually in each repo. Would you consider this a realistic goal, or do you see major obstacles in doing that (e.g., ownership boundaries, CI quotas, coordination across teams)?

More or less, the jungle of docker files has grown organically, so while the CI set up is often similar, there may be some exceptions that need to be dealt with. No deal breakers as far as I can see, just a bit more effort to get it all into a working state. On the plus-side, we can gradually roll out the changes on a per-repo basis, deal with any fallout as we find it.

[cf-sewe]

Thanks, that all makes sense — especially starting small and iterating per repo.

I’d suggest we begin with a repo that has clear benefits and relatively simple build logic. piraeus-csi might be a good candidate: it’s under your control, frequently used, and has a standalone build without too many upstream packaging dependencies.

Regarding the tagging strategy: I think introducing new tags for base-image-only rebuilds is essential for traceability and cache consistency. Mutable tags would make it harder for users to verify what’s actually running. The vX.Y.Z-rebuildYYYYMMDD or a simple patch bump (v1.7.1 → v1.7.2) both work and can be tracked via GitHub Releases or changelogs. There’s no real downside for users — if they have automation (e.g., Renovate, Flux2, or similar), they'll pick up the new tag; if not, they’re doing manual updates anyway. Here's a good write-up on this: Best Practices for Immutable Tags.

Happy to contribute a draft GitHub Actions workflow for linstor-csi if that helps.

---

While reviewing the repo in more detail, I also looked at how things are currently set up and where changes could land:

• The Dockerfile uses golang:1 and debian:bookworm-slim — both of which benefit from regular refreshes.
• The image is built and pushed via make upload with buildx and multi-arch enabled.
• CHANGELOG.md is maintained manually and tied to actual software releases.
• Dependabot is already configured for Go modules — nice!
• No workflow exists yet for rebuilding regularly or responding to Dockerfile changes.

With that in mind, I’d suggest:

• Add Docker image update tracking to dependabot.yml:

  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"

• Add a lightweight workflow (e.g. .github/workflows/auto-rebuild.yml) that triggers on:

  • PR merges touching Dockerfile, go.mod, go.sum
  • optionally, a weekly schedule to catch untagged base image updates

• Build using make upload, derive tag from git describe or a suffix like v1.8.1-20250708, and push to Quay

---

🔁 Optional/Extended Proposal: Switch to Explicit Base Image Tags

As an alternative to scheduled rebuilds and base image digest tracking, we could simplify the model by switching to explicit image tags in the Dockerfile, e.g.:

FROM golang:1.22.4-bookworm AS build
FROM debian:bookworm-20250630-slim AS runtime

These tags are versioned and immutable. They’re published to Docker Hub and updated regularly. By using those:

• Dependabot can detect new tag versions and open a PR (no digest tracking needed)
• We rebuild the image only when the Dockerfile or go.mod actually changes
• No need for scheduled workflows or external digest comparison
• No rebuild logic needed — the PR merge itself is the trigger
• Workflow trigger would just be:

on:
  push:
    branches: [ master ]
    paths:
      - "Dockerfile"
      - "go.mod"
      - "go.sum"

This gives clean, reproducible builds with full visibility into when and why a rebuild happened, while keeping the workflow simple and predictable.

⸻

Before sending a PR, a few questions:

• Do you think starting with piraeusdatastore/piraeus-csi repository makes sense, or do you have a better candidate in mind?

• Is it okay to publish images with version suffixes like v1.7.1-20250708 without a matching Git tag or GitHub release?
  This avoids release churn but still allows users to explicitly opt into a fresher base image.

• Should latest be updated to point to these rebuilt images?
  My take: yes — it provides an easy path to get patched images. If someone really wants the old image, they can pin it.

• Would you prefer using make upload in CI, or switch to native GitHub Actions Docker build steps?
  I’d lean toward keeping make to stay consistent with current workflows unless there’s a reason to switch.

• Are you open to switching from floating to specific Dockerfile image tags (like bookworm-20250630-slim)?
  This would allow Dependabot to trigger PRs without needing scheduled digest checks or rebuild logic.

• For efficient development & testing — I plan to fork the repo and test the proposed workflow + Dependabot config there first.
  That way I can iterate and verify the build behavior (incl. tagging and push logic) without affecting this repo.
  Once I have a working setup, I’ll prepare a minimal PR for review.

Let me know what you think.

[WanzenBug]

I don't see the point of the "Explicit Base Image Tags" approach: what if we need to rebuild because one of our extra dependencies (like xfsprogs) needs to be updated? This is then not covered by that approach.

I also don't see the point in having a dependabot configuration for the Dockerfile: we already use "floating" tags, there would be nothing to update unless there would be a sudden release of golang:2, but I doubt that.

Lastly, weekly rebuilds sound good. What I would suggest is the following strategy:

• For every release, we create a new branch named release-vX.Y.Z
• We update dependabot.yml to also target updates to the latest release branch in addition to the main branch
• We update our workflow to have a scheduled workflow that rebuilds the release-* branch

That way, we ensure that we rebuild a proper release, not just any version that is currently on the main branch. That should prevent any surprising new feature from slipping into the rebuilt images.

I would also appreciate if I was talking to an actual human. I believe the initial discussion was quite productive. Adding hallucinated links about "Best Practices for Immutable Tags" I find less productive. :-/