osbuild: pause using container-storage for builds #4374
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using container storage in the supermin VM was first introduced in 9190a34. This was a nice optimization but it's causing issues where the digest of the deployed container within the bootimages is different than what is in our metadata and what is pushed to the registry. See coreos/fedora-coreos-tracker#2066 for more information.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request pauses the usage of container storage for builds by commenting out the logic that checks for local container images. This is a temporary measure to address an issue with inconsistent container digests that was affecting Zincati, as detailed in the linked issue. The change is clear, correct, and the added comment provides good context for why this optimization is being disabled. The approach of commenting out the code is appropriate for a temporary pause.
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Nov 24, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the releases OCI image. This results in deployed being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Nov 24, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the released OCI image. This results in deployed nodes being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. Note that we won't update the last release in the graph, so Zincati have a valid checksum to fetch from the registry. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Nov 30, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the released OCI image. This results in deployed nodes being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. Note that we won't update the last release in the graph, so Zincati have a valid checksum to fetch from the registry. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Dec 2, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the released OCI image. This results in deployed nodes being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. Note that we won't update the last release in the graph, so Zincati have a valid checksum to fetch from the registry. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Dec 2, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the released OCI image. This results in deployed nodes being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. Note that we won't update the last release in the graph, so Zincati have a valid checksum to fetch from the registry. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
jbtrystram
added a commit
to jbtrystram/fedora-coreos-cincinnati
that referenced
this pull request
Dec 4, 2025
A bug introduced in coreos-assembler[1] (reverted [2]) ended up creating disk images that had different OCI digests than the released OCI image. This results in deployed nodes being unable to update [3] because Zincati look for the deployed OCI checksum in the graph to determines the update path [4]. This introduces a workaround for it : one day a week we will change the image pullspec in the OCI graph with the invalid values to give a chance to Zincati to trigger an update. This will get the node back to a valid state. Note that we won't update the last release in the graph, so Zincati have a valid checksum to fetch from the registry. [1] coreos/coreos-assembler@9190a34 [2] coreos/coreos-assembler#4374 [3] coreos/fedora-coreos-tracker#2066 [4] https://github.com/coreos/zincati/blob/238a79a9c2d11a39d7b7f9c6e71888b75d2c6ab3/src/cincinnati/mod.rs#L230-L245
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using container storage in the supermin VM was first introduced in 9190a34. This was a nice optimization but it's causing issues where the digest of the deployed container within the bootimages is different than what is in our metadata and what is pushed to the registry.
See coreos/fedora-coreos-tracker#2066 for more information.