Skip to content

Fix missing extensions in RHCOS release browser #4009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ravanelli merged 3 commits into from
Jan 31, 2025
Merged

Fix missing extensions in RHCOS release browser #4009

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bcf0d88
extensions: Make tar file references and checksum optional
ravanelli Jan 24, 2025
7d06d26
Fix missing extensions in RHCOS release browser for 4.16+
ravanelli Jan 24, 2025
6a47b02
Fix: Replace go install with local build of schematyper
ravanelli Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion cmd/build-extensions-container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,34 @@ func buildExtensionContainer() error {
SizeInBytes: float64(stat.Size()),
SkipCompression: true,
}
cosaBuild.MetaStamp = float64(time.Now().UnixNano())
fmt.Printf("Generating meta.json `extensions` entry for: %s\n", buildID)
// The /tmp/extensions.json file is generated as part of the extension container build process.
// For more details, refer to:
// https://github.com/openshift/os/blob/master/extensions/Dockerfile
extensionsFilePath:= "./tmp/extensions.json"
fileContent, err := os.ReadFile(extensionsFilePath)
if err != nil {
fmt.Printf("Error reading JSON file: %v\n", err)
return err
}
// Parse the JSON content into a map
packages := make(map[string]string)
err = json.Unmarshal(fileContent, &packages)
if err != nil {
fmt.Printf("Error parsing JSON content: %v\n", err)
return err
}

// Convert to extensionsInterfaceMap
extensionsInterfaceMap := make(map[string]interface{})
for key, value := range packages {
extensionsInterfaceMap[key] = value
}

cosaBuild.Extensions = &cosa.Extensions {
Manifest: extensionsInterfaceMap,
}
cosaBuild.MetaStamp = float64(time.Now().UnixNano())
newBytes, err := json.MarshalIndent(cosaBuild, "", " ")
if err != nil {
return err
Expand Down
8 changes: 4 additions & 4 deletions pkg/builds/cosa_v1.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package builds

// generated by 'make schema'
// source hash: 4bb2fd8591ae1cc9179181b9efcb550542137a01c9544fd048e66794e0fb176a
// source hash: d7b6e01415317b89e346747884eb3536cc9e78049a2c71615739afd51fa65569

type AdvisoryDiff []AdvisoryDiffItems

Expand Down Expand Up @@ -129,9 +129,9 @@ type Cloudartifact struct {

type Extensions struct {
Manifest map[string]interface{} `json:"manifest"`
Path string `json:"path"`
RpmOstreeState string `json:"rpm-ostree-state"`
Sha256 string `json:"sha256"`
Copy link
Member

@dustymabe dustymabe Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that I don't think we need these any longer. My only concern is do we somehow want to maintain backward compatibility in case we encounter this (i.e. using current tools to look at an older build).

Copy link
Member Author

@ravanelli ravanelli Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would have already noticed this since no extensions were being created at all. I suggest backporting until the point where we stopped generating extensions in meta.json.

Copy link
Member

@dustymabe dustymabe Jan 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand.. but let me ask the question a different way..

If you do bake this PR into a COSA image and then try to parse (i.e. buildfetch and cosa meta --build=415.92.202501281917-0 --get extensions) does it barf on you?

Copy link
Member

@dustymabe dustymabe Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's important to use 4.15 in this test because those builds do have the extensions in the meta.json

Copy link
Member Author

@ravanelli ravanelli Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does work:

cosa meta --get extensions
warning: /sys/fs/selinux appears to be mounted but should not be; enabling workaround
extensions: {'path': 'rhcos-415.92.202501281917-0-extensions.x86_64.tar', 'sha256': '32166c5e89e461d2b5fe1487281e4d5c76369ad51cef5a7252de30b0cad9da9d',
'rpm-ostree-state': '50ac76b51fccfe9cd628f009e3513747c3c9c0ca645315dfea249fc37b2a176e4874b84afe437d86a3e964d25cec91d764af007164b4424b603b7fcd2718009f', 
'manifest': {'NetworkManager-libreswan': '1.2.14-6.el9_2.x86_64',
...

The only diff for now is that we won't have path, sha256 and rpm-ostree-state, but does work in the same way.

Copy link
Member

@dustymabe dustymabe Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. I was just worried that the schema would error if there were components in there that it didn't expect.

Copy link
Member

@jlebon jlebon Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works because the schema doesn't disallow additional properties. So the fields are essentially "unregistered". I think instead of this, we should make those properties optional. That way we still have type modelling for them. This could be useful in the future for GC.

Path string `json:"path,omitempty"`
RpmOstreeState string `json:"rpm-ostree-state,omitempty"`
Sha256 string `json:"sha256,omitempty"`
}

type Gcp struct {
Expand Down
8 changes: 5 additions & 3 deletions pkg/builds/schema_doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Generated by ./generate-schema.sh
// Source hash: 4bb2fd8591ae1cc9179181b9efcb550542137a01c9544fd048e66794e0fb176a
// Source hash: d7b6e01415317b89e346747884eb3536cc9e78049a2c71615739afd51fa65569
// DO NOT EDIT

package builds
Expand Down Expand Up @@ -707,10 +707,12 @@ var generatedSchemaJSON = `{
"$id": "#/properties/extensions",
"type": "object",
"title": "Extensions",
"required": [
"optional": [
"path",
"sha256",
"rpm-ostree-state",
"rpm-ostree-state"
],
"required": [
"manifest"
],
"properties": {
Expand Down
9 changes: 7 additions & 2 deletions schema/generate-schema.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,13 @@ else
schematyper=$(which schematyper 2>/dev/null || true)
fi
if test -z "${schematyper}"; then
env GOBIN=${topdir}/bin go install github.com/idubinskiy/schematyper@latest
schematyper=${topdir}/bin/schematyper
if [ ! -d "${topdir}/schematyper" ]; then
git clone https://github.com/idubinskiy/schematyper.git ${topdir}/schematyper
go -C ${topdir}/schematyper mod edit -replace gopkg.in/alecthomas/kingpin.v2=github.com/alecthomas/kingpin/[email protected]
go -C ${topdir}/schematyper mod tidy && go -C ${topdir}/schematyper mod vendor
go -C ${topdir}/schematyper build
fi
schematyper=${topdir}/schematyper/schematyper
fi

${schematyper} \
Expand Down
2 changes: 2 additions & 0 deletions src/build-extensions-container.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ if [[ -f "${workdir}/src/config.json" ]]; then
fi

mkdir "${ctx_dir}/hotfixes"
touch "${workdir}/tmp/extensions.json"
tar -xC "${ctx_dir}/hotfixes" -f /dev/disk/by-id/virtio-hotfixes

# Build the image, replacing the FROM directive with the local image we have.
Expand All @@ -33,6 +34,7 @@ img=localhost/extensions-container
(set -x; podman build --from oci-archive:"$ostree_ociarchive" --network=host \
--build-arg COSA=true --build-arg VARIANT="${variant}" --label version="$buildid" \
--volume /etc/pki/ca-trust:/etc/pki/ca-trust:ro \
--volume "${workdir}"/tmp/extensions.json:/tmp/extensions.json \
-t "${img}" -f extensions/Dockerfile "${ctx_dir}")

# Call skopeo to export it from the container storage to an oci-archive.
Expand Down
2 changes: 1 addition & 1 deletion src/cmd-coreos-prune
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Build = collections.namedtuple("Build", ["id", "images", "arch", "meta_json"])
# set metadata caching to 5m
CACHE_MAX_AGE_METADATA = 60 * 5
# These lists are up to date as of schema hash
# 4bb2fd8591ae1cc9179181b9efcb550542137a01c9544fd048e66794e0fb176a. If changing
# d7b6e01415317b89e346747884eb3536cc9e78049a2c71615739afd51fa65569. If changing
# this hash, ensure that the list of SUPPORTED and UNSUPPORTED artifacts below
# is up to date.
SUPPORTED = ["amis", "gcp"]
Expand Down
6 changes: 4 additions & 2 deletions src/v1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -701,10 +701,12 @@
"$id": "#/properties/extensions",
"type": "object",
"title": "Extensions",
"required": [
"optional": [
"path",
"sha256",
"rpm-ostree-state",
"rpm-ostree-state"
],
"required": [
"manifest"
],
"properties": {
Expand Down
Loading