Skip to content

osbuild: more refactor to prep for calling osbuild to generate multiple artifacts #3925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

osbuild: more refactor to prep for calling osbuild to generate multiple artifacts #3925

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b35477e
cmd-buildextend-metal: drop ext4 fsverify handling
dustymabe Nov 4, 2024
358e7f5
cmd-buildextend-metal: drop platforms.json code
dustymabe Nov 4, 2024
8b84f39
runvm-osbuild: get platform information from --platform option
dustymabe Nov 5, 2024
2e03161
cmd-buildextend-metal: some rework of generated config for runvm-osbuild
dustymabe Nov 5, 2024
f0d16f4
osbuild: move generation of runvm-osbuild-config.json to build stage
dustymabe Nov 5, 2024
7cd2a6b
cmd-buildextend-metal: variable rename s/image_type/platform
dustymabe Nov 5, 2024
ce64a6c
osbuild: allow for multiple files to be output from a pipeline
dustymabe Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions src/cmd-build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,84 @@ build_followup_targets() {
done
}

# Parse the passed config JSON and extract a mandatory value
getconfig() {
k=$1
config=$2
jq -re .\""$k"\" < "${config}"
}

# Return a configuration value, or default if not set
getconfig_def() {
k=$1
shift
default=$1
config=$2
jq -re .\""$k"\"//\""${default}"\" < "${config}"
}

# Here we generate the input JSON we pass to runvm_osbuild for all of our image builds
generate_runvm_osbuild_config() {
# Grab a few values from $image_json
deploy_via_container=$(getconfig_def "deploy-via-container" "" "${image_json}")
extra_kargs="$(python3 -c 'import sys, json; args = json.load(sys.stdin)["extra-kargs"]; print(" ".join(args))' < "${image_json}")"

# The OSTree repo is at $tmprepo and the commit is $commit
ostree_repo="${tmprepo}"
ostree_commit="${commit}"

# OStree container ociarchive file path and container_imgref
builddir=$(get_build_dir "${buildid}")
ostree_container="${builddir}/${ostree_tarfile_path}"
# If no container_imgref was set let's just set it to some professional
# looking default. The name of the ociarchive file should suffice.
container_imgref_default="ostree-image-signed:oci-archive:/$(basename "${ostree_container}")"
container_imgref=$(getconfig_def "container_imgref" "${container_imgref_default}" "${image_json}")

echo "Estimating disk size..."
# The additional 35% here is obviously a hack, but we can't easily completely fill the filesystem,
# and doing so has apparently negative performance implications.
ostree_size_json="$(/usr/lib/coreos-assembler/estimate-commit-disk-size --repo "$ostree_repo" "$commit" --add-percent 35)"
rootfs_size_mb="$(jq '."estimate-mb".final' <<< "${ostree_size_json}")"
# The minimum size of a disk image we'll need will be the rootfs_size
# estimate plus the size of the non-root partitions. We'll use this
# size for the metal images, but for the IaaS/virt image we'll use
# the size set in the configs since some of them have minimum sizes that
# the platforms require and we want a "default" disk size that has some
# free space.
nonroot_partition_sizes=513
# On s390x there is one more build - Secure Execution case, which has
# different image layout. We add the sizes of the se and verity
# partitions so that they don't "eat into" the 35% buffer (though note
# this is all blown away on first boot anyway). For 's390x.mpp.yaml'
# simplicity all s390x images have same size (of secex image).
if [[ $basearch == "s390x" ]]; then
nonroot_partition_sizes=$((nonroot_partition_sizes + 200 + 128 + 256 + 1))
fi
metal_image_size_mb="$(( rootfs_size_mb + nonroot_partition_sizes ))"
cloud_image_size_mb="$(jq -r ".size*1024" < "${image_json}")"
echo "Disk sizes: metal: ${metal_image_size_mb}M (estimated), cloud: ${cloud_image_size_mb}M"

# Generate the JSON describing the disk we want to build
runvm_osbuild_config_json="runvm-osbuild-config.json"
yaml2json /dev/stdin "tmp/${runvm_osbuild_config_json}" <<EOF
container-imgref: "${container_imgref}"
deploy-via-container: "${deploy_via_container}"
osname: "${name}"
ostree-container: "${ostree_container}"
ostree-ref: "${ref}"
extra-kargs-string: "${extra_kargs}"
ostree-repo: "${ostree_repo}"
metal-image-size: "${metal_image_size_mb}"
cloud-image-size: "${cloud_image_size_mb}"
# Note: this is only used in the secex case; there, the rootfs is
# not the last partition on the disk so we need to explicitly size it
rootfs-size: "${rootfs_size_mb}"
EOF
/usr/lib/coreos-assembler/finalize-artifact "tmp/${runvm_osbuild_config_json}" "${runvm_osbuild_config_json}"
Copy link
Member

@jlebon jlebon Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, though should it be in the build dir? E.g. this will end up in S3. I guess there might be value in archiving it, though I would just leave it in tmp/ to start.

Copy link
Member Author

@dustymabe dustymabe Nov 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok reworking this to acommodate I just went to the next step (i was heading towards anyway) and opened #3930

}


prepare_build

ostree --version
Expand Down Expand Up @@ -616,6 +694,10 @@ mv "${lockfile_out}" .
"${dn}"/commitmeta_to_json "${tmprepo}" "${commit}" > commitmeta.json.tmp
/usr/lib/coreos-assembler/finalize-artifact commitmeta.json{.tmp,}

# Generate the runvm-osbuild config file that will be used for all
# later image builds.
generate_runvm_osbuild_config

# Clean up our temporary data
saved_build_tmpdir="${workdir}/tmp/last-build-tmp"
rm -rf "${saved_build_tmpdir}"
Expand Down
83 changes: 1 addition & 82 deletions src/cmd-buildextend-metal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,20 +116,6 @@ fi
# reread these values from the build itself rather than rely on the ones loaded
# by prepare_build since the config might've changed since then
name=$(meta_key name)
ref=$(meta_key ref)
if [ "${ref}" = "None" ]; then
ref=""
fi
commit=$(meta_key ostree-commit)

ostree_repo=${tmprepo}
# Ensure that we have the cached unpacked commit
import_ostree_commit_for_build "${build}"
# Note this overwrote the bits generated in prepare_build
# for image_json. In the future we expect to split prepare_build
# into prepare_ostree_build and prepare_diskimage_build; the
# latter path would only run this.
image_json=${workdir}/tmp/image.json

image_format=raw
if [[ "${image_type}" == "qemu" || "${image_type}" == "qemu-secex" ]]; then
Expand All @@ -139,73 +125,6 @@ fi
imgname=${name}-${build}-${image_type}.${basearch}.${image_format}
imgpath=${PWD}/${imgname}

# Parse the passed config JSON and extract a mandatory value
getconfig() {
k=$1
config=$2
jq -re .\""$k"\" < "${config}"
}
# Return a configuration value, or default if not set
getconfig_def() {
k=$1
shift
default=$1
config=$2
jq -re .\""$k"\"//\""${default}"\" < "${config}"
}

# Grab a few values from ${image_json}
deploy_via_container=$(getconfig_def "deploy-via-container" "" "${image_json}")
extra_kargs="$(python3 -c 'import sys, json; args = json.load(sys.stdin)["extra-kargs"]; print(" ".join(args))' < "${image_json}")"

# OStree container ociarchive file path
ostree_container="${builddir}/$(meta_key images.ostree.path)"
# If no container_imgref was set let's just set it to some professional
# looking default. The name of the ociarchive file should suffice.
container_imgref_default="ostree-image-signed:oci-archive:/$(basename "${ostree_container}")"
container_imgref=$(getconfig_def "container_imgref" "${container_imgref_default}" "${image_json}")

echo "Estimating disk size..."
# The additional 35% here is obviously a hack, but we can't easily completely fill the filesystem,
# and doing so has apparently negative performance implications.
ostree_size_json="$(/usr/lib/coreos-assembler/estimate-commit-disk-size --repo "$ostree_repo" "$commit" --add-percent 35)"
rootfs_size_mb="$(jq '."estimate-mb".final' <<< "${ostree_size_json}")"
# The minimum size of a disk image we'll need will be the rootfs_size
# estimate plus the size of the non-root partitions. We'll use this
# size for the metal images, but for the IaaS/virt image we'll use
# the size set in the configs since some of them have minimum sizes that
# the platforms require and we want a "default" disk size that has some
# free space.
nonroot_partition_sizes=513
# On s390x there is one more build - Secure Execution case, which has
# different image layout. We add the sizes of the se and verity
# partitions so that they don't "eat into" the 35% buffer (though note
# this is all blown away on first boot anyway). For 's390x.mpp.yaml'
# simplicity all s390x images have same size (of secex image).
if [[ $basearch == "s390x" ]]; then
nonroot_partition_sizes=$((nonroot_partition_sizes + 200 + 128 + 256 + 1))
fi
metal_image_size_mb="$(( rootfs_size_mb + nonroot_partition_sizes ))"
cloud_image_size_mb="$(jq -r ".size*1024" < "${image_json}")"
echo "Disk sizes: metal: ${metal_image_size_mb}M (estimated), cloud: ${cloud_image_size_mb}M"

# Generate the JSON describing the disk we want to build
runvm_osbuild_config_json="${tmp_builddir}/runvm-osbuild-config.json"
yaml2json /dev/stdin "${runvm_osbuild_config_json}" <<EOF
container-imgref: "${container_imgref}"
deploy-via-container: "${deploy_via_container}"
osname: "${name}"
ostree-container: "${ostree_container}"
ostree-ref: "${ref}"
extra-kargs-string: "${extra_kargs}"
ostree-repo: "${ostree_repo}"
metal-image-size: "${metal_image_size_mb}"
cloud-image-size: "${cloud_image_size_mb}"
# Note: this is only used in the secex case; there, the rootfs is
# not the last partition on the disk so we need to explicitly size it
rootfs-size: "${rootfs_size_mb}"
EOF

# In the jenkins pipelines we build the qemu image first and that operation
# will do a lot of the same work required for later artifacts (metal, metal4k, etc)
# so we want the cached output from that run to persist. The later artifacts get
Expand All @@ -214,7 +133,7 @@ EOF
# This is OK because we don't checkpoint (cache) any of those stages.
[ "${image_type}" == "qemu" ] && snapshot="off" || snapshot="on"
runvm_with_cache_snapshot "$snapshot" -- /usr/lib/coreos-assembler/runvm-osbuild \
--config "${runvm_osbuild_config_json}" \
--config "${builddir}/runvm-osbuild-config.json" \
--mpp "/usr/lib/coreos-assembler/osbuild-manifests/coreos.osbuild.${basearch}.mpp.yaml" \
--filepath "${imgpath}" \
--platform "${image_type}"
Expand Down