Use skopeo for hermetic bootstrap images by tkopecek · Pull Request #1717 · rpm-software-management/mock

tkopecek · 2026-02-20T15:51:34Z

Better replacement for #1716

gemini-code-assist

Code Review

The pull request successfully transitions from podman pull|save to skopeo for handling hermetic bootstrap images, which is a more robust approach for managing image digests and offline transports. However, there is a critical mismatch between how skopeo sync --scoped prepares the directory structure and how the configuration logic attempts to locate it. Switching to skopeo copy with a predictable destination directory (e.g., bootstrap/) would resolve this and maintain consistency with the previous implementation.

gemini-code-assist · 2026-02-20T15:56:13Z

mock/py/mock-hermetic-repo.py

+    pull_cmd = ["skopeo", "sync", "--src", "docker", "--dest", "dir",
+                "--scoped", "--preserve-digests"]
+    if "architecture" in bootstrap_data:
+        pull_cmd += ["--override-arch", bootstrap_data["architecture"]]
    if "pull_digest" in bootstrap_data:
        image_specification +=  "@" + bootstrap_data["pull_digest"]
-    if "architecture" in bootstrap_data:
-        pull_cmd += ["--arch", bootstrap_data["architecture"]]
-    pull_cmd += [image_specification]
+    pull_cmd += [image_specification, outputdir]
    log.info("Pulling like: %s", ' '.join(pull_cmd))


The use of skopeo sync --scoped creates a nested directory hierarchy based on the registry and repository (e.g., docker.io/library/fedora), which does not match the flat path construction in mockbuild/config.py. Since this script is intended to fetch a specific bootstrap image for a build, skopeo copy is more appropriate as it allows specifying a predictable destination directory. Additionally, subprocess.check_call is preferred over check_output when the output is not being captured.

Suggested change

pull_cmd = ["skopeo", "sync", "--src", "docker", "--dest", "dir",

"--scoped", "--preserve-digests"]

if "architecture" in bootstrap_data:

pull_cmd += ["--override-arch", bootstrap_data["architecture"]]

if "pull_digest" in bootstrap_data:

image_specification += "@" + bootstrap_data["pull_digest"]

if "architecture" in bootstrap_data:

pull_cmd += ["--arch", bootstrap_data["architecture"]]

pull_cmd += [image_specification]

pull_cmd += [image_specification, outputdir]

log.info("Pulling like: %s", ' '.join(pull_cmd))

bootstrap_dir = os.path.join(outputdir, "bootstrap")

pull_cmd = ["skopeo", "copy"]

if "architecture" in bootstrap_data:

pull_cmd += ["--override-arch", bootstrap_data["architecture"]]

if "pull_digest" in bootstrap_data:

image_specification += "@" + bootstrap_data["pull_digest"]

pull_cmd += ["docker://" + image_specification, "dir:" + bootstrap_dir]

log.info("Pulling like: %s", ' '.join(pull_cmd))

subprocess.check_call(pull_cmd)

Should work in every case. docker.io/library/fedora is still the part of boostrap_image string, so nested directories will be part of final path?

gemini-code-assist · 2026-02-20T15:56:14Z

mock/py/mockbuild/config.py

+    image_pullspec = f"{data['config']['bootstrap_image']}@{data['bootstrap']['pull_digest']}"
+    bootstrap_dir = os.path.join(final_offline_repo, image_pullspec)
+    config_opts["bootstrap_image"] = f"dir:{bootstrap_dir}"


The current path construction for bootstrap_dir assumes a flat directory named after the pullspec, which conflicts with the hierarchical structure created by skopeo sync --scoped. To ensure the build can correctly locate the offline image, it is better to use a fixed directory name like bootstrap, matching the suggested change in mock-hermetic-repo.py.

Suggested change

image_pullspec = f"{data['config']['bootstrap_image']}@{data['bootstrap']['pull_digest']}"

bootstrap_dir = os.path.join(final_offline_repo, image_pullspec)

config_opts["bootstrap_image"] = f"dir:{bootstrap_dir}"

# Use the offline image for bootstrapping.

bootstrap_dir = os.path.join(final_offline_repo, "bootstrap")

config_opts["bootstrap_image"] = f"dir:{bootstrap_dir}"

praiskup · 2026-02-23T08:45:36Z

CI failure:

      INFO:__main__:Pulling like: skopeo sync --src docker --dest dir --scoped --preserve-digests --override-arch amd64 registry.fedoraproject.org/fedora:rawhide@sha256:926c55dd6993b0332abe54abbc993929b578fa0212bb05d23c187c424f203bbf /tmp/mock-tests-local-repo-3gghtsxg
      time="2026-02-20T16:17:55Z" level=info msg="Tag presence check" imagename="registry.fedoraproject.org/fedora:rawhide@sha256:926c55dd6993b0332abe54abbc993929b578fa0212bb05d23c187c424f203bbf" tagged=true
      time="2026-02-20T16:17:55Z" level=fatal msg="Cannot obtain a valid image reference for transport \"docker\" and reference \"registry.fedoraproject.org/fedora:rawhide@sha256:926c55dd6993b0332abe54abbc993929b578fa0212bb05d23c187c424f203bbf\": Docker references with both a tag and digest are currently not supported"
      Traceback (most recent call last):
        File "/usr/bin/mock-hermetic-repo", line 199, in <module>
          _main()
          ~~~~~^^
        File "/usr/bin/mock-hermetic-repo", line 194, in _main
          prepare_image(data["config"]["bootstrap_image"], data["bootstrap"],
          ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        options.output_repo)
                        ^^^^^^^^^^^^^^^^^^^^
        File "/usr/bin/mock-hermetic-repo", line 138, in prepare_image
          subprocess.check_output(pull_cmd)
          ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
        File "/usr/lib64/python3.14/subprocess.py", line 472, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     **kwargs).stdout
                     ^^^^^^^^^
        File "/usr/lib64/python3.14/subprocess.py", line 577, in run
          raise CalledProcessError(retcode, process.args,
                                   output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '['skopeo', 'sync', '--src', 'docker', '--dest', 'dir', '--scoped', '--preserve-digests', '--override-arch', 'amd64', 'registry.fedoraproject.org/fedora:rawhide@sha256:926c55dd6993b0332abe54abbc993929b578fa0212bb05d23c187c424f203bbf', '/tmp/mock-tests-local-repo-3gghtsxg']' returned non-zero exit status 1.

    And a hermetic build is retriggered with the lockfile and repository                 # None
    Then the build succeeds                                                              # None

praiskup · 2026-02-23T08:48:04Z

mock/py/mock-hermetic-repo.py

    """
-    pull_cmd = ["podman", "pull"]
+    pull_cmd = ["skopeo", "sync", "--src", "docker", "--dest", "dir",
+                "--scoped", "--preserve-digests"]


As discussed before, this worked for me correctly - but I didn't even use --scoped and --preserve-digests 🤷 please document why we need those.

praiskup · 2026-02-23T08:49:07Z

mock/py/mock-hermetic-repo.py

    Store the tarball into the same directory where the RPMs are
    """
-    pull_cmd = ["podman", "pull"]
+    pull_cmd = ["skopeo", "sync", "--src", "docker", "--dest", "dir",


We probably need skopeo in Recommends.

Yes, it is there.

praiskup · 2026-02-23T13:13:48Z

      INFO:__main__:Pulling like: skopeo sync --src docker --dest dir --scoped --preserve-digests --override-arch amd64 quay.io/centos/centos:stream9@sha256:8e47e1ac4bfb7ae13d18403389d0ae0d6b44aa3c5a66e4561bc6cb881c140cd4 /tmp/mock-tests-local-repo-29xlxzf_
      time="2026-02-23T13:09:45Z" level=info msg="Tag presence check" imagename="quay.io/centos/centos:stream9@sha256:8e47e1ac4bfb7ae13d18403389d0ae0d6b44aa3c5a66e4561bc6cb881c140cd4" tagged=true
      time="2026-02-23T13:09:45Z" level=fatal msg="Cannot obtain a valid image reference for transport \"docker\" and reference \"quay.io/centos/centos:stream9@sha256:8e47e1ac4bfb7ae13d18403389d0ae0d6b44aa3c5a66e4561bc6cb881c140cd4\": Docker references with both a tag and digest are currently not supported"
      Traceback (most recent call last):
        File "/usr/bin/mock-hermetic-repo", line 204, in <module>
          _main()
          ~~~~~^^
        File "/usr/bin/mock-hermetic-repo", line 199, in _main
          prepare_image(data["config"]["bootstrap_image"], data["bootstrap"],
          ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        options.output_repo)
                        ^^^^^^^^^^^^^^^^^^^^
        File "/usr/bin/mock-hermetic-repo", line 143, in prepare_image
          subprocess.check_output(pull_cmd)
          ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
        File "/usr/lib64/python3.14/subprocess.py", line 472, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     **kwargs).stdout
                     ^^^^^^^^^
        File "/usr/lib64/python3.14/subprocess.py", line 577, in run
          raise CalledProcessError(retcode, process.args,
                                   output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '['skopeo', 'sync', '--src', 'docker', '--dest', 'dir', '--scoped', '--preserve-digests', '--override-arch', 'amd64', 'quay.io/centos/centos:stream9@sha256:8e47e1ac4bfb7ae13d18403389d0ae0d6b44aa3c5a66e4561bc6cb881c140cd4', '/tmp/mock-tests-local-repo-29xlxzf_']' returned non-zero exit status 1.

    And a hermetic build is retriggered with the lockfile and repository                                # None
    Then the build succeeds                                                                             # None
    And the produced lockfile is validated properly                                                     # None

Feature: Test the "library" methods # features/library.feature:1

  @library @simple_load_config
  Scenario: The --resultdir option is incompatible with --chain                                    # features/library.feature:4

tkopecek · 2026-02-23T13:55:20Z

#1717 (comment) - interesting - works for me locally, I'll try to debug it more.

tkopecek · 2026-02-23T14:02:55Z

#1717 (comment) - interesting - works for me locally, I'll try to debug it more.

ouch, missed a file to commit

praiskup

Thank you!

praiskup · 2026-02-23T15:03:20Z

Would you mind squashing the PR?

praiskup · 2026-02-23T15:07:53Z

mock/py/mock-hermetic-repo.py

+    #            also it matches with full pullspec from config's bootstrap_image
+    # --preserve-digests - it is not needed for most images, but some could end with
+    #            modified digests (semantically identical copy, but modified digest,
+    #            e.g. embedded docker reference for v1 images), so better safe than sorry


These are in EL8, too 👍

tkopecek mentioned this pull request Feb 20, 2026

Drop pull_digest from hermetic boostrap image #1716

Closed

tkopecek force-pushed the hermetic-skopeo branch from eeb59fe to e98c1e6 Compare February 20, 2026 15:54

gemini-code-assist bot reviewed Feb 20, 2026

View reviewed changes

praiskup reviewed Feb 23, 2026

View reviewed changes

praiskup mentioned this pull request Feb 23, 2026

Don't install docs/weak deps konflux-ci/rpmbuild-pipeline-environment-container#125

Merged

praiskup approved these changes Feb 23, 2026

View reviewed changes

praiskup mentioned this pull request Feb 23, 2026

Podman save using id. #1642

Closed

praiskup reviewed Feb 23, 2026

View reviewed changes

Use skopeo for hermetic bootstrap images

22b6859

tkopecek force-pushed the hermetic-skopeo branch from 086e46b to 22b6859 Compare February 23, 2026 15:27

praiskup merged commit 649b22c into rpm-software-management:main Feb 23, 2026
31 of 32 checks passed

praiskup mentioned this pull request Feb 25, 2026

Loading saved image results in: Digest of source image's manifest would not match destination reference containers/podman#27323

Open

Conversation

tkopecek commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

tkopecek Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

praiskup commented Feb 23, 2026

Uh oh!

praiskup Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

tkopecek Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

praiskup Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

tkopecek Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

praiskup commented Feb 23, 2026

Uh oh!

tkopecek commented Feb 23, 2026

Uh oh!

tkopecek commented Feb 23, 2026

Uh oh!

praiskup left a comment

Choose a reason for hiding this comment

Uh oh!

praiskup commented Feb 23, 2026

Uh oh!

praiskup Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tkopecek commented Feb 20, 2026 •

edited

Loading