OCPBUGS-56451: Use the ImageDigest SourcePolicy to configure image registry on bootstrap host #10073

sadasu · 2025-11-12T00:17:49Z

A new policy was added to the ImageDigestSource configuration that allows the user to specify policy when there is a failure pulling an image from the source.

Update the image registry configuration on the bootstrap host with this SourcePolicy.

openshift-ci-robot · 2025-11-12T00:17:57Z

@sadasu: This pull request references Jira Issue OCPBUGS-56451, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @gpei

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

A new policy was added to the ImageDigestSource configuration that allows the user to specify policy when there is a failure pulling an image from the source.

Update the image registry configuration on the bootstrap host with this SourcePolicy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tthvo · 2025-11-12T00:30:33Z

/cc @patrickdillon

tthvo

I followed the below steps to test this PR:

With the install-config block:

imageDigestSources:
- mirrors:
  - ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release
  source: registry.build10.ci.openshift.org/ci-ln-wmr7tpk/release
  sourcePolicy: NeverContactSource
- mirrors:
  - ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release
  source: registry.build10.ci.openshift.org/ci-ln-wmr7tpk/stable

Generating the ignition-configs:

./openshift-install create ignition-configs --dir=.

Check the content of bootstrap ignition:

 cat ./bootstrap.ign | jq -r '.storage.files[] | select(.path=="/etc/containers/registries.conf") | .contents.source' | cut -d , -f2 | base64 -d

I did see the blocked field, but unfortunately, it is always false regardless of source policy. I have a few suggestions above that will help 🤔

pkg/asset/imagebased/image/registriesconf.go

tthvo · 2025-11-12T02:21:22Z

With the suggestions above, I followed the same steps as #10073 (comment) and I can now see the correct registries.conf:

$ cat ./bootstrap.ign | jq -r '.storage.files[] | select(.path=="/etc/containers/registries.conf") | .contents.source' | cut -d , -f2 | base64 -d
[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-wmr7tpk/release"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release"
insecure = false


[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-wmr7tpk/stable"
insecure = false
mirror-by-digest-only = true
blocked = false

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release"
insecure = false

gpei · 2025-11-12T08:06:41Z

I did see the blocked field, but unfortunately, it is always false regardless of source policy.

+1, when the sourcePolicy is set to NeverContactSource to prevent pulling images from the original registry, the file /etc/containers/registries.conf on bootstrap remains in this state:

[root@ip-10-0-59-60 ~]# cat /etc/containers/registries.conf
[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-sp22d22/release"
insecure = false
mirror-by-digest-only = true
blocked = false

[[registry.mirror]]
location = "xx/ci-ln-sp22d22/release"
insecure = false


[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-sp22d22/stable"
insecure = false
mirror-by-digest-only = true
blocked = false

[[registry.mirror]]
location = "xx/ci-ln-sp22d22/release"
insecure = false

A new policy was added to the ImageDigestSource configuration that allows the user to specify policy when there is a failure pulling an image from the source. Update the image registry configuration on the bootstrap host with this SourcePolicy.

Update the image registry configuration with the newly added Image source policy field that specifies the fallback policy when image pull fails.

With the introduction of the fallback source policy when Image source and mirrors are specified, image registry configuration for ABI should be update to include that.

pkg/asset/ignition/bootstrap/registries.go

tthvo · 2025-11-14T00:51:47Z

Side note: I just realized we can use struct as key in golang https://go.dev/blog/maps 🤣 wow, so much more to learn haha

sadasu · 2025-11-14T17:41:08Z

@andfasano and @mresvanis could you PTAL?

tthvo · 2025-11-14T20:57:04Z

/test e2e-aws-ovn

tthvo

/lgtm

This worked for me 🥳 With the install-config snippet:

imageDigestSources:
- mirrors:
  - ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release
  source: registry.build10.ci.openshift.org/ci-ln-wmr7tpk/release
  sourcePolicy: NeverContactSource
- mirrors:
  - ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release
  source: registry.build10.ci.openshift.org/ci-ln-wmr7tpk/stable

The following bootstrap registry config is generated:

cat ./bootstrap.ign | jq -r '.storage.files[] | select(.path=="/etc/containers/registries.conf") | .contents.source' | cut -d , -f2 | base64 -d
[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-wmr7tpk/release"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release"
insecure = false


[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-wmr7tpk/stable"
insecure = false
mirror-by-digest-only = true
blocked = false

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-wmr7tpk/release"
insecure = false

tthvo · 2025-11-16T05:58:23Z

/test e2e-aws-ovn

gpei · 2025-11-17T03:29:52Z

@tthvo Thanks for helping to verify the generated content of registry.conf in these two scenarios.
I made an additional verification step, which confirmed that when setting to 'block', the file ultimately created on the bootstrap node is expected and the container runtime are indeed unable to pull images from the blocked registry address.

[root@ip-10-0-52-101 ~]# cat /etc/containers/registries.conf
[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-52ibs72/release"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-52ibs72/release"
insecure = false


[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-52ibs72/stable"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-52ibs72/release"
insecure = false


[root@ip-10-0-52-101 ~]# crictl pull registry.build10.ci.openshift.org/ci-ln-52ibs72/release
E1117 03:17:43.722036    5257 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/release:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d; artifact err: get manifest: build image source: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d" image="registry.build10.ci.openshift.org/ci-ln-52ibs72/release"
FATA[0000] pulling image: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/release:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d; artifact err: get manifest: build image source: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d 

[root@ip-10-0-52-101 ~]# podman pull registry.build10.ci.openshift.org/ci-ln-52ibs72/stable
Trying to pull registry.build10.ci.openshift.org/ci-ln-52ibs72/stable:latest...
Error: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/stable:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/stable is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d

/verified by @gpei

openshift-ci-robot · 2025-11-17T03:30:02Z

@gpei: This PR has been marked as verified by @gpei.

In response to this:

@tthvo Thanks for helping to verify the generated content of registry.conf in these two scenarios.
I made an additional verification step, which confirmed that when setting to 'block', the file ultimately created on the bootstrap node is expected and the container runtime are indeed unable to pull images from the blocked registry address.

[root@ip-10-0-52-101 ~]# cat /etc/containers/registries.conf
[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-52ibs72/release"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-52ibs72/release"
insecure = false


[[registry]]
location = "registry.build10.ci.openshift.org/ci-ln-52ibs72/stable"
insecure = false
mirror-by-digest-only = true
blocked = true

[[registry.mirror]]
location = "ec2-xxx.compute-1.amazonaws.com:5000/ci-ln-52ibs72/release"
insecure = false


[root@ip-10-0-52-101 ~]# crictl pull registry.build10.ci.openshift.org/ci-ln-52ibs72/release
E1117 03:17:43.722036    5257 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/release:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d; artifact err: get manifest: build image source: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d" image="registry.build10.ci.openshift.org/ci-ln-52ibs72/release"
FATA[0000] pulling image: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/release:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d; artifact err: get manifest: build image source: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/release is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d 

[root@ip-10-0-52-101 ~]# podman pull registry.build10.ci.openshift.org/ci-ln-52ibs72/stable
Trying to pull registry.build10.ci.openshift.org/ci-ln-52ibs72/stable:latest...
Error: initializing source docker://registry.build10.ci.openshift.org/ci-ln-52ibs72/stable:latest: registry registry.build10.ci.openshift.org/ci-ln-52ibs72/stable is blocked in /etc/containers/registries.conf or /root/.config/containers/registries.conf.d

/verified by @gpei

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

patrickdillon · 2025-11-18T16:33:50Z

/approve

openshift-ci · 2025-11-18T16:40:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [patrickdillon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andfasano · 2025-11-18T17:23:50Z

Changes look good also from ABI point of view

openshift-ci · 2025-11-18T21:35:25Z

@sadasu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agent-two-node-fencing-ipv4	`4752371`	link	false	`/test e2e-agent-two-node-fencing-ipv4`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-11-18T21:54:44Z

/retest-required

Remaining retests: 0 against base HEAD ccad892 and 2 for PR HEAD 4752371 in total

openshift-ci-robot · 2025-11-19T01:16:57Z

@sadasu: Jira Issue Verification Checks: Jira Issue OCPBUGS-56451
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-56451 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

In response to this:

A new policy was added to the ImageDigestSource configuration that allows the user to specify policy when there is a failure pulling an image from the source.

Update the image registry configuration on the bootstrap host with this SourcePolicy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-11-23T14:24:07Z

Fix included in accepted release 4.21.0-0.nightly-2025-11-22-193140

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Nov 12, 2025

openshift-ci bot requested a review from gpei November 12, 2025 00:18

sadasu force-pushed the fix-mirrorSourcePolicy branch from 52046fe to 99d43aa Compare November 12, 2025 00:18

openshift-ci bot requested review from bfournie and javipolo November 12, 2025 00:20

openshift-ci bot requested a review from patrickdillon November 12, 2025 00:30

tthvo reviewed Nov 12, 2025

View reviewed changes

pkg/asset/imagebased/image/registriesconf.go Show resolved Hide resolved

pkg/asset/imagebased/image/registriesconf.go Show resolved Hide resolved

sadasu added 3 commits November 13, 2025 16:18

Update Image registry configuration for image based install

fb38639

Update the image registry configuration with the newly added Image source policy field that specifies the fallback policy when image pull fails.

Update Image registry configuration during ABI

a27c0c7

With the introduction of the fallback source policy when Image source and mirrors are specified, image registry configuration for ABI should be update to include that.

sadasu force-pushed the fix-mirrorSourcePolicy branch from 99d43aa to 7f87dd6 Compare November 13, 2025 22:31

tthvo reviewed Nov 14, 2025

View reviewed changes

pkg/asset/ignition/bootstrap/registries.go Outdated Show resolved Hide resolved

pkg/asset/ignition/bootstrap/registries.go Outdated Show resolved Hide resolved

sadasu force-pushed the fix-mirrorSourcePolicy branch from 7f87dd6 to be61527 Compare November 14, 2025 16:24

Fix linter errors

4752371

sadasu force-pushed the fix-mirrorSourcePolicy branch from be61527 to 4752371 Compare November 14, 2025 17:22

tthvo reviewed Nov 14, 2025

View reviewed changes

openshift-ci bot assigned tthvo Nov 14, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 14, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Nov 17, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 18, 2025

openshift-merge-bot bot merged commit 02535f7 into openshift:main Nov 19, 2025
27 of 28 checks passed

OCPBUGS-56451: Use the ImageDigest SourcePolicy to configure image registry on bootstrap host #10073

OCPBUGS-56451: Use the ImageDigest SourcePolicy to configure image registry on bootstrap host #10073

Uh oh!

Conversation

sadasu commented Nov 12, 2025

Uh oh!

openshift-ci-robot commented Nov 12, 2025

Uh oh!

tthvo commented Nov 12, 2025

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tthvo commented Nov 12, 2025

Uh oh!

gpei commented Nov 12, 2025

Uh oh!

Uh oh!

Uh oh!

tthvo commented Nov 14, 2025

Uh oh!

sadasu commented Nov 14, 2025

Uh oh!

tthvo commented Nov 14, 2025

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

tthvo commented Nov 16, 2025

Uh oh!

gpei commented Nov 17, 2025

Uh oh!

openshift-ci-robot commented Nov 17, 2025

Uh oh!

patrickdillon commented Nov 18, 2025

Uh oh!

openshift-ci bot commented Nov 18, 2025

Uh oh!

andfasano commented Nov 18, 2025

Uh oh!

openshift-ci bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 18, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 19, 2025

Uh oh!

openshift-merge-robot commented Nov 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

openshift-ci bot commented Nov 18, 2025 •

edited

Loading