Skip to content

Commit cfb83e9

Browse files
mburke5678mburke5678
authored
Merge pull request #70881 from mburke5678/nodes-icsp-idms-interoperate
OCPSTRAT-898 - Need to ensure ICSP and IDMS interoperate especially in multi-tenant scenario
2 parents 2e50da9 + b088e08 commit cfb83e9

File tree

9 files changed

+307
-268
lines changed

9 files changed

+307
-268
lines changed

‎modules/builds-image-source.adoc

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,7 @@ source:
4343
<5> The location of the file to be copied out of the referenced image.
4444
<6> An optional secret provided if credentials are needed to access the input image.
4545
+
46-
[NOTE]
47-
====
48-
If your cluster uses an `ImageDigestMirrorSet` or `ImageTagMirrorSet` object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.
49-
====
46+
include::snippets/idms-global-pull-secret.adoc[]
5047

5148
.Images that require pull secrets
5249

‎modules/images-configuration-registry-mirror-configuring.adoc

Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * openshift_images/image-configuration.adoc
4+
// * post_installation_configuration/preparing-for-users.adoc
5+
// * updating/updating_a_cluster/updating_disconnected_cluster/disconnected-update.adoc
6+
7+
:_mod-docs-content-type: PROCEDURE
8+
[id="images-configuration-registry-mirror-configuring_{context}"]
9+
= Configuring image registry repository mirroring
10+
11+
You can create postinstallation mirror configuration custom resources (CR) to redirect image pull requests from a source image registry to a mirrored image registry.
12+
13+
.Prerequisites
14+
ifndef::openshift-rosa,openshift-dedicated[]
15+
* Access to the cluster as a user with the `cluster-admin` role.
16+
endif::openshift-rosa,openshift-dedicated[]
17+
ifdef::openshift-rosa,openshift-dedicated[]
18+
* Access to the cluster as a user with the `dedicated-admin` role.
19+
endif::openshift-rosa,openshift-dedicated[]
20+
21+
.Procedure
22+
23+
. Configure mirrored repositories, by either:
24+
+
25+
* Setting up a mirrored repository with Red Hat Quay, as described in link:https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/manage_red_hat_quay/repo-mirroring-in-red-hat-quay[Red Hat Quay Repository Mirroring]. Using Red Hat Quay allows you to copy images from one repository to another and also automatically sync those repositories repeatedly over time.
26+
27+
* Using a tool such as `skopeo` to copy images manually from the source repository to the mirrored repository.
28+
+
29+
For example, after installing the skopeo RPM package on a Red Hat Enterprise Linux (RHEL) 7 or RHEL 8 system, use the `skopeo` command as shown in this example:
30+
+
31+
[source,terminal]
32+
----
33+
$ skopeo copy \
34+
docker://registry.access.redhat.com/ubi9/ubi-minimal:latest@sha256:5cf... \
35+
docker://example.io/example/ubi-minimal
36+
----
37+
+
38+
In this example, you have a container image registry that is named `example.io` with an image repository named `example` to which you want to copy the `ubi9/ubi-minimal` image from `registry.access.redhat.com`. After you create the mirrored registry, you can configure your {product-title} cluster to redirect requests made of the source repository to the mirrored repository.
39+
40+
. Log in to your {product-title} cluster.
41+
42+
. Create a postinstallation mirror configuration CR, by using one of the following examples:
43+
44+
* Create an `ImageDigestMirrorSet` or `ImageTagMirrorSet` CR, as needed, replacing the source and mirrors with your own registry and repository pairs and images:
45+
+
46+
[source,yaml]
47+
----
48+
apiVersion: config.openshift.io/v1 <1>
49+
kind: ImageDigestMirrorSet <2>
50+
metadata:
51+
name: ubi9repo
52+
spec:
53+
imageDigestMirrors: <3>
54+
- mirrors:
55+
- example.io/example/ubi-minimal <4>
56+
- example.com/example/ubi-minimal <5>
57+
source: registry.access.redhat.com/ubi9/ubi-minimal <6>
58+
mirrorSourcePolicy: AllowContactingSource <7>
59+
- mirrors:
60+
- mirror.example.com/redhat
61+
source: registry.redhat.io/openshift4 <8>
62+
mirrorSourcePolicy: AllowContactingSource
63+
- mirrors:
64+
- mirror.example.com
65+
source: registry.redhat.io <9>
66+
mirrorSourcePolicy: AllowContactingSource
67+
- mirrors:
68+
- mirror.example.net/image
69+
source: registry.example.com/example/myimage <10>
70+
mirrorSourcePolicy: AllowContactingSource
71+
- mirrors:
72+
- mirror.example.net
73+
source: registry.example.com/example <11>
74+
mirrorSourcePolicy: AllowContactingSource
75+
- mirrors:
76+
- mirror.example.net/registry-example-com
77+
source: registry.example.com <12>
78+
mirrorSourcePolicy: AllowContactingSource
79+
----
80+
<1> Indicates the API to use with this CR. This must be `config.openshift.io/v1`.
81+
<2> Indicates the kind of object according to the pull type:
82+
** `ImageDigestMirrorSet`: Pulls a digest reference image.
83+
** `ImageTagMirrorSet`: Pulls a tag reference image.
84+
<3> Indicates the type of image pull method, either:
85+
** `imageDigestMirrors`: Use for an `ImageDigestMirrorSet` CR.
86+
** `imageTagMirrors`: Use for an `ImageTagMirrorSet` CR.
87+
<4> Indicates the name of the mirrored image registry and repository.
88+
<5> Optional: Indicates a secondary mirror repository for each target repository. If one mirror is down, the target repository can use another mirror.
89+
<6> Indicates the registry and repository source, which is the repository that is referred to in image pull specifications.
90+
<7> Optional: Indicates the fallback policy if the image pull fails:
91+
** `AllowContactingSource`: Allows continued attempts to pull the image from the source repository. This is the default.
92+
** `NeverContactSource`: Prevents continued attempts to pull the image from the source repository.
93+
<8> Optional: Indicates a namespace inside a registry, which allows you to use any image in that namespace. If you use a registry domain as a source, the object is applied to all repositories from the registry.
94+
<9> Optional: Indicates a registry, which allows you to use any image in that registry. If you specify a registry name, the object is applied to all repositories from a source registry to a mirror registry.
95+
<10> Pulls the image `registry.example.com/example/myimage@sha256:...` from the mirror `mirror.example.net/image@sha256:..`.
96+
<11> Pulls the image `registry.example.com/example/image@sha256:...` in the source registry namespace from the mirror `mirror.example.net/image@sha256:...`.
97+
<12> Pulls the image `registry.example.com/myimage@sha256` from the mirror registry `example.net/registry-example-com/myimage@sha256:...`.
98+
99+
* Create an `ImageContentSourcePolicy` custom resource, replacing the source and mirrors with your own registry and repository pairs and images:
100+
+
101+
[source,yaml]
102+
----
103+
apiVersion: operator.openshift.io/v1alpha1
104+
kind: ImageContentSourcePolicy
105+
metadata:
106+
name: mirror-ocp
107+
spec:
108+
repositoryDigestMirrors:
109+
- mirrors:
110+
- mirror.registry.com:443/ocp/release <1>
111+
source: quay.io/openshift-release-dev/ocp-release <2>
112+
- mirrors:
113+
- mirror.registry.com:443/ocp/release
114+
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
115+
----
116+
<1> Specifies the name of the mirror image registry and repository.
117+
<2> Specifies the online registry and repository containing the content that is mirrored.
118+
119+
. Create the new object:
120+
+
121+
[source,terminal]
122+
----
123+
$ oc create -f registryrepomirror.yaml
124+
----
125+
+
126+
After the object is created, the Machine Config Operator (MCO) drains the nodes for `ImageTagMirrorSet` objects only. The MCO does not drain the nodes for `ImageDigestMirrorSet` and `ImageContentSourcePolicy` objects.
127+
128+
. To check that the mirrored configuration settings are applied, do the following on one of the nodes.
129+
130+
.. List your nodes:
131+
+
132+
[source,terminal]
133+
----
134+
$ oc get node
135+
----
136+
+
137+
.Example output
138+
[source,terminal]
139+
----
140+
NAME STATUS ROLES AGE VERSION
141+
ip-10-0-137-44.ec2.internal Ready worker 7m v1.28.5
142+
ip-10-0-138-148.ec2.internal Ready master 11m v1.28.5
143+
ip-10-0-139-122.ec2.internal Ready master 11m v1.28.5
144+
ip-10-0-147-35.ec2.internal Ready worker 7m v1.28.5
145+
ip-10-0-153-12.ec2.internal Ready worker 7m v1.28.5
146+
ip-10-0-154-10.ec2.internal Ready master 11m v1.28.5
147+
----
148+
149+
.. Start the debugging process to access the node:
150+
+
151+
[source,terminal]
152+
----
153+
$ oc debug node/ip-10-0-147-35.ec2.internal
154+
----
155+
+
156+
.Example output
157+
[source,terminal]
158+
----
159+
Starting pod/ip-10-0-147-35ec2internal-debug ...
160+
To use host binaries, run `chroot /host`
161+
----
162+
163+
.. Change your root directory to `/host`:
164+
+
165+
[source,terminal]
166+
----
167+
sh-4.2# chroot /host
168+
----
169+
170+
.. Check the `/etc/containers/registries.conf` file to make sure the changes were made:
171+
+
172+
[source,terminal]
173+
----
174+
sh-4.2# cat /etc/containers/registries.conf
175+
----
176+
+
177+
The following output represents a `registries.conf` file where postinstallation mirror configuration CRs were applied. The final two entries are marked `digest-only` and `tag-only` respectively.
178+
+
179+
.Example output
180+
[source,terminal]
181+
----
182+
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
183+
short-name-mode = ""
184+
185+
[[registry]]
186+
prefix = ""
187+
location = "registry.access.redhat.com/ubi9/ubi-minimal" <1>
188+
189+
[[registry.mirror]]
190+
location = "example.io/example/ubi-minimal" <2>
191+
pull-from-mirror = "digest-only" <3>
192+
193+
[[registry.mirror]]
194+
location = "example.com/example/ubi-minimal"
195+
pull-from-mirror = "digest-only"
196+
197+
[[registry]]
198+
prefix = ""
199+
location = "registry.example.com"
200+
201+
[[registry.mirror]]
202+
location = "mirror.example.net/registry-example-com"
203+
pull-from-mirror = "digest-only"
204+
205+
[[registry]]
206+
prefix = ""
207+
location = "registry.example.com/example"
208+
209+
[[registry.mirror]]
210+
location = "mirror.example.net"
211+
pull-from-mirror = "digest-only"
212+
213+
[[registry]]
214+
prefix = ""
215+
location = "registry.example.com/example/myimage"
216+
217+
[[registry.mirror]]
218+
location = "mirror.example.net/image"
219+
pull-from-mirror = "digest-only"
220+
221+
[[registry]]
222+
prefix = ""
223+
location = "registry.redhat.io"
224+
225+
[[registry.mirror]]
226+
location = "mirror.example.com"
227+
pull-from-mirror = "digest-only"
228+
229+
[[registry]]
230+
prefix = ""
231+
location = "registry.redhat.io/openshift4"
232+
233+
[[registry.mirror]]
234+
location = "mirror.example.com/redhat"
235+
pull-from-mirror = "digest-only"
236+
[[registry]]
237+
prefix = ""
238+
location = "registry.access.redhat.com/ubi9/ubi-minimal"
239+
blocked = true <4>
240+
241+
[[registry.mirror]]
242+
location = "example.io/example/ubi-minimal-tag"
243+
pull-from-mirror = "tag-only" <5>
244+
----
245+
<1> Indicates the repository that is referred to in a pull spec.
246+
<2> Indicates the mirror for that repository.
247+
<3> Indicates that the image pull from the mirror is a digest reference image.
248+
<4> Indicates that the `NeverContactSource` parameter is set for this repository.
249+
<5> Indicates that the image pull from the mirror is a tag reference image.
250+
251+
.. Pull an image to the node from the source and check if it is resolved by the mirror.
252+
+
253+
[source,terminal]
254+
----
255+
sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi9/ubi-minimal@sha256:5cf...
256+
----
257+
258+
.Troubleshooting repository mirroring
259+
260+
If the repository mirroring procedure does not work as described, use the following information about how repository mirroring works to help troubleshoot the problem.
261+
262+
* The first working mirror is used to supply the pulled image.
263+
* The main registry is only used if no other mirror works.
264+
* From the system context, the `Insecure` flags are used as fallback.
265+
* The format of the `/etc/containers/registries.conf` file has changed recently. It is now version 2 and in TOML format.

‎modules/images-configuration-registry-mirror-convert.adoc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ Using an `ImageContentSourcePolicy` (ICSP) object to configure repository mirror
1212

1313
ICSP objects are being replaced by `ImageDigestMirrorSet` and `ImageTagMirrorSet` objects to configure repository mirroring. If you have existing YAML files that you used to create `ImageContentSourcePolicy` objects, you can use the `oc adm migrate icsp` command to convert those files to an `ImageDigestMirrorSet` YAML file. The command updates the API to the current version, changes the `kind` value to `ImageDigestMirrorSet`, and changes `spec.repositoryDigestMirrors` to `spec.imageDigestMirrors`. The rest of the file is not changed.
1414

15+
Because the migration does not change the `registries.conf` file, the cluster does not need to reboot.
16+
1517
For more information about `ImageDigestMirrorSet` or `ImageTagMirrorSet` objects, see "Configuring image registry repository mirroring" in the previous section.
1618

1719
.Prerequisites
@@ -69,3 +71,6 @@ where:
6971
`<file_name>`:: Specifies the name of the `ImageDigestMirrorSet` YAML.
7072
--
7173

74+
. Remove the ICSP objects after the IDMS objects are rolled out.
75+
76+

0 commit comments

Comments
 (0)