|
| 1 | +--- |
| 2 | +title: "Cannot Pull Image" |
| 3 | +date: 2019-03-23T08:08:19-04:00 |
| 4 | +draft: false |
| 5 | +--- |
| 6 | + |
| 7 | +> My domain will not start and I see errors like `ImagePullBackoff` or `Cannot pull image` |
| 8 | +
|
| 9 | +When you see these kinds of errors it means that Kubernetes cannot find your Docker image. |
| 10 | +The most common causes are: |
| 11 | + |
| 12 | +* The `image` value in your domain resource is set incorrectly, meaning Kubernetes will be |
| 13 | + trying to pull the wrong image |
| 14 | +* The image requires authentication or permission in order to pull it and you have not |
| 15 | + configured Kubernetes with the necessary credentials, for example in an `imagePullSecret` |
| 16 | +* You built the image on a machine that is not where your `kubelet` is running and Kubernetes |
| 17 | + cannot see the image, meaning you need to copy the image to the worker nodes or put it in |
| 18 | + a Docker registry that is accessible the to all of the worker nodes |
| 19 | + |
| 20 | +Let's review what happens when Kubernetes starts a pod. |
| 21 | + |
| 22 | + |
| 23 | + |
| 24 | +The definition of the pod contains a list of container specifications. Each container |
| 25 | +specificiation contains the name (and optionally tag) of the image that should be used |
| 26 | +to run that container. In the example above, there is a container called `c1` which is |
| 27 | +configured to use the Docker image `some.registry.com/owner/domain1:1.0`. This image |
| 28 | +name is in the format `registry address / owner / name : tag`, so in this case the |
| 29 | +registry is `some.registry.com`, then owner is `owner`, the image name is `domain` |
| 30 | +and the tag is `1.0`. Tags are a lot like version numbers, but they are not required |
| 31 | +to be numbers or to be in any particular sequence or format. If you omit the tag, it |
| 32 | +is assumed to be `latest`. |
| 33 | + |
| 34 | +{{% notice note %}} |
| 35 | +The Docker tag `latest` is confusing - it does not actually mean the latest version of |
| 36 | +the image that was created or published in the registry, it just literally means |
| 37 | +whichever version the owner decided to call "latest". Docker and Kubernetes make |
| 38 | +some assumptions about latest, and it is generally recommended to avoid using it and instead |
| 39 | +specify the actual version/tag that you really want. |
| 40 | +{{% /notice %}} |
| 41 | + |
| 42 | +First, Kubernetes will check to see if the requested image is available in the local |
| 43 | +Docker image store on whichever worker node the pod was scheduled on. If it is there, |
| 44 | +then it will use that image to start the container. If it is not there, then Kubernetes |
| 45 | +will attempt to pull the image from a remote Docker registry. |
| 46 | + |
| 47 | +{{% notice note %}} |
| 48 | +There is another setting called `imagePullPolicy` that can be used to force Kubernetes |
| 49 | +to always pull the image, even if it is already present in the local Docker image |
| 50 | +store. |
| 51 | +{{% /notice %}} |
| 52 | + |
| 53 | +If the image is available in the remote registry and it is public, that is it does not |
| 54 | +require authentication, then Kubernetes will pull the image to the local Docker image |
| 55 | +store and start the container. |
| 56 | + |
| 57 | +#### Images that require authentication |
| 58 | + |
| 59 | +If the remote Docker registry requires authentication, then you will need to provide |
| 60 | +the authentication details in a Kubernetes `docker-registry` secret and tell Kubernetes |
| 61 | +to use that secret when pulling the image. |
| 62 | + |
| 63 | +To create a secret, you can use the following command: |
| 64 | + |
| 65 | +``` |
| 66 | +kubectl create secret docker-registry secret1 \ |
| 67 | + --docker-server=some.registry.com \ |
| 68 | + --docker-username=bob \ |
| 69 | + --docker-password=bigSecret \ |
| 70 | + |
| 71 | + --namespace=default |
| 72 | +``` |
| 73 | + |
| 74 | +In this command, you would replace `secret1` with the name of the secret; the `docker-server` |
| 75 | +is set to the registry name, without the `https://` prefix; the `docker-username`, `docker-password` |
| 76 | +and `docker-email` are set to match the credentials you use to authenticate to the remote |
| 77 | +Docker registry; and the `namespace` must be set to the same namespace where you intend to |
| 78 | +use the image. |
| 79 | + |
| 80 | +{{% notice note %}} |
| 81 | +Some registries may need a suffix making the `docker-server` something like `some.registry.com/v2` |
| 82 | +for example. You will need to check with your registry provider's documentation to see if this is needed. |
| 83 | +{{% /notice %}} |
| 84 | + |
| 85 | +After the secret is created, you need to tell Kubernetes to use it. This is done by adding |
| 86 | +an `imagePullSecret` to your Kubernetes YAML file. In the case of a WebLogic domain, you |
| 87 | +add the secret name to the `imagePullSecret` in the domain custom resource YAML file. |
| 88 | + |
| 89 | +Here is an example of part of a domain custom resource file with the `imagePullSecret` above |
| 90 | +specified: |
| 91 | + |
| 92 | +``` |
| 93 | +apiVersion: "weblogic.oracle/v2" |
| 94 | +kind: Domain |
| 95 | +metadata: |
| 96 | + name: domain1 |
| 97 | + namespace: default |
| 98 | + labels: |
| 99 | + weblogic.resourceVersion: domain-v2 |
| 100 | + weblogic.domainUID: domain1 |
| 101 | +spec: |
| 102 | + domainHome: /u01/oracle/user_projects/domains/domain1 |
| 103 | + domainHomeInImage: true |
| 104 | + image: "some.registry.com/owner/domain1:1.0" |
| 105 | + imagePullPolicy: "IfNotPresent" |
| 106 | + imagePullSecrets: |
| 107 | + - name: secret1 |
| 108 | +``` |
| 109 | + |
| 110 | +Alternatively, you can associate the secret with the service account that will be used to run |
| 111 | +the pod. If you do this, then you will not need to add the `imagePullSecret` to the domain |
| 112 | +resource. This is useful if you are running multiple domains in the same namespace. |
| 113 | + |
| 114 | +To add the secret shown above to the `default` service account in the `weblogic` namespace, you |
| 115 | +would use a command like this: |
| 116 | + |
| 117 | +``` |
| 118 | +kubectl patch serviceaccount default \ |
| 119 | + -n weblogic \ |
| 120 | + -p '{"imagePullSecrets": [{"name": "secret1"}]}' |
| 121 | +``` |
| 122 | + |
| 123 | +{{% notice note %}} |
| 124 | +You can provide mutliple `imagePullSecrets` if you need to pull Docker images from multiple |
| 125 | +remote Docker registries or if your images require different authentication credentials. |
| 126 | +For more information, see [Docker Image Protection]({{<relref "/security/domain-security/image-protection.md#weblogic-domain-in-docker-image-protection">}}) |
| 127 | +under **Security**. |
| 128 | +{{% /notice %}} |
| 129 | + |
| 130 | +#### Manually copying the image to your worker nodes |
| 131 | + |
| 132 | +If you are not able to use a remote Docker registry, for example if your Kubernetes cluster is |
| 133 | +in a secure network with no external access, you can manually copy the Docker images to the |
| 134 | +cluster instead. |
| 135 | + |
| 136 | +On the machine where you created the image, export it into a tar file using this command: |
| 137 | + |
| 138 | +``` |
| 139 | +docker save domain1:1.0 > domain1.tar |
| 140 | +``` |
| 141 | + |
| 142 | +Then copy that tar file to each worker node in your Kubernetes cluster and run this command |
| 143 | +on each node: |
| 144 | + |
| 145 | +``` |
| 146 | +docker load < domain1.tar |
| 147 | +``` |
| 148 | + |
| 149 | +#### Restart pods to clear the error |
| 150 | + |
| 151 | +After you have ensured that the images are accessible on all worker nodes, you may need to restart |
| 152 | +the pods so that Kubernetes will attempt to pull the images again. You can do this by |
| 153 | +deleting the pods themselves, or deleting the domain resource and then recreating it. |
0 commit comments