image not updated in openshift deployment

[claudio4j]

hi, I'm trying to troubleshoot an issue with openshift deployment. We have two openshift clusters, let's say ocp-1 and ocp-2.

When I deploy to ocp-1, the oc:deploy task sets the container image address in target/jkube/applyJson/cmiranda-dev/deployment-httplog-sandbox.json, which in turn sets this image in the target/kubernetes/openshift.yml

image" : "image-registry.openshift-image-registry.svc:5000/cmiranda-dev/httplog-sandbox@sha256:b322a040fd330f940ffc907348d27306a7c2d0cef995eae0683c34a4d66fd5d2",

Then, the pod starts with the above mentioned image and works correctly.

However when deploying to ocp-2, the same project and parameters, the container image is not updated in target/jkube/applyJson/default/deployment-httplog-shared.json, resulting with the Deployment in target/kubernetes/openshift.yml containing the image: httplog-shared:1.0-SNAPSHOT, then the pod fails to start with:

Failed to pull image "httplog-shared:1.0-SNAPSHOT": reading manifest 1.0-SNAPSHOT in docker.io/library/httplog-shared: requested access to the resource is denied

This is the maven output

[INFO] --- oc:1.18.1:build (default) @ httplog-shared ---
[INFO] oc: Using OpenShift build with strategy Docker
[INFO] oc: [httplog-shared:1.0-SNAPSHOT]: Created docker source tar /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/docker/httplog-shared/1.0-SNAPSHOT/tmp/docker-build.tar
[INFO] oc: Creating BuildServiceConfig httplog-shared-s2i for Docker build
[INFO] oc: Creating ImageStream httplog-shared
[INFO] oc: Starting Build httplog-shared-s2i
[INFO] oc: Waiting for build httplog-shared-s2i-1 to complete...
[INFO] oc: Adding cluster TLS certificate authority to trust store
[INFO] oc: Receiving source from STDIN as archive ...
[INFO] oc: Adding cluster TLS certificate authority to trust store
[INFO] oc: Replaced Dockerfile FROM image mirror.gcr.io/library/eclipse-temurin:17
[INFO] oc: Adding cluster TLS certificate authority to trust store
[INFO] oc: time="2025-03-11T19:34:58Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
[INFO] oc: I0311 19:34:58.979832       1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
[INFO] oc: Caching blobs under "/var/cache/blobs".
[INFO] oc: 
[INFO] oc: Pulling image mirror.gcr.io/library/eclipse-temurin:17 ...
[INFO] oc: Trying to pull mirror.gcr.io/library/eclipse-temurin:17...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:3d2700d3f64bcbf1b67ccf65a715330aa34cb16c0c1e8c9146c15011233d6fc4
[INFO] oc: Copying blob sha256:5a7813e071bfadf18aaa6ca8318be4824a9b6297b3240f2cc84c1db6f4113040
[INFO] oc: Copying blob sha256:0b4fff2c206cae13745f200704b38134105ac3a4bcf862dc0d04d08f6aa9d2ce
[INFO] oc: Copying blob sha256:a2ff804eb32a079e6f9a54eb31c0f43d9d6c56fc09c7735ec59e2ad948fc6af6
[INFO] oc: Copying blob sha256:da5768705e3b7e7e3c61f95a6ed0478a61076d084e1adedfe24e09ce95d5c433
[INFO] oc: Copying config sha256:3003266067237a5e9757aa2256c9b253e916822545cc4c14d593c59cfb35a8da
[INFO] oc: Writing manifest to image destination
[INFO] oc: Adding transient rw bind mount for /run/secrets/rhsm
[INFO] oc: STEP 1/6: FROM mirror.gcr.io/library/eclipse-temurin:17
[INFO] oc: STEP 2/6: COPY /quarkus-files/maven /maven/
[INFO] oc: --> 030158805d01
[INFO] oc: STEP 3/6: COPY /jkube-generated-layer-final-artifact/maven /maven/
[INFO] oc: --> f2ee57eb0537
[INFO] oc: STEP 4/6: ENTRYPOINT ["java","-jar","/maven/quarkus/quarkus-run.jar"]
[INFO] oc: --> 80aff276d675
[INFO] oc: STEP 5/6: ENV "OPENSHIFT_BUILD_NAME"="httplog-shared-s2i-1" "OPENSHIFT_BUILD_NAMESPACE"="default"
[INFO] oc: --> cd51e4eb1de4
[INFO] oc: STEP 6/6: LABEL "io.openshift.build.name"="httplog-shared-s2i-1" "io.openshift.build.namespace"="default"
[INFO] oc: COMMIT temp.builder.openshift.io/default/httplog-shared-s2i-1:0695c213
[INFO] oc: --> 2f05082d130b
[INFO] oc: Successfully tagged temp.builder.openshift.io/default/httplog-shared-s2i-1:0695c213
[INFO] oc: 2f05082d130bbb778a343660881357d7371322273aa934a5807106ae78f85e09
[INFO] oc: 
[INFO] oc: Pushing image image-registry.openshift-image-registry.svc:5000/default/httplog-shared:1.0-SNAPSHOT ...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:25af1d37db6cae787247ba3c392ee913a823d29c92ec14ba776a7b081aa9d468
[INFO] oc: Copying blob sha256:a2ff804eb32a079e6f9a54eb31c0f43d9d6c56fc09c7735ec59e2ad948fc6af6
[INFO] oc: Copying blob sha256:5a7813e071bfadf18aaa6ca8318be4824a9b6297b3240f2cc84c1db6f4113040
[INFO] oc: Copying blob sha256:0b4fff2c206cae13745f200704b38134105ac3a4bcf862dc0d04d08f6aa9d2ce
[INFO] oc: Copying blob sha256:da5768705e3b7e7e3c61f95a6ed0478a61076d084e1adedfe24e09ce95d5c433
[INFO] oc: Copying blob sha256:3d2700d3f64bcbf1b67ccf65a715330aa34cb16c0c1e8c9146c15011233d6fc4
[INFO] oc: Copying blob sha256:0c9d9006f713f398f7adc935f9c85feec93e8439e84c741094a97a353eb4b2f4
[INFO] oc: Copying config sha256:2f05082d130bbb778a343660881357d7371322273aa934a5807106ae78f85e09
[INFO] oc: Writing manifest to image destination
[INFO] oc: Successfully pushed image-registry.openshift-image-registry.svc:5000/default/httplog-shared@sha256:d52babe8b5e35591fdb5713d3d2f1228f2ad9b79fc8a3098026133a34183b5e3
[INFO] oc: Push successful
[INFO] oc: Build httplog-shared-s2i-1 in status Complete
[INFO] oc: Found tag on ImageStream httplog-shared tag: sha256:d52babe8b5e35591fdb5713d3d2f1228f2ad9b79fc8a3098026133a34183b5e3
[INFO] oc: ImageStream httplog-shared written to /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/httplog-shared-is.yml
[INFO] 
[INFO] --- oc:1.18.1:resource (default) @ httplog-shared ---
[INFO] oc: Using container image name of namespace: default
[INFO] oc: Using resource templates from /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/src/main/jkube
[INFO] oc: jkube-healthcheck-quarkus: Adding readiness probe on port 8080, path='/observe/health/ready', scheme='HTTP', with initial delay 5 seconds
[INFO] oc: jkube-healthcheck-quarkus: Adding liveness probe on port 8080, path='/observe/health/live', scheme='HTTP', with initial delay 10 seconds
[INFO] oc: jkube-healthcheck-quarkus: Adding startup probe on port 8080, path='/observe/health/started', scheme='HTTP', with initial delay 5 seconds
[INFO] oc: jkube-service-discovery: Using first mentioned service port '80' 
[INFO] oc: jkube-revision-history: Adding revision history limit to 2
[INFO] oc: validating /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/classes/META-INF/jkube/openshift/httplog-shared-deployment.yml resource
[WARNING] Unknown keyword serializer - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
[WARNING] Unknown keyword deserializer - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
[INFO] oc: validating /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/classes/META-INF/jkube/openshift/httplog-shared-service.yml resource
[INFO] oc: validating /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/classes/META-INF/jkube/openshift/httplog-shared-route.yml resource
[INFO] 
[INFO] --- oc:1.18.1:push (default) @ httplog-shared ---
[WARNING] oc: Image is pushed to OpenShift's internal registry during oc:build goal. Skipping...
[INFO] 
[INFO] --- resources:3.3.1:copy-resources (default) @ httplog-shared ---
[INFO] Copying 4 resources from target/classes/META-INF/jkube to target/kubernetes
[INFO] 
[INFO] --- failsafe:3.5.2:integration-test (default) @ httplog-shared ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- failsafe:3.5.2:verify (default) @ httplog-shared ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- install:3.1.2:install (default-install) @ httplog-shared ---
[INFO] Installing /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/pom.xml to /home/claudio/.m2/repository/org/example/project/httplog-shared/1.0-SNAPSHOT/httplog-shared-1.0-SNAPSHOT.pom
[INFO] Installing /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/httplog-shared-1.0-SNAPSHOT.jar to /home/claudio/.m2/repository/org/example/project/httplog-shared/1.0-SNAPSHOT/httplog-shared-1.0-SNAPSHOT.jar
[INFO] Installing /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/httplog-shared-is.yml to /home/claudio/.m2/repository/org/example/project/httplog-shared/1.0-SNAPSHOT/httplog-shared-1.0-SNAPSHOT-is.yml
[INFO] Installing /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/classes/META-INF/jkube/openshift.yml to /home/claudio/.m2/repository/org/example/project/httplog-shared/1.0-SNAPSHOT/httplog-shared-1.0-SNAPSHOT-openshift.yml
[INFO] 
[INFO] <<< oc:1.18.1:deploy (default-cli) < install @ httplog-shared <<<
[INFO] 
[INFO] 
[INFO] --- oc:1.18.1:deploy (default-cli) @ httplog-shared ---
[INFO] oc: OpenShift platform detected
[INFO] oc: Using OpenShift at https://api.shrocp4upi417ovn.lab.upshift.rdu2.redhat.com:6443/ in namespace null with manifest /home/claudio/alphaworks/projects/camel/.camel-jbang-run/httplog-shared/target/classes/META-INF/jkube/openshift.yml 
[INFO] oc: Creating a Service in default namespace with name httplog-shared from openshift.yml
[INFO] oc: Created Service: target/jkube/applyJson/default/service-httplog-shared.json
[INFO] oc: Creating a Deployment in default namespace with name httplog-shared from openshift.yml
[INFO] oc: Created Deployment: target/jkube/applyJson/default/deployment-httplog-shared.json
[INFO] oc: Creating Route default:httplog-shared host: null
[INFO] oc: HINT: Use the command `oc get pods -w` to watch your pods start up

You can see the image was pushed to

Successfully pushed image-registry.openshift-image-registry.svc:5000/default/httplog-shared@sha256:d52babe8b5e35591fdb5713d3d2f1228f2ad9b79fc8a3098026133a34183b5e3

But in the end, the container image was not updated in the deployment for the oc:deploy task.

So, do you have any hint where to look or debug at ?

[manusa]

It looks like the logs for the second cluster are missing the call to oc:resource.
This means that the YAML resources aren't regenerated, hence the problem you describe.

How are you calling Maven for the second cluster? is it a single call of all the log that we see? or have you merged the outputs of two separate invocations?

[claudio4j]

The mvn commands are the same for both cluster, I just change the oc login to each different cluster. The mvn command is mvn oc:deploy whose jkube openshift-maven-plugin configuration is:

<executions>
    <execution>
        <goals>
            <goal>build</goal>
            <goal>resource</goal>
            <goal>push</goal>
        </goals>
        <phase>package</phase>
    </execution>
</executions>

with properties:

<jkube.image.name>httplog-qe:1.0-SNAPSHOT</jkube.image.name>
<jkube.container-image.name>httplog-qe:1.0-SNAPSHOT</jkube.container-image.name>
<jkube.skip.push>false</jkube.skip.push>
<jkube.build.switchToDeployment>true</jkube.build.switchToDeployment>
<jkube.base.image>mirror.gcr.io/library/eclipse-temurin:17</jkube.base.image>
<jkube.build.strategy>docker</jkube.build.strategy>
<jkube.version>1.18.1</jkube.version>

I had debugged the code and noticed the ApplyService.java in the doCreate method (line 785), when it invokes answer = kubernetesClient.resource(resource).inNamespace(namespace).create(); the resource represents the Deployment whose image is httplog-shared:1.0-SNAPSHOT, then after it is created, the answer Deployment, the image field is updated to the correct image-registry.openshift-image-registry.svc:5000/cmiranda-dev/httplog-sandbox@sha256:b322a040fd330f940ffc907348d27306a7c2d0cef995eae0683c34a4d66fd5d2, this is for the ocp-1 cluster.

So, the Deployment is created with image: httplog-shared:1.0-SNAPSHOT, but somehow the deployment on cluster side is created with the image-registry.openshift-image-registry.svc:5000/cmiranda-dev/httplog-sandbox@sha256:b322a040fd330f940ffc907348d27306a7c2d0cef995eae0683c34a4d66fd5d2

I think this is not jkube conf, but some configuration on our cluster, I will compare other settings, but it's strange.

I will close this question.

[claudio4j]

So, to summarize, the behavior

ocp-1 cluster

// before create invocation
resource - image: httplog-shared:1.0-SNAPSHOT
answer = kubernetesClient.resource(resource).inNamespace(namespace).create();

// after
answer - image: image-registry.openshift-image-registry.svc:5000/cmiranda-dev/httplog-sandbox@sha256:b322a040fd330f940ffc907348d27306a7c2d0cef995eae0683c34a4d66f

ocp-2 cluster

// before create invocation
resource - image: httplog-shared:1.0-SNAPSHOT
answer = kubernetesClient.resource(resource).inNamespace(namespace).create();

// after
answer - image: httplog-shared:1.0-SNAPSHOT

Do you have any idea that could make the creation method on the ocp-2 cluster, to not update the image field of the Deployment, in the answer object ?

I'm trying to understanding some server side kubectl api configuration, but nothing concrete yet.

[manusa]

I guess there might be some controller in cluster 1 that's performing this change that is not present or active in cluster 2.
However, my OpenShift knowledge is very limited, so I don't know what can this entity be.

[claudio4j]

Hi @manusa, I'm coming back to this, since I have a similar use case related to openshift and s2i builds not updating the target/jkube/applyJson/cmiranda-dev/deployment-httplog-sandbox.json.

Is there any enricher that updates the the mentioned manifest with the s2i image build in the applyjson directory after the oc:build and before oc:deploy ?
I'm looking at ApplyService.java but couldn't figure out yet.