chore: Improve mounting certs into users containers by tolusha · Pull Request #1980 · eclipse-che/che-operator

tolusha · 2025-03-17T11:39:28Z

What does this PR do?

Removing the che-trusted-ca-certs ConfigMap from user namespaces. These ConfigMaps were used to mount certificates into the /public-certs directory.
The ca-certs-merged ConfigMap is created in the user namespace and is merged either into the /public-certs directory or /etc/pki/ca-trust/extracted/pem, depending on the value of spec.devEnvironments.trustedCerts.disableWorkspaceCaBundleMount

Screenshot/screencast of this PR

N/A

What issues does this PR fix or reference?

https://issues.redhat.com/browse/CRW-8316

How to test this PR?

Deploy a next Eclipse Che version

chectl update next
chectl server:deploy -p openshift

Login Eclipse Che (open dashboard)
Observe a size of two ConfigMaps in a user namespace (pretty big)

~ oc get configmaps -n <..> ca-certs-merged -o json | jq '.data."tls-ca-bundle.pem" | length'
238783
~ oc get configmaps -n <..> che-trusted-ca-certs -o json | jq '.data."tls-ca-bundle.pem" | length'
~ 238783

Update Eclipse Che with a new changes

chectl server:update --che-operator-image=quay.io/abazko/operator:CRW-8316

Wait until operator completes reconciliation
Check that che-trusted-ca-certs doesn't exist in a user namespace

~ oc get configmaps -n <..> che-trusted-ca-certs 
configmaps "che-trusted-ca-certs" not found

Start an empty workspace
Ensure there is no anymore /public-certs directory mounted

oc exec -n <user_namespace> <workspace_pod> -c universal-developer-image -- ls /public-certs

Ensure that tls-ca-bundle.pem file is a new file in the directory

oc exec -n <user_namespace> <workspace_pod> -c universal-developer-image -- ls /etc/pki/ca-trust/extracted/pem -l

Check it content, ensure it contains content from kube-root-ca.crt, self-signed-certificate and ca-certs ConfigMaps

oc exec -n <user_namespace> <workspace_pod> -c universal-developer-image -- cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | less

Stop a user workspace
Add custom certificate into OpenShift following the doc:
https://docs.openshift.com/container-platform/4.17/security/certificates/updating-ca-bundle.html#ca-bundle-replacing_updating-ca-bundle
After a while start a user workspace again
Recheck step 11, ensure tls-ca-bundle.pem contains recently added certificate from step 13
Disable workspace mounting certs

oc patch checluster eclipse-che -n eclipse-che --patch '{"spec": {"devEnvironments": {"trustedCerts": {"disableWorkspaceCaBundleMount": true}}}}' --type=merge

Wait workspace is restarted
Step 10 is not correct anymore, all files have the same creation date
The size of ca-certs-merged is pretty small:

~ oc get configmaps -n <..> ca-certs-merged -o json | jq '.data."tls-ca-bundle.pem" | length'
10986

Check the content of tls-ca-bundle.pem in /public-certs dir. Ensure it contains content from kube-root-ca.crt, self-signed-certificate and ca-certs ConfigMaps

oc exec -n <user_namespace> <workspace-pod> -c universal-developer-image -- cat /public-certs/tls-ca-bundle.pem

Ensure that ca-certs ConfigMap contains only certs from step 3

oc get configmaps -n eclipse-che ca-certs -o yaml

Do any editor tests

PR Checklist

As the author of this Pull Request I made sure that:

Reviewers

Reviewers, please comment how you tested the PR when approving it.

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

openshift-ci · 2025-03-17T11:39:32Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

codecov · 2025-03-17T11:41:47Z

Codecov Report

Attention: Patch coverage is 63.63636% with 48 lines in your changes missing coverage. Please review.

Project coverage is 60.07%. Comparing base (e58ad4a) to head (e4fdd7e).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/deploy/configmap.go	42.10%	22 Missing ⚠️
pkg/deploy/sync.go	0.00%	12 Missing ⚠️
pkg/deploy/labels.go	0.00%	11 Missing ⚠️
pkg/deploy/tls/certificates.go	95.65%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1980      +/-   ##
==========================================
- Coverage   60.08%   60.07%   -0.02%     
==========================================
  Files          76       76              
  Lines        9824     9895      +71     
==========================================
+ Hits         5903     5944      +41     
- Misses       3517     3547      +30     
  Partials      404      404

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

dmytro-ndp · 2025-03-18T16:12:06Z

@tolusha : thank you for detailed test scenario!

Could you please explain where to get a valid custom-ca for the next step ? :

Add custom certificate into OpenShift following the doc:
https://docs.openshift.com/container-platform/4.17/security/certificates/updating-ca-bundle.html#ca-bundle-replacing_updating-ca-bundle

And it's not clear how long to wait—minutes, ten minutes, or more—when executing the followup step: 14. After a while start a user workspace again.

tolusha · 2025-03-18T16:22:23Z

The file appears /tmp/cheCA.crt after deploying Eclipse Che with chectl
After a while for me sounds like 5m

dmytro-ndp · 2025-03-18T20:25:12Z

@tolusha: I ran test scenario expressed in the PR description against Eclipse Che deployed to test OCP 4.17.
There was expected results at step 1-16, only content of /tmp/cheCA.crt was doubled at the step 15.

At the same time I faced different behavior after disabling workspace mounting certs at step 16.

Wait workspace is restarted

Actually, workspace hadn't restarted within 10 minutes, then I restarted it manually.

Step 10 is not correct anymore, all files have the same creation date

The actual creation date of tls-ca-bundle.pem file was different to other files.

$ oc exec -n  admin-che workspacec2c09bc0db8446cd-68c6b5d9fc-kxqz7 -c universal-developer-image -- ls /etc/pki/ca-trust/extracted/pem -l
total 892
-rw-rw-r--. 1 root root    898 Jul 24  2024 README
-r--r--r--. 1 root root 165521 Oct 31 00:03 email-ca-bundle.pem
-r--r--r--. 1 root root 502506 Oct 31 00:03 objsign-ca-bundle.pem
-rw-r-----. 1 root user 236259 Mar 18 19:21 tls-ca-bundle.pem

The size of ca-certs-merged is pretty small:

There was the same size of ca-certs-merged as before.

Ensure that ca-certs ConfigMap contains only certs from step 3

There were no any certs in the ca-certs ConfigMap:

$ oc get configmaps -n eclipse-che ca-certs -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    openshift.io/owning-component: Networking / cluster-network-operator
  creationTimestamp: "2025-03-18T17:24:25Z"
  labels:
    app.kubernetes.io/component: ca-bundle
    app.kubernetes.io/instance: che
    app.kubernetes.io/managed-by: che-operator
    app.kubernetes.io/name: che
    app.kubernetes.io/part-of: che.eclipse.org
  name: ca-certs
  namespace: eclipse-che
  ownerReferences:
  - apiVersion: org.eclipse.che/v2
    blockOwnerDeletion: true
    controller: true
    kind: CheCluster
    name: eclipse-che
    uid: f1863a11-e533-4df0-8a26-3f194d485329
  resourceVersion: "51051003"
  uid: 0fd501c4-7346-4ee3-8080-0a90221b0323

Then I deleted CheCluster and created it again from scratch with property devEnvironments: trustedCerts: "disableWorkspaceCaBundleMount": true.

But che-operator failed to install Eclipse Che.

In the "che-operator" pod logs I found the next text: che-operator.logs.txt

I would appreciate it if you could take a look at the test results.

tolusha · 2025-03-19T09:17:21Z

From what I can see, step 16 wasn't completed successfully.
Please ensure, that CheCluster CR has been updated

dmytro-ndp · 2025-03-19T09:22:12Z

@tolusha : thanks for the comment.
I ensured, that CheCluster CR had been updated when executed step 16.

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

ibuziuk · 2025-03-19T13:08:37Z

api/v2/zz_generated.deepcopy.go

 	"github.com/devfile/api/v2/pkg/apis/workspaces/v1alpha2"
 	"github.com/devfile/devworkspace-operator/apis/controller/v1alpha1"
-	v1 "k8s.io/api/core/v1"
+	"k8s.io/api/core/v1"


if I remember correctly, this change needs to be reverted every time after update-dev-resources

ibuziuk · 2025-03-19T13:18:09Z

controllers/usernamespace/usernamespace_controller.go

+	// Remove trusted-ca-certs ConfigMap from the target namespace to reduce the number of ConfigMaps
+	// and avoid mounting the same certificates under different paths.
+	// See cerificates#syncCheCABundleCerts
+	trustedCACertsCMKey := client.ObjectKey{Name: prefixedName("trusted-ca-certs"), Namespace: targetNs}


on dogfooding, I see there is che-trusted-ca-certs, is it going to be removed by prefix?

Also, testing configuration of 3.19, am I correct that che-trusted-ca-certs and ca-certs-merged are pretty much identical?

che-trusted-ca-certs will be removed completedly

yes, those CMs are identical

ibuziuk · 2025-03-19T13:27:13Z

pkg/common/utils/utils.go

+		//if m == nil {
+		//	continue
+		//}


Signed-off-by: Anatolii Bazko <abazko@redhat.com>

tolusha · 2025-03-20T10:16:28Z

@dmytro-ndp
It is ready to review again

dmytro-ndp · 2025-03-20T15:13:41Z

@tolusha : seems like container image quay.io/abazko/operator:CRW-8316 mentioned in step 5 of test scenario from PR description doesn't include latest commit 28b87c4

Could you please update that image?

dmytro-ndp

The test scenario from the PR description passed when run using Eclipse Che Next with the dashboard image quay.io/abazko/operator@sha256:265f6fb39412b5a4df067bba95baf24ca121a5f580927603d63843483b40747b

Test details:

workspace was restarted automatically at step 17, after setting up "devEnvironments.trustedCerts.disableWorkspaceCaBundleMount: true"
result of step "18. Step 10 is not correct anymore, all files have the same creation date":

$ oc exec -n admin-che workspace30fc90bb823e42dd-d79b5f65d-rj585 -c universal-developer-image -- ls /etc/pki/ca-trust/extracted/pem -l
total 884
-rw-rw-r--. 1 root root    898 Jul 24  2024 README
-r--r--r--. 1 root root 165521 Oct 31 00:03 email-ca-bundle.pem
-r--r--r--. 1 root root 502506 Oct 31 00:03 objsign-ca-bundle.pem
-r--r--r--. 1 root root 226489 Oct 31 00:03 tls-ca-bundle.pem

result of step "19. The size of ca-certs-merged is pretty small":

$ oc get configmaps -n admin-che ca-certs-merged -o json | jq '.data."tls-ca-bundle.pem" | length'
10888

$ oc get configmaps -n admin-che che-trusted-ca-certs -o json | jq '.data."tls-ca-bundle.pem" | length'
Error from server (NotFound): configmaps "che-trusted-ca-certs" not found

"Step 22. Do any editor tests"
A few sample workspaces - .NET 5.0, React - started successfully. VS Code Editors opened, and I was able to start the React application and open it in a separate tab.

Well done, @tolusha !

openshift-ci · 2025-03-21T20:30:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dmytro-ndp, tolusha

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

devspacesbuild · 2025-03-24T07:43:41Z

Build 3.21 :: operator_3.x/450: Console, Changes, Git Data

devspacesbuild · 2025-03-26T07:03:51Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9127: Console, Changes, Git Data

devspacesbuild · 2025-03-26T07:05:55Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9127:

devspaces-operator-bundle : 3.x :: Failed in 67114466 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-26T10:48:31Z

Build 3.21 :: operator-bundle_3.x/4639: Console, Changes, Git Data

devspacesbuild · 2025-03-26T10:51:50Z

Build 3.21 :: sync-to-downstream_3.x/9008: Console, Changes, Git Data

devspacesbuild · 2025-03-26T10:56:41Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9128: Console, Changes, Git Data

devspacesbuild · 2025-03-26T11:08:47Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9128:

devspaces-operator-bundle : 3.x :: Failed in 67116633 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-26T14:49:31Z

Build 3.21 :: operator-bundle_3.x/4640: Console, Changes, Git Data

devspacesbuild · 2025-03-26T14:53:01Z

Build 3.21 :: sync-to-downstream_3.x/9009: Console, Changes, Git Data

devspacesbuild · 2025-03-26T14:57:21Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9129: Console, Changes, Git Data

devspacesbuild · 2025-03-26T15:37:25Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9129:

devspaces-operator-bundle : 3.x :: Failed in 67119459 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-26T18:57:11Z

Build 3.21 :: operator-bundle_3.x/4641: Console, Changes, Git Data

devspacesbuild · 2025-03-26T19:05:02Z

Build 3.21 :: sync-to-downstream_3.x/9010: Console, Changes, Git Data

devspacesbuild · 2025-03-26T19:09:21Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9131: Console, Changes, Git Data

devspacesbuild · 2025-03-26T19:11:58Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9131:

devspaces-operator-bundle : 3.x :: Failed in 67123288 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-26T22:52:59Z

Build 3.21 :: operator-bundle_3.x/4644: Console, Changes, Git Data

devspacesbuild · 2025-03-26T22:59:23Z

Build 3.21 :: sync-to-downstream_3.x/9013: Console, Changes, Git Data

devspacesbuild · 2025-03-26T23:03:20Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9135: Console, Changes, Git Data

devspacesbuild · 2025-03-26T23:12:08Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9135:

devspaces-operator-bundle : 3.x :: Failed in 67126362 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-27T02:49:16Z

Build 3.21 :: operator-bundle_3.x/4645: Console, Changes, Git Data

devspacesbuild · 2025-03-27T02:52:30Z

Build 3.21 :: sync-to-downstream_3.x/9014: Console, Changes, Git Data

devspacesbuild · 2025-03-27T02:57:02Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9136: Console, Changes, Git Data

devspacesbuild · 2025-03-27T02:59:40Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9136:

devspaces-operator-bundle : 3.x :: Failed in 67128187 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-27T06:52:21Z

Build 3.21 :: operator-bundle_3.x/4646: Console, Changes, Git Data

devspacesbuild · 2025-03-27T06:59:39Z

Build 3.21 :: sync-to-downstream_3.x/9015: Console, Changes, Git Data

devspacesbuild · 2025-03-27T07:04:11Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9137: Console, Changes, Git Data

devspacesbuild · 2025-03-27T07:06:35Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9137:

devspaces-operator-bundle : 3.x :: Failed in 67130972 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

devspacesbuild · 2025-03-27T08:19:29Z

Build 3.21 :: operator-bundle_3.x/4648: Console, Changes, Git Data

devspacesbuild · 2025-03-27T08:26:14Z

Build 3.21 :: sync-to-downstream_3.x/9020: Console, Changes, Git Data

devspacesbuild · 2025-03-27T08:26:51Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9142: Console, Changes, Git Data

devspacesbuild · 2025-03-27T08:29:03Z

Build 3.21 :: get-sources-rhpkg-container-build_3.x/9142:

devspaces-operator-bundle : 3.x :: Failed in 67131361 : BREW:BUILD/STATUS:UNKNOWN
FAILURE:; copied to quay

chore: Improve mounting certs into users containers

e4fdd7e

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

openshift-ci bot added the do-not-merge/work-in-progress label Mar 17, 2025

fixup

dba6274

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

tolusha marked this pull request as ready for review March 18, 2025 11:53

tolusha requested review from SDawley, ibuziuk and mkuznyetsov as code owners March 18, 2025 11:53

openshift-ci bot removed the do-not-merge/work-in-progress label Mar 18, 2025

ibuziuk requested a review from dmytro-ndp March 18, 2025 15:04

Fixup

fb50bba

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

ibuziuk reviewed Mar 19, 2025

View reviewed changes

pkg/common/utils/utils.go Outdated

Comment on lines 227 to 229

//if m == nil {

// continue

//}

Copy link

Member

ibuziuk Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leftover?

fixup

28b87c4

Signed-off-by: Anatolii Bazko <abazko@redhat.com>

dmytro-ndp approved these changes Mar 21, 2025

View reviewed changes

openshift-ci bot assigned dmytro-ndp Mar 21, 2025

openshift-ci bot added the lgtm label Mar 21, 2025

tolusha merged commit ae7054e into main Mar 24, 2025
21 checks passed

tolusha deleted the CRW-8316 branch March 24, 2025 07:43

Conversation

tolusha commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Screenshot/screencast of this PR

What issues does this PR fix or reference?

How to test this PR?

PR Checklist

Reviewers

Uh oh!

openshift-ci bot commented Mar 17, 2025

Uh oh!

codecov bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dmytro-ndp commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tolusha commented Mar 18, 2025

Uh oh!

dmytro-ndp commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tolusha commented Mar 19, 2025

Uh oh!

dmytro-ndp commented Mar 19, 2025

Uh oh!

ibuziuk Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

tolusha Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

ibuziuk Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

ibuziuk Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

tolusha Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

ibuziuk Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

tolusha commented Mar 20, 2025

Uh oh!

dmytro-ndp commented Mar 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmytro-ndp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Mar 21, 2025

Uh oh!

Uh oh!

devspacesbuild commented Mar 24, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

devspacesbuild commented Mar 26, 2025

Uh oh!

tolusha commented Mar 17, 2025 •

edited

Loading

codecov bot commented Mar 17, 2025 •

edited

Loading

dmytro-ndp commented Mar 18, 2025 •

edited

Loading

dmytro-ndp commented Mar 18, 2025 •

edited

Loading

dmytro-ndp commented Mar 20, 2025 •

edited by openshift-ci bot

Loading

dmytro-ndp left a comment •

edited

Loading