Add MCSP IDP configuration update to common service database restore process #2602

YCShen1010 · 2025-08-14T22:00:01Z

What this PR does / why we need it:
Adds automatic IDP URL updating functionality to the common-services database restore script. After restoring the account_iam database, the it now automatically updates IDP configuration URLs to match the current cluster domain.

Which issue(s) this PR fixes:
Fixes # https://github.ibm.com/IBMPrivateCloud/roadmap/issues/67335

How to test:

oc login into a cluster and install UM operator

Manually change the idp in database to other URL in common-service-db terminal by using command

psql -U postgres
 \l
 \c account_iam
 \dn
 \dt accountiam.*;
 SELECT * FROM accountiam.idp_config;

UPDATE accountiam.idp_config
SET idp = 'https://test-idp-url.example.com/idprovider/v1/auth'

Create a simple test script for the new functions and execute.
Check if the idp URL changes back

Test result:

➜  common-service-db git:(update_idp) ✗ bash test-idp-update.sh mcsp  
Getting primary PostgreSQL pod...
Primary pod: common-service-db-1
Checking if account_iam database exists...
Current IDP configuration BEFORE update:
                 uid                  |    realm     |                                  idp                                   |          modified_ts          
--------------------------------------+--------------+------------------------------------------------------------------------+-------------------------------
 0a928673-f2a7-4804-b781-ffa9f54e7985 | PrimaryRealm | https://test-idp-url.example.com/idprovider/v1/auth | 2025-08-14 21:38:05.730994+00
 ed890909-2896-4a7a-951c-954f524892ad | SAML         | https://test-idp-url.example.com/idprovider/v1/auth | 2025-08-14 21:38:05.730994+00
 2d13f58e-19bb-4601-9d29-e8d9c295cf71 | OIDC         | https://test-idp-url.example.com/idprovider/v1/auth | 2025-08-14 21:38:05.730994+00
 d26bb99a-ed17-4b4c-8918-04ef966c52ee | PrimaryRealm | https://test-idp-url.example.com/idprovider/v1/auth | 2025-08-14 21:38:05.730994+00
(4 rows)

Getting cluster domain...
Detected cluster domain: apps.cutie1.cp.fyre.ibm.com
New IDP URL will be: https://cp-console.mcsp.apps.cutie1.cp.fyre.ibm.com/idprovider/v1/auth
Updating IDP configuration...
UPDATE 4
IDP configuration AFTER update:
                 uid                  |    realm     |                                  idp                                   |          modified_ts          
--------------------------------------+--------------+------------------------------------------------------------------------+-------------------------------
 0a928673-f2a7-4804-b781-ffa9f54e7985 | PrimaryRealm | https://cp-console.mcsp.apps.cutie1.cp.fyre.ibm.com/idprovider/v1/auth | 2025-08-14 21:55:25.937126+00
 ed890909-2896-4a7a-951c-954f524892ad | SAML         | https://cp-console.mcsp.apps.cutie1.cp.fyre.ibm.com/idprovider/v1/auth | 2025-08-14 21:55:25.937126+00
 2d13f58e-19bb-4601-9d29-e8d9c295cf71 | OIDC         | https://cp-console.mcsp.apps.cutie1.cp.fyre.ibm.com/idprovider/v1/auth | 2025-08-14 21:55:25.937126+00
 d26bb99a-ed17-4b4c-8918-04ef966c52ee | PrimaryRealm | https://cp-console.mcsp.apps.cutie1.cp.fyre.ibm.com/idprovider/v1/auth | 2025-08-14 21:55:25.937126+00
(4 rows)

Test completed!

ibm-ci-bot · 2025-08-14T22:00:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: YCShen1010

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [YCShen1010]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bluzarraga · 2025-08-15T14:14:53Z

@YCShen1010 can you share the test script you used with me?

bluzarraga

I think my main issue is that we don't want to be adding new Roles/Rolebindings in new namespaces to obtain the domain info. Preferably we want to get this value in a way that requires less permissions. One place to check for the cluster domain value is the configmap ibmcloud-cluster-info in the services namespace. We do not BR this configmap, it is generated by the IM operator so it should always contain the correct value. We may just have to make sure the configmap exists.

The logic you have in the pr looks like it would work (I haven't gotten a chance to try it yet) but to know if this PR would work, we want to see it run as a restore hook. The script you have to test is a good proof of concept to make sure the logic is doing what you want it to but somethings we will only see when testing more along the lines of how they will be used. For example, the test script would not have picked up the permissions problem but running the restore would have.

bluzarraga · 2025-08-19T16:10:12Z

velero/schedule/common-service-db/cs-db-br-script-cm-4.6.10.4.11.yaml

+        info "Updating IDP configuration with actual cluster domain..."
+
+        # Get the cluster domain from the management ingress
+        CLUSTER_DOMAIN=$(oc get route console -n openshift-console -o jsonpath='{.spec.host}' | sed 's/^console-openshift-console\.//')


Do we need to add permissions to the BR role to enable this? Right now the br resources are limited to the services namespace. Can we determine this value from somewhere else?

bluzarraga · 2025-08-19T16:11:04Z

velero/schedule/common-service-db/cs-db-br-script-cm-4.6.10.4.11.yaml

+        # Check if account_iam database exists
+        ACCOUNT_IAM_EXISTS=$(oc -n $CSDB_NAMESPACE exec -t $CNPG_PRIMARY_POD -c postgres -- psql -U postgres -c "\list" | grep "account_iam" || echo False)
+
+        if [[ $ACCOUNT_IAM_EXISTS != "False" ]]; then


I think we should include the call to check the cluster domain value inside this if. That way we only look for this value when we need it

bluzarraga · 2025-08-19T16:13:53Z

velero/schedule/common-service-db/cs-db-br-script-cm-4.6.10.4.11.yaml

+
+            success "IDP configuration updated successfully."
+        else
+            warning "account_iam database not found, skipping IDP configuration update."


nitpick but I think this should be info as right now it is more likely to not be installed given current adopters' usage. I think we could change the message from "skipping IDP configuration update" to something less ominous. If you are reading these logs and don't know what you are skipping, you may think that the backup and restore process did not work properly because the IDP config wasn't updated when what really happened was the MCSP specific URL wasn't updated. Paired with the warning it could send the wrong message

bluzarraga

I'm getting this error when I try to run the BR hook:

pg_dump: error: connection to server on socket "/controller/run/.s.PGSQL.5432" failed: FATAL:  database "cloudpak" does not exist
command terminated with exit code 1

I think I remember something about removing the cloudpak database from the cs db setup since it was unused. Can we go ahead and remove it from this script as well? Removing it in my cluster moved the backup forward but eventually my restore failed because of use of xargs

bluzarraga · 2025-08-25T19:42:41Z

velero/schedule/common-service-db/cs-db-br-script-cm-4.6.10.4.11.yaml

+
+        if [[ $ACCOUNT_IAM_EXISTS != "False" ]]; then
+            # Check current IDP configuration
+            CURRENT_IDP=$(oc -n $CSDB_NAMESPACE exec -t $CNPG_PRIMARY_POD -c postgres -- psql -U postgres -d account_iam -t -c "SELECT DISTINCT idp FROM accountiam.idp_config WHERE idp LIKE '%/idprovider/v1/%' LIMIT 1;" | xargs || echo "")


I just finished a full BR with these changes and realized that you are using xargs here. Unfortunately, we cannot use xargs because it has a number of security vulnerabilities and we have been unable to clear it with Tod for inclusion in the cpfs-utils image. We'll need to update this command to not rely on xargs. Very annoying I know but unfortunately not very flexible

I used another approach excluding xargs to get the same IDP and have pushed the commit now, thanks for pointing this out.

bluzarraga · 2025-08-25T20:13:02Z

it looks like this pr was supposed to have removed the cloud pak database https://github.com/IBM/ibm-common-service-operator/pull/2532/files so I am not sure why it is still present here

Signed-off-by: YuChen <[email protected]>

YCShen1010 · 2025-08-26T15:31:15Z

Hi @bluzarraga, I have done the rebase, the cloudpak database is not in the script anymore.

Signed-off-by: YuChen <[email protected]>

YCShen1010 requested a review from bluzarraga August 14, 2025 22:00

bluzarraga requested changes Aug 19, 2025

View reviewed changes

YCShen1010 force-pushed the update_idp branch from 1bea478 to 5cdfd7a Compare August 21, 2025 16:26

bluzarraga requested changes Aug 25, 2025

View reviewed changes

YCShen1010 added 5 commits August 26, 2025 08:24

Add IDP configuration update function for mcsp BR

5b1b4bc

Signed-off-by: YuChen <[email protected]>

update modify idp function to verification step

be3cab9

Signed-off-by: YuChen <[email protected]>

test idp config update

1eddfb2

Signed-off-by: YuChen <[email protected]>

get route from IM configmap

bf8dbbf

Signed-off-by: YuChen <[email protected]>

remove xargs vulnerability

fc1f1b0

Signed-off-by: YuChen <[email protected]>

YCShen1010 force-pushed the update_idp branch from 0322df9 to fc1f1b0 Compare August 26, 2025 15:25

YCShen1010 added 2 commits August 28, 2025 09:53

removed test script

ad3c4d8

Signed-off-by: YuChen <[email protected]>

add restart account iam pod after updating idp

1a79f09

Signed-off-by: YuChen <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add MCSP IDP configuration update to common service database restore process #2602

Add MCSP IDP configuration update to common service database restore process #2602

Uh oh!

YCShen1010 commented Aug 14, 2025 •

edited

Loading

Uh oh!

ibm-ci-bot commented Aug 14, 2025

Uh oh!

bluzarraga commented Aug 15, 2025

Uh oh!

bluzarraga left a comment •

edited

Loading

Uh oh!

bluzarraga Aug 19, 2025

Uh oh!

bluzarraga Aug 19, 2025

Uh oh!

bluzarraga Aug 19, 2025

Uh oh!

bluzarraga left a comment

Uh oh!

bluzarraga Aug 25, 2025

Uh oh!

YCShen1010 Aug 26, 2025

Uh oh!

bluzarraga commented Aug 25, 2025

Uh oh!

YCShen1010 commented Aug 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

Add MCSP IDP configuration update to common service database restore process #2602

Are you sure you want to change the base?

Add MCSP IDP configuration update to common service database restore process #2602

Uh oh!

Conversation

YCShen1010 commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ibm-ci-bot commented Aug 14, 2025

Uh oh!

bluzarraga commented Aug 15, 2025

Uh oh!

bluzarraga left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bluzarraga Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

bluzarraga Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

bluzarraga Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

bluzarraga left a comment

Choose a reason for hiding this comment

Uh oh!

bluzarraga Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

YCShen1010 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

bluzarraga commented Aug 25, 2025

Uh oh!

YCShen1010 commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

YCShen1010 commented Aug 14, 2025 •

edited

Loading

bluzarraga left a comment •

edited

Loading

YCShen1010 commented Aug 26, 2025 •

edited

Loading