Restart auth service by qpdpQ · Pull Request #2321 · IBM/ibm-common-service-operator

qpdpQ · 2024-12-03T20:56:07Z

What this PR does / why we need it:
Platform-auth-service relies on the common-service-db data to create login options file, so we need to restart platform-auth-service after restore cs-db.
Which issue(s) this PR fixes:
Fixes # https://github.ibm.com/IBMPrivateCloud/roadmap/issues/65098

Special notes for your reviewer:

How the test is done?

How to backport this PR to other branch:

Add label to this PR with the target branch name backport <branch-name>
The PR will be automatically created in the target branch after merging this PR
If this PR is already merged, you can still add the label with the target branch name backport <branch-name> and leave a comment /backport to trigger the backport action

Signed-off-by: Allen Li <liyuchen223@gmail.com>

ibm-ci-bot · 2024-12-03T20:56:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qpdpQ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [qpdpQ]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Allen Li <liyuchen223@gmail.com>

qpdpQ · 2024-12-10T18:12:59Z

test logs:

sh-4.4$ ./cs-db/br_cs-db.sh restore cpfs
[INFO] Mode is set to restore, beginning restore process.
[INFO] Embedded Postgres DB in use, beginning data restore.
[INFO] Waiting for EDB Cluster CR common-service-db to complete in namespace cpfs.
[INFO] EDB cluster common-service-db ready.
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
                                                  List of databases
   Name    |  Owner   | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules |   Access privileges   
-----------+----------+----------+-----------------+---------+-------+------------+-----------+-----------------------
 cloudpak  | cpadmin  | UTF8     | libc            | C       | C     |            |           | 
 im        | im_user  | UTF8     | libc            | C       | C     |            |           | =Tc/im_user          +
           |          |          |                 |         |       |            |           | im_user=CTc/im_user
 postgres  | postgres | UTF8     | libc            | C       | C     |            |           | 
 template0 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres          +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 template1 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres          +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 zen       | zen_user | UTF8     | libc            | C       | C     |            |           | =Tc/zen_user         +
           |          |          |                 |         |       |            |           | zen_user=CTc/zen_user
(6 rows)

      List of schemas
  Name  |       Owner       
--------+-------------------
 public | pg_database_owner
(1 row)

                                 List of roles
     Role name     |                         Attributes                         
-------------------+------------------------------------------------------------
 cpadmin           | 
 im_user           | 
 postgres          | Superuser, Create role, Create DB, Replication, Bypass RLS
 streaming_replica | Replication
 zen_user          | 

[INFO] Beginning restore of cloudpak database...
pg_restore: connecting to database for restore
pg_restore: implied data-only restore
[INFO] Beginning restore of im database...
pg_restore: connecting to database for restore
pg_restore: dropping FK CONSTRAINT users_groups users_groups_user_uid_fkey
pg_restore: dropping FK CONSTRAINT users_groups users_groups_group_uid_fkey
pg_restore: dropping FK CONSTRAINT scim_server_users_custom scim_server_users_custom_scim_server_user_uid_fkey
pg_restore: dropping FK CONSTRAINT scim_server_groups_custom scim_server_groups_custom_scim_server_group_uid_fkey
pg_restore: dropping FK CONSTRAINT zen_instances_users fk_zenuser_fk
pg_restore: dropping FK CONSTRAINT zen_instances_users fk_userzen_fk
pg_restore: dropping FK CONSTRAINT users_preferences fk_userpref_fk
pg_restore: dropping FK CONSTRAINT users_attributes fk_useratt_fk
pg_restore: dropping INDEX oauth20cache_expires
pg_restore: dropping CONSTRAINT zen_instances_users zeninstances_users_uid
pg_restore: dropping CONSTRAINT zen_instances zen_instances_instance_id
pg_restore: dropping CONSTRAINT users users_userid
pg_restore: dropping CONSTRAINT users users_uid
pg_restore: dropping CONSTRAINT users_preferences users_preferences_useruid
pg_restore: dropping CONSTRAINT users_preferences users_preferences_uid
pg_restore: dropping CONSTRAINT users_groups users_groups_pkey
pg_restore: dropping CONSTRAINT users_attributes users_attributes_uid
pg_restore: dropping CONSTRAINT scim_server_users scim_server_users_pkey
pg_restore: dropping CONSTRAINT scim_server_users_custom scim_server_users_custom_pkey
pg_restore: dropping CONSTRAINT scim_server_groups scim_server_groups_pkey
pg_restore: dropping CONSTRAINT scim_server_groups_custom scim_server_groups_custom_pkey
pg_restore: dropping CONSTRAINT scim_attributes scim_attributes_id
pg_restore: dropping CONSTRAINT scim_attributes_mappings scim_attributemappings_idp_id
pg_restore: dropping CONSTRAINT rolemappings rolemappings_pkey
pg_restore: dropping CONSTRAINT idp_configs idp_configs_uid
pg_restore: dropping CONSTRAINT groups groups_pkey
pg_restore: dropping CONSTRAINT groups groups_group_id_realm_id
pg_restore: dropping CONSTRAINT apikeys apikeys_pkey
pg_restore: dropping CONSTRAINT oauthtoken pk_lookupkey
pg_restore: dropping CONSTRAINT oauthclient pk_compidclientid
pg_restore: dropping DEFAULT apikeys id
pg_restore: dropping DEFAULT oauthtoken _id
pg_restore: dropping DEFAULT oauthconsent _id
pg_restore: dropping DEFAULT oauthclient _id
pg_restore: dropping TABLE zen_instances_users
pg_restore: dropping TABLE zen_instances
pg_restore: dropping VIEW view_scim_server_users_custom
pg_restore: dropping VIEW view_scim_server_groups_custom
pg_restore: dropping TABLE users_preferences
pg_restore: dropping TABLE users_groups
pg_restore: dropping TABLE users_attributes
pg_restore: dropping TABLE users
pg_restore: dropping TABLE scim_server_users_custom
pg_restore: dropping TABLE scim_server_users
pg_restore: dropping TABLE scim_server_groups_custom
pg_restore: dropping TABLE scim_server_groups
pg_restore: dropping TABLE scim_attributes_mappings
pg_restore: dropping TABLE scim_attributes
pg_restore: dropping TABLE rolemappings
pg_restore: dropping TABLE idp_configs
pg_restore: dropping TABLE groups
pg_restore: dropping SEQUENCE apikeys_id_seq
pg_restore: dropping TABLE apikeys
pg_restore: dropping SEQUENCE oauthtoken__id_seq
pg_restore: dropping TABLE oauthtoken
pg_restore: dropping SEQUENCE oauthconsent__id_seq
pg_restore: dropping TABLE oauthconsent
pg_restore: dropping SEQUENCE oauthclient__id_seq
pg_restore: dropping TABLE oauthclient
pg_restore: dropping SCHEMA platformdb
pg_restore: dropping SCHEMA oauthdbschema
pg_restore: creating SCHEMA "oauthdbschema"
pg_restore: creating SCHEMA "platformdb"
pg_restore: creating TABLE "oauthdbschema.oauthclient"
pg_restore: creating SEQUENCE "oauthdbschema.oauthclient__id_seq"
pg_restore: creating SEQUENCE OWNED BY "oauthdbschema.oauthclient__id_seq"
pg_restore: creating TABLE "oauthdbschema.oauthconsent"
pg_restore: creating SEQUENCE "oauthdbschema.oauthconsent__id_seq"
pg_restore: creating SEQUENCE OWNED BY "oauthdbschema.oauthconsent__id_seq"
pg_restore: creating TABLE "oauthdbschema.oauthtoken"
pg_restore: creating SEQUENCE "oauthdbschema.oauthtoken__id_seq"
pg_restore: creating SEQUENCE OWNED BY "oauthdbschema.oauthtoken__id_seq"
pg_restore: creating TABLE "platformdb.apikeys"
pg_restore: creating SEQUENCE "platformdb.apikeys_id_seq"
pg_restore: creating SEQUENCE OWNED BY "platformdb.apikeys_id_seq"
pg_restore: creating TABLE "platformdb.groups"
pg_restore: creating TABLE "platformdb.idp_configs"
pg_restore: creating TABLE "platformdb.rolemappings"
pg_restore: creating TABLE "platformdb.scim_attributes"
pg_restore: creating TABLE "platformdb.scim_attributes_mappings"
pg_restore: creating TABLE "platformdb.scim_server_groups"
pg_restore: creating TABLE "platformdb.scim_server_groups_custom"
pg_restore: creating TABLE "platformdb.scim_server_users"
pg_restore: creating TABLE "platformdb.scim_server_users_custom"
pg_restore: creating TABLE "platformdb.users"
pg_restore: creating TABLE "platformdb.users_attributes"
pg_restore: creating TABLE "platformdb.users_groups"
pg_restore: creating TABLE "platformdb.users_preferences"
pg_restore: creating VIEW "platformdb.view_scim_server_groups_custom"
pg_restore: creating VIEW "platformdb.view_scim_server_users_custom"
pg_restore: creating TABLE "platformdb.zen_instances"
pg_restore: creating TABLE "platformdb.zen_instances_users"
pg_restore: creating DEFAULT "oauthdbschema.oauthclient _id"
pg_restore: creating DEFAULT "oauthdbschema.oauthconsent _id"
pg_restore: creating DEFAULT "oauthdbschema.oauthtoken _id"
pg_restore: creating DEFAULT "platformdb.apikeys id"
pg_restore: processing data for table "oauthdbschema.oauthclient"
pg_restore: processing data for table "oauthdbschema.oauthconsent"
pg_restore: processing data for table "oauthdbschema.oauthtoken"
pg_restore: processing data for table "platformdb.apikeys"
pg_restore: processing data for table "platformdb.groups"
pg_restore: processing data for table "platformdb.idp_configs"
pg_restore: processing data for table "platformdb.rolemappings"
pg_restore: processing data for table "platformdb.scim_attributes"
pg_restore: processing data for table "platformdb.scim_attributes_mappings"
pg_restore: processing data for table "platformdb.scim_server_groups"
pg_restore: processing data for table "platformdb.scim_server_groups_custom"
pg_restore: processing data for table "platformdb.scim_server_users"
pg_restore: processing data for table "platformdb.scim_server_users_custom"
pg_restore: processing data for table "platformdb.users"
pg_restore: processing data for table "platformdb.users_attributes"
pg_restore: processing data for table "platformdb.users_groups"
pg_restore: processing data for table "platformdb.users_preferences"
pg_restore: processing data for table "platformdb.zen_instances"
pg_restore: processing data for table "platformdb.zen_instances_users"
pg_restore: executing SEQUENCE SET oauthclient__id_seq
pg_restore: executing SEQUENCE SET oauthconsent__id_seq
pg_restore: executing SEQUENCE SET oauthtoken__id_seq
pg_restore: executing SEQUENCE SET apikeys_id_seq
pg_restore: creating CONSTRAINT "oauthdbschema.oauthclient pk_compidclientid"
pg_restore: creating CONSTRAINT "oauthdbschema.oauthtoken pk_lookupkey"
pg_restore: creating CONSTRAINT "platformdb.apikeys apikeys_pkey"
pg_restore: creating CONSTRAINT "platformdb.groups groups_group_id_realm_id"
pg_restore: creating CONSTRAINT "platformdb.groups groups_pkey"
pg_restore: creating CONSTRAINT "platformdb.idp_configs idp_configs_uid"
pg_restore: creating CONSTRAINT "platformdb.rolemappings rolemappings_pkey"
pg_restore: creating CONSTRAINT "platformdb.scim_attributes_mappings scim_attributemappings_idp_id"
pg_restore: creating CONSTRAINT "platformdb.scim_attributes scim_attributes_id"
pg_restore: creating CONSTRAINT "platformdb.scim_server_groups_custom scim_server_groups_custom_pkey"
pg_restore: creating CONSTRAINT "platformdb.scim_server_groups scim_server_groups_pkey"
pg_restore: creating CONSTRAINT "platformdb.scim_server_users_custom scim_server_users_custom_pkey"
pg_restore: creating CONSTRAINT "platformdb.scim_server_users scim_server_users_pkey"
pg_restore: creating CONSTRAINT "platformdb.users_attributes users_attributes_uid"
pg_restore: creating CONSTRAINT "platformdb.users_groups users_groups_pkey"
pg_restore: creating CONSTRAINT "platformdb.users_preferences users_preferences_uid"
pg_restore: creating CONSTRAINT "platformdb.users_preferences users_preferences_useruid"
pg_restore: creating CONSTRAINT "platformdb.users users_uid"
pg_restore: creating CONSTRAINT "platformdb.users users_userid"
pg_restore: creating CONSTRAINT "platformdb.zen_instances zen_instances_instance_id"
pg_restore: creating CONSTRAINT "platformdb.zen_instances_users zeninstances_users_uid"
pg_restore: creating INDEX "oauthdbschema.oauth20cache_expires"
pg_restore: creating FK CONSTRAINT "platformdb.users_attributes fk_useratt_fk"
pg_restore: creating FK CONSTRAINT "platformdb.users_preferences fk_userpref_fk"
pg_restore: creating FK CONSTRAINT "platformdb.zen_instances_users fk_userzen_fk"
pg_restore: creating FK CONSTRAINT "platformdb.zen_instances_users fk_zenuser_fk"
pg_restore: creating FK CONSTRAINT "platformdb.scim_server_groups_custom scim_server_groups_custom_scim_server_group_uid_fkey"
pg_restore: creating FK CONSTRAINT "platformdb.scim_server_users_custom scim_server_users_custom_scim_server_user_uid_fkey"
pg_restore: creating FK CONSTRAINT "platformdb.users_groups users_groups_group_uid_fkey"
pg_restore: creating FK CONSTRAINT "platformdb.users_groups users_groups_user_uid_fkey"
[INFO] Beginning restore of zen database...
pg_restore: connecting to database for restore
pg_restore: implied data-only restore
                                                  List of databases
   Name    |  Owner   | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules |   Access privileges   
-----------+----------+----------+-----------------+---------+-------+------------+-----------+-----------------------
 cloudpak  | cpadmin  | UTF8     | libc            | C       | C     |            |           | 
 im        | im_user  | UTF8     | libc            | C       | C     |            |           | =Tc/im_user          +
           |          |          |                 |         |       |            |           | im_user=CTc/im_user
 postgres  | postgres | UTF8     | libc            | C       | C     |            |           | 
 template0 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres          +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 template1 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres          +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 zen       | zen_user | UTF8     | libc            | C       | C     |            |           | =Tc/zen_user         +
           |          |          |                 |         |       |            |           | zen_user=CTc/zen_user
(6 rows)

      List of schemas
  Name  |       Owner       
--------+-------------------
 public | pg_database_owner
(1 row)

                                 List of roles
     Role name     |                         Attributes                         
-------------------+------------------------------------------------------------
 cpadmin           | 
 im_user           | 
 postgres          | Superuser, Create role, Create DB, Replication, Bypass RLS
 streaming_replica | Replication
 zen_user          | 

[INFO] Rerunning OIDC registration job...
job.batch "oidc-client-registration" deleted
[INFO] Wait for previous job to delete...
Warning: resource jobs/oidc-client-registration is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
job.batch/oidc-client-registration configured
[INFO] Waiting for job oidc-client-registration to complete in namespace cpfs.
[INFO] Job oidc-client-registration completed.
[INFO] “Restarting
pod "platform-auth-service-7b89b46669-mdw9h" deleted
[INFO] Wait for previous pod to delete...
[INFO] Waiting for pod platform-auth-service to restart in namespace cpfs.
[INFO] Wait for pod platform-auth-service to complete. Try again in 5s.
[INFO] Pod platform-auth-service restarted.
[✔] Restore completed successfully.

bluzarraga

@qpdpQ The code is working except for the wait condition. My script has hung waiting for the pod to come ready again but I can see that the pod is ready. I ran into a problem with the column ordering not being consistent with the oidc-registration-job which I have tried to address with #2335 but from what I can tell the column is what we expect. Maybe it would be more consistent to check the platform-auth-service deployment? Something like comparing the availableReplicas/readyReplicas to the replicas value?

velero/schedule/common-service-db/cs-db-br-script-cm.yaml

Signed-off-by: Allen Li <liyuchen223@gmail.com>

velero/schedule/common-service-db/cs-db-br-script-cm.yaml

Signed-off-by: Allen Li <liyuchen223@gmail.com>

qpdpQ · 2025-01-07T21:26:56Z

test result looks good to me if we apply the fix in #2335


[INFO] Creating dummy route to update ROKS_URL value...
route.route.openshift.io/dummy created
[INFO] Grabbing new hostname...
[INFO] New hostname: allen.cp.fyre.ibm.com
[INFO] New ROKS_URL value: https://oauth-openshift.apps.allen.cp.fyre.ibm.com
[INFO] Updating ROKS_URL value in platform-auth-idp configmap...
configmap/platform-auth-idp patched (no change)
[INFO] Deleting dummy route from namespace cs2...
route.route.openshift.io "dummy" deleted
[INFO] Rerunning OIDC registration job...
job.batch "oidc-client-registration" deleted
[INFO] Wait for previous job to delete...
Warning: resource jobs/oidc-client-registration is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
job.batch/oidc-client-registration configured
[INFO] Waiting for job oidc-client-registration to complete in namespace cs2.
[INFO] Job oidc-client-registration completed.
[INFO] Restarting platform-auth-service...
pod "platform-auth-service-6f47d5b969-97mqk" deleted
[INFO] Wait for previous pod to delete...
[INFO] Waiting for pod platform-auth-service to restart in namespace cs2.
[INFO] Wait for pod platform-auth-service to complete. Try again in 5s.
[INFO] Wait for pod platform-auth-service to complete. Try again in 5s.
[INFO] Wait for pod platform-auth-service to complete. Try again in 5s.
[INFO] Pod platform-auth-service restarted.
[✔] Restore completed successfully.

bluzarraga · 2025-01-16T19:30:58Z

/lgtm

qpdpQ added 2 commits December 3, 2024 15:50

restart platform-auth-service after restore cs-db

56cbd68

Signed-off-by: Allen Li <liyuchen223@gmail.com>

change format

354b05b

Signed-off-by: Allen Li <liyuchen223@gmail.com>

ibm-ci-bot requested review from Daniel-Fan and YCShen1010 December 3, 2024 20:56

ibm-ci-bot added approved size/M labels Dec 3, 2024

add configmap permission

2778597

Signed-off-by: Allen Li <liyuchen223@gmail.com>

bluzarraga mentioned this pull request Dec 12, 2024

More consistently check OIDC job readiness #2335

Merged

bluzarraga requested changes Dec 12, 2024

View reviewed changes

velero/schedule/common-service-db/cs-db-br-script-cm.yaml Outdated Show resolved Hide resolved

consistently check auth-service

3b65cb2

Signed-off-by: Allen Li <liyuchen223@gmail.com>

bluzarraga requested changes Dec 19, 2024

View reviewed changes

velero/schedule/common-service-db/cs-db-br-script-cm.yaml Show resolved Hide resolved

qpdpQ and others added 2 commits December 19, 2024 15:03

reset values in while loop

9bbf1df

Signed-off-by: Allen Li <liyuchen223@gmail.com>

Merge branch 'scripts-dev' into restart-auth-service

e94ce88

ibm-ci-bot assigned bluzarraga Jan 16, 2025

ibm-ci-bot added the lgtm label Jan 16, 2025

ibm-ci-bot merged commit 9d66bcc into IBM:scripts-dev Jan 16, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restart auth service#2321

Restart auth service#2321
ibm-ci-bot merged 6 commits intoIBM:scripts-devfrom
qpdpQ:restart-auth-service

qpdpQ commented Dec 3, 2024

Uh oh!

ibm-ci-bot commented Dec 3, 2024

Uh oh!

qpdpQ commented Dec 10, 2024

Uh oh!

bluzarraga left a comment

Uh oh!

Uh oh!

Uh oh!

qpdpQ commented Jan 7, 2025

Uh oh!

bluzarraga commented Jan 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

qpdpQ commented Dec 3, 2024

Uh oh!

ibm-ci-bot commented Dec 3, 2024

Uh oh!

qpdpQ commented Dec 10, 2024

Uh oh!

bluzarraga left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

qpdpQ commented Jan 7, 2025

Uh oh!

bluzarraga commented Jan 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants