Upgrading from Qhub 0.4.4 to Nebari 2022.11.01

[rsignell-usgs]

We would like to upgrade the ESIP qhub deployment, currently qhub 0.4.4 to Nebari 2022.11.01.

We see the documentation on manual backup, and on migrating the namespaces, but are unsure whether we should try to upgrade our existing deployment, or deploy Nebari fresh and then migrate our users, namespaces, etc.

How best to proceed?

[viniciusdc]

Hi @rsignell-usgs, I think the more stable approach would be to deploy a fresh instance of nebari and migrate the data. This would make it easier to fix any conflicts with paths/groups/permissions if they do appear.

c.c. @costrouc @iameskild for a second look

[viniciusdc]

the approach would be similar to what you saw in the docs, but the manual backups would be installed into the new cluster

[iameskild]

I agree with @viniciusdc. Trying to perform an in-place upgrade will destroy all Nebari related resources so it's better to deploy a fresh cluster and migrate the data over. Then when you're confident everything has been transferred and things are working, then delete the old cluster.

[jkellndorfer]

We have a new deployment running but are unclear how to best migrate the users in keycloak. Can you point us to best practices for that?

[rsignell-usgs]

As @jkellndorfer says, we know how to do the manual backup from qhub and restore to the new nebari deployment, and we know how to redo our environments in the new namespaces, but what we don't know how to migrate all our users into the nebari keycloak.

We tried exporting from keycloak qhub and importing to nebari keycloak, but it seemed to only handle the groups, not the users?

[viniciusdc]

Sorry for the late response, folks. Regarding User migration on Keycloak. As of now, Nebari does not support this out-of-box, though this is something that we would like to do in the future.

As of now, there seem to be two ways of doing this:

• according to Keycloak docs, when starting up the Keycloak server (any startup, e,g no need to be the first one), you can add exporting settings to the service to toggle the migration system. This blog post covers the exact details in a targeted manner. So, to replicate this behavior on Nebari, we might easily override these values by populating the extraEnv filed on keycloak's values. However, this needs to be explored and tested.
• Another possible way of doing this is by communicating with Keycloak's admin API. By using the following endpoint https://$KEYCLOAK_IP/auth/admin/realms/$REALM_NAME/users we can retrieve user information from Nebari's realm. and By switching the request method to PUT, we can add new users to it -- wich could work as a migration utility, though the exported users do not come with their passwords, wich would require some exploration to generate a working/functional solution.

We will create an issue to track this discussion and explore those paths.

[rsignell-usgs]

At one time there was a user migration approach that worked: we used it to move the ESIP deployment from qhub pre-keycloak, where users were specified in the config, to qhub with keycloak. Remember this @jkellndorfer ? Could we use a similar approach here?

[jkellndorfer]

I seem to recall that we were able to export the users/groups to a json file that we could import. Have to look at notes.

[viniciusdc]

Yup, that was created to move users from Qhub 0.3.x to 0.4.x, it worked before because we used a similar system to Linux permissions and then related it to keycloak add-user command. What could be done is use the above endpoint to generate a similar .json file as the above mentioned command and then use add-user to re-include the users

[iameskild]

Hey @rsignell-usgs @jkellndorfer, I had to go back and check how we did this and it turns out we generated that "export" user json file from this script. Given that this users will likely need/want this feature in the future, I have opened this PR to allow us to interact with the Keycloak API from the Nebari CLI (as @viniciusdc described above).

Once this is merged, this would allow us to easily generate this export user json as follows:

nebari keycloak export-users -c nebari-config.yaml >> users.json

[iameskild]

Hi @rsignell-usgs @jkellndorfer did the above solution work for you?

[rsignell-usgs]

@iameskild sorry, we didn't test this -- we decided we didn't want to add all our users back anyway -- only those who were still interested and active. So we have just been adding folks back manually as they ask.

[iameskild]

No worries, adding an export-users command will surely still be useful in the future :)