Add workflow to run conda-store user journey tests

[soapy1]

Reference Issues or PRs

fixes #2760

• Runs conda-store integration tests as part of test_local_integration
• Adds the option to specify a --group flag when adding a keycloak user

To create a keycloak user with access to a particular group, run the command

nebari keycloak adduser --user username password --config nebari-config.yaml --groups mygroup

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe): CI change

[soapy1]

depends on conda-incubator/conda-store#1040

[viniciusdc]

Since these tests depend on a live cluster, I suggest extending the already existing local integration tests with this extra check https://github.com/nebari-dev/nebari/blob/main/.github/workflows/test_local_integration.yaml -- since it uses CiRun under the hood you should not be affected by any flakiness of the GH runner in case of constrained resources.

[viniciusdc]

Hey @soapy1 no rush on this, but could you update us on this? Just considering adding to our next Feb release

[soapy1]

Heya @viniciusdc I've gotten a bit stuck on this one. I'm not able to find a way to pull an admin token for conda-store from nebari in order to run the tests (full notes in this comment - #2895 (comment)).

Things that I've tried:

• pulling a service token from the terraform state - this works but is a little janky
• logging in with the test user and generating a token - this user does not have enough permissions to create an admin token
• logging in with the root keycloak user to generate a token - this user also doesn't have enough permissions to create an admin token

Other options I've considered (but haven't tried yet):

• is there a way to seed a new service token for conda store (like a config option to add some service tokens)
• is there a way to use the keycloak api to add a user with admin permissions, and then generate a token from there

Definitely open to other ideas if you have a suggestion!

[viniciusdc]

I left a comment to your thread there, but basically I think your best options are those two:

is there a way to seed a new service token for conda store (like a config option to add some service tokens)
is there a way to use the keycloak api to add a user with admin permissions, and then generate a token from there

1. For the first one, conda-store has a service token attribute that could be used, though this would require some extension to allow this to be customizable by the user to include new services:

nebari/src/_nebari/stages/kubernetes_services/__init__.py

Lines 446 to 448 in ecaa94b

	conda_store_service_token_scopes: Dict[str, Dict[str, Any]] = Field(
	alias="conda-store-service-token-scopes"
	)

nebari/qhub/stages/input_vars.py

Lines 271 to 278 in 81ff2fb

	"conda-store-service-token-scopes": {
	"cdsdashboards": {
	"primary_namespace": "cdsdashboards",
	"role_bindings": {
	"*/*": ["viewer"],
	},
	}
	},

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/server.tf

Lines 31 to 33 in ecaa94b

	service-tokens = {
	for service, value in var.services : base64encode(random_password.conda_store_service_token[service].result) => value
	}

Used for example here:

nebari/src/_nebari/stages/kubernetes_services/template/jupyterhub.tf

Line 181 in ecaa94b

conda-store-jhub-apps-token = module.kubernetes-conda-store-server.service-tokens.jhub-apps

2. I think this one might be easier to get going, nebari already has a command to create users from CLI -- nebari keycloak --adduser, which could be used for creating the new user. Right now, there is no way of programmatically changing the group, we can have a script under tests that do this:

try:
    # Get user ID
    users = keycloak_admin.get_users({"username": USER_TO_ADD})
    if not users:
        print(f"User '{USER_TO_ADD}' not found.")
        exit(1)
    
    user_id = users[0]["id"]
    
    # Get group ID
    groups = keycloak_admin.get_groups()
    group_id = next((g["id"] for g in groups if g["name"] == "superadmin"), None)
    
    if not group_id:
        print(f"Group 'superadmin' not found.")
        exit(1)
    
    # Add user to group
    keycloak_admin.group_user_add(user_id, group_id)
    print(f"User '{USER_TO_ADD}' successfully added to 'superadmin' group.")

except KeycloakError as e:
    print(f"Error: {e}")

for the main API KeycloakAdmin class, you might be able to use this function

nebari/src/_nebari/keycloak.py

Line 104 in ecaa94b

def get_keycloak_admin_from_config(config: schema.Main):

I wonder if this should be configured directly trough nebari's CLI anyway, we already have root access level credentials in the yaml, so it would not be a security overlook to allow the group in which the user will be created to appear as a custom flag as well....

[soapy1]

I wonder if this should be configured directly trough nebari's CLI anyway

ya, I was thinking about adding this. Something like nebari keycloak adduser -c nebari-config.yaml --user test password --role superadmin. I think it would be good to have some mechanism so that users can't create a user with more permissions than they ought to be able to.

so it would not be a security overlook to allow the group in which the user will be created to appear as a custom flag as well

could make a v1 of this be something like only root users are allowed to set a role?

[soapy1]

I'm not a fan of this approach to passing arguments around but did this to follow the pattern already defined in this part of the code.
I would be happy to refactor this module to make it a more developer friendly if that's something the nebari team is into.
Could also split that into a separate PR.

[dcmcand]

why don't you split that into a different pr. Thanks!

[soapy1]

Looks like this functionality is also implemented in #2917. Will rebase my changes when it get's merged.

[viniciusdc]

@soapy1 this requires the nebari keycloak command update to work, right? If that's the case, @Adam-D-Lewis do you mind splitting that ENH into another PR to decouple from the service-account one? We can also keep it in here as well

[Adam-D-Lewis]

@soapy1 this requires the nebari keycloak command update to work, right? If that's the case, @Adam-D-Lewis do you mind splitting that ENH into another PR to decouple from the service-account one? We can also keep it in here as well

Yeah, I can split that out into a separate PR today.

[Adam-D-Lewis]

Here's the keycloak cli refactor PR - #2968

[dcmcand]

@soapy1 please let me know when this is ready to go so we can get it merged

[marcelovilla]

This looks great @soapy1 !

In the spirit of achieving testing parity, I think it would be worthwhile to also run these tests on cloud deployments. That's beyond the scope of this PR but something to keep in mind when we start running other test suites in cloud deployments too.