Run integration tests against deployment

[soapy1]

Reference Issues or PRs

#2855, #2950

What does this implement/fix?

This PR updates the integration tests to support running against an existing cluster. You can do this by providing the EXISTING_DEPLOYMENT_DIR env var (opposed to the --cloud flag) when running the integration tests.
EXISTING_DEPLOYMENT_DIR gets a path to an existing nebari config dir (must contain the nebari-config.yaml and the output dir /stages). For example see How to test this PR?

Also adds these tests to the Local Integration Tests GHA workflow.

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Create a local deploy of nebari and then run the integration tests.

$ mkdir nebari-local
$ nebari init ....
$ nebari deploy ....
$ cd ./tests/tests_integration
$ EXISTING_DEPLOYMENT_DIR=../../nebari-local pytest -vvv -s .

[marcelovilla]

@soapy1 I know this is still a draft PR, but I wanted to check if you’re planning to address #2950. If so, it might be a good opportunity to offload as much cloud deployment logic as possible from pytest.

Let me know what you think!

[soapy1]

@marcelovilla yep, definitely can do. Just to clarify, you mean like update the ci workflows (eg. test_azure_integration.yaml) to use the change proposed by this PR?

[marcelovilla]

@soapy1 I was actually referring to the --cloud flag and the deployment fixtures, which will no longer be needed once we address #2950 (which seems to overlap with—or potentially duplicate—#2855). That said, we don’t need to remove them in this PR. I just wanted to bring it up in case it impacts your approach, especially if working around them requires extra effort when they might not exist in the future.

Regarding the use of the changes proposed in this PR for CI workflows in cloud integration tests, ideally, it would be great to have both local and cloud integration tests behave consistently, with the only difference being the deployment/destroy process. In other words, it would be beneficial if our tests could run against any Nebari cluster, regardless of how it was deployed. I’m not sure whether incorporating the changes from this PR into those CI workflows is the best approach or if we’d be better off having a separate workflow that allows running tests against any deployment. That is beyond the scope of this PR, but maybe it's worthwhile to keep it mind for the future.

Let me know if this makes sense!

[soapy1]

@marcelovilla looks like tests are passing and this is setup correctly. This PR will make the test_<cloud>_integration.yaml tests fail.
It looks like it might be ok to run these integration tests as part of the test-provider.yaml workflow. Do you have any preferences on how to use these changes to test in ci?

[marcelovilla]

@soapy1 that's great, thanks!

Now that this PR allows to run tests against an existing cluster, I think we need to split the Integration Tests step of the test_<cloud>_integration.yaml workflows into two steps: (1) deploy Nebari using the CLI and (2) run the tests using the EXISTING_DEPLOYMENT_DIR env var.

It looks like it might be ok to run these integration tests as part of the test-provider.yaml workflow.

The test-provider.yaml workflow only renders the templates used by Nebari but does not deploy anything so we wouldn't have a cluster to run the tests against.

During this discussion, I have noticed the following things:

1. We are only running the integration tests when deploying Nebari to the different cloud providers. We should be probably running deployment and end-to-end tests too.
2. Our integration tests are not great—they are just checking the healthcheck endpoints of the various services. I guess that's not too different from what we're already doing with kuberhealthy when deploying Nebari.

I think the ideal scenario would be to run all of the tests (i.e., deployment, integration, and end-to-end) against both local and cloud deployments, while having separate logic for deploying Nebari and running the tests.

I'd be happy to know your thoughts about this and see if you have any ideas on how to properly set this up. If it's easier, we can also discuss it during one of our internal syncs or community meetings.

[soapy1]

If it's easier, we can also discuss it during one of our internal syncs or community meetings.

sure, I'll pop into tomorrow's meeting. Thanks!

[soapy1]

Closing this PR in favour of a slightly different approach. Follow up PR coming soon.