Integration tests #3417
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| docs/ | ||
| tests/ | ||
| tests_integration/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| .github/workflows/test-integration.yml @ytausch | ||
| conda_forge_tick/contexts.py @ytausch | ||
| conda_forge_tick/git_utils.py @ytausch | ||
| conda_forge_tick/models/* @ytausch | ||
| tests/github_api/* @ytausch | ||
| tests/conda_forge_tick/* @ytausch | ||
| tests/model/* @ytausch | ||
| tests/test_contexts.py @ytausch | ||
| tests/test_git_utils.py @ytausch | ||
| tests_integration/* @ytausch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,319 @@ | ||
| name: Integration Tests | ||
| on: | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| merge_group: null | ||
ytausch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
|
|
||
| # Integration tests interact with GitHub resources in the integration test infrastructure and therefore | ||
| # cannot run concurrently with other integration tests. | ||
| concurrency: | ||
| group: cf-scripts-integration-tests | ||
| cancel-in-progress: false | ||
|
|
||
| defaults: | ||
| run: | ||
| shell: bash -leo pipefail {0} | ||
|
|
||
| env: | ||
| # Test Instrumentation | ||
| # Note: This passes secrets to steps that don't need them. | ||
| # We would not do this in a production setting. | ||
| BOT_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }} | ||
|
There was a problem hiding this comment. Yep I'd rather set this in each step individually. No need to overprovision. There was a problem hiding this comment. The reason for this is as follows: I once dug deeper into the question which steps of the bot need In an ideal world, this would be clearly documented for each job. Here, I did not want to make the problem worse by creating the impression that some jobs need In a separate PR, we should definitely clean this up. Added to my to-do list. |
||
| # Bot | ||
| CF_TICK_OVERRIDE_CONDA_FORGE_ORG: conda-forge-bot-staging | ||
| CF_TICK_GRAPH_DATA_BACKENDS: file | ||
| CF_FEEDSTOCK_OPS_CONTAINER_NAME: conda-forge-tick | ||
| CF_FEEDSTOCK_OPS_CONTAINER_TAG: test | ||
| # Deploy to GitHub step | ||
| CF_TICK_GRAPH_GITHUB_BACKEND_REPO: regro-staging/cf-graph-countyfair # also used in the "Install Bot Code" step | ||
| RUN_URL: "https://github.com/regro/cf-scripts/actions/runs/${{ github.run_id }}" | ||
|
|
||
| jobs: | ||
| setup-repositories: | ||
| name: Set up Integration Tests | ||
| # if triggered by pull_request, only run on non-fork PRs (secrets access needed) | ||
| # Nevertheless, this check is always run in the merge queue. | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| scenario_ids: ${{ steps.setup_repositories.outputs.scenario_ids }} | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
| with: | ||
| path: cf-scripts | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@6524bf65af31da8d45b59e8c27de4bd072b392f5 # v3.8.0 | ||
|
|
||
| - name: Build Docker Image | ||
| uses: docker/build-push-action@b32b51a8eda65d6793cd0494a773d4f6bcef32dc # v6.11.0 | ||
| with: | ||
| context: cf-scripts | ||
| push: false | ||
| load: true | ||
| tags: conda-forge-tick:test | ||
| cache-from: type=gha | ||
| cache-to: type=gha,mode=max | ||
| outputs: type=docker,dest=/tmp/image.tar | ||
|
|
||
| - name: Upload Docker Image | ||
| uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0 | ||
| with: | ||
| name: conda-forge-tick | ||
| path: /tmp/image.tar | ||
|
|
||
| - name: setup-micromamba | ||
| uses: mamba-org/setup-micromamba@4b9113af4fba0e9e1124b252dd6497a419e7396d # v1.11.0 | ||
| with: | ||
| environment-file: cf-scripts/conda-lock.yml | ||
| environment-name: cf-scripts | ||
| condarc-file: cf-scripts/autotick-bot/condarc | ||
|
|
||
| - name: Run pip install | ||
| run: | | ||
| cd cf-scripts | ||
| pip install --no-deps --no-build-isolation -e . | ||
|
|
||
| - name: Set up Integration Test Repositories | ||
| id: setup_repositories | ||
| run: | | ||
| cd cf-scripts | ||
| python -m tests_integration.setup_repositories | ||
ytausch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| env: | ||
| TEST_SETUP_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }} | ||
|
|
||
| run-test-scenario: | ||
|
There was a problem hiding this comment. This job has a lot of sequential steps that require cleanup in between. Wouldn't separate (maybe parallel) jobs be better suited for this? Or are we trying to reduce the setup overhead? There was a problem hiding this comment. Parallel does not work, because we need to run the steps of a scenario sequentially and we cannot execute scenarios in parallel. Not separating into different jobs is mainly about setup overhead: reinstalling is very quick, whilst "Download Docker Image" and "setup-micromamba" are slower:
Even with pixi, we could only get rid of the slow setup-micromamba times; reducing the overhead for "Download Docker Image" would require us to optimize the image. Additionally, I am not sure how to deal with the scenarios executed as a job matrix. I'd fear it might be complicated or impossible to execute a group of jobs sequentially with a matrix. |
||
| name: Run Scenarios | ||
| # if triggered by pull_request, only run on non-fork PRs (secrets access needed) | ||
| # Nevertheless, this check is always run in the merge queue. | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
| runs-on: ubuntu-latest | ||
| needs: setup-repositories | ||
| strategy: | ||
| matrix: | ||
| scenario_id: ${{ fromJson(needs.setup-repositories.outputs.scenario_ids) }} | ||
| max-parallel: 1 | ||
| fail-fast: false | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
| with: | ||
| path: cf-scripts | ||
| submodules: 'true' | ||
|
|
||
| - name: Download Docker Image | ||
| uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 | ||
| with: | ||
| name: conda-forge-tick | ||
| path: /tmp | ||
|
|
||
| - name: setup-micromamba | ||
| uses: mamba-org/setup-micromamba@4b9113af4fba0e9e1124b252dd6497a419e7396d # v1.11.0 | ||
| with: | ||
| environment-file: cf-scripts/conda-lock.yml | ||
| environment-name: cf-scripts | ||
| condarc-file: cf-scripts/autotick-bot/condarc | ||
|
|
||
| - name: Install Bot Code (no graph clone) | ||
| run: | | ||
| # this is the first time, so we clean disk space, but don't clone the graph | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-pull-container --no-clone-graph | ||
|
|
||
| - name: Load Docker Image | ||
| run: | | ||
| docker load --input /tmp/image.tar | ||
| docker image ls -a | ||
|
|
||
| - name: Prepare Scenario | ||
| run: | | ||
| cd cf-scripts | ||
| python -m tests_integration.step_prepare # this creates the cf-graph repository | ||
| env: | ||
| SCENARIO_ID: ${{ matrix.scenario_id }} | ||
| TEST_SETUP_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }} | ||
|
|
||
| - name: Install Bot Code | ||
| run: | | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-pull-container --no-clean-disk-space | ||
|
|
||
| - name: Start HTTP Proxy | ||
| run: | | ||
| cd cf-scripts | ||
| ./tests_integration/mock_proxy_start.sh & | ||
| sleep 10 | ||
| # Install CA Certificate | ||
| sudo wget -e use_proxy=yes -e http_proxy=127.0.0.1:8080 -O /usr/local/share/ca-certificates/mitmproxy.crt \ | ||
| http://mitm.it/cert/pem | ||
| sudo update-ca-certificates | ||
| env: | ||
| SCENARIO_ID: ${{ matrix.scenario_id }} | ||
| PYTHONPATH: ${{ github.workspace }}/cf-scripts | ||
|
|
||
| - name: "[Test] Gather all Feedstocks" | ||
| run: | | ||
| cd cf-graph | ||
ytausch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # don't use proxy for git CLI operations (applies to all further steps) | ||
| git config --global http."https://github.com".proxy "" | ||
| conda-forge-tick --debug gather-all-feedstocks | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
|
There was a problem hiding this comment. Why are these reinstalls needed? There was a problem hiding this comment. The individual steps interact with the cf-graph and conda-forge-pinning-feedstock repositories. For cf-graph, we explicitly want to test that bot steps communicate with each other via the cf-graph GitHub repository and not via some weird local state of the filesystem. Therefore, we discard cf-graph (and conda-forge-pinning-feedstock) in between steps. For example, we don't want to forgive the bot if it forgets to push changes to GitHub and changes are only visible in the local clone. Therefore, the reinstall clears up the runner and thus needs to "install" the bot again, i.e., clone the repositories again. |
||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] Make Graph (Nodes and Edges)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug make-graph --update-nodes-and-edges | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] Make Graph (Node Attributes)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug make-graph | ||
| env: | ||
| CF_TICK_FRAC_MAKE_GRAPH: 1.0 | ||
ytausch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] update-upstream-versions" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug update-upstream-versions | ||
| env: | ||
| http_proxy: http://127.0.0.1:8080 | ||
| https_proxy: http://127.0.0.1:8080 | ||
| SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt | ||
| REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt | ||
| CF_FEEDSTOCK_OPS_CONTAINER_PROXY_MODE: true | ||
| CF_FEEDSTOCK_OPS_PROXY_IN_CONTAINER: http://172.17.0.1:8080 | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] make-migrators (1/2)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug make-migrators | ||
| env: | ||
| http_proxy: http://127.0.0.1:8080 | ||
| https_proxy: http://127.0.0.1:8080 | ||
| SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt | ||
| REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt | ||
| CF_FEEDSTOCK_OPS_CONTAINER_PROXY_MODE: true | ||
| CF_FEEDSTOCK_OPS_PROXY_IN_CONTAINER: http://172.17.0.1:8080 | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] auto-tick (1/2)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug auto-tick | ||
| env: | ||
| http_proxy: http://127.0.0.1:8080 | ||
| https_proxy: http://127.0.0.1:8080 | ||
| SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt | ||
| REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt | ||
| CF_FEEDSTOCK_OPS_CONTAINER_PROXY_MODE: true | ||
| CF_FEEDSTOCK_OPS_PROXY_IN_CONTAINER: http://172.17.0.1:8080 | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] make-migrators (2/2)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug make-migrators | ||
| env: | ||
| http_proxy: http://127.0.0.1:8080 | ||
| https_proxy: http://127.0.0.1:8080 | ||
| SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt | ||
| REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt | ||
| CF_FEEDSTOCK_OPS_CONTAINER_PROXY_MODE: true | ||
| CF_FEEDSTOCK_OPS_PROXY_IN_CONTAINER: http://172.17.0.1:8080 | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Reinstall Bot Code | ||
| run: | | ||
| source cf-scripts/tests_integration/clear_runner.sh | ||
| source cf-scripts/autotick-bot/install_bot_code.sh --no-clean-disk-space --no-pull-container | ||
|
|
||
| - name: "[Test] auto-tick (2/2)" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug auto-tick | ||
| env: | ||
| http_proxy: http://127.0.0.1:8080 | ||
| https_proxy: http://127.0.0.1:8080 | ||
| SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt | ||
| REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt | ||
| CF_FEEDSTOCK_OPS_CONTAINER_PROXY_MODE: true | ||
| CF_FEEDSTOCK_OPS_PROXY_IN_CONTAINER: http://172.17.0.1:8080 | ||
|
|
||
| - name: "[Test] Deploy to GitHub" | ||
| run: | | ||
| cd cf-graph | ||
| conda-forge-tick --debug deploy-to-github | ||
|
|
||
| - name: Validate Scenario | ||
| run: | | ||
| cd cf-scripts | ||
| python -m tests_integration.step_validate | ||
| env: | ||
| SCENARIO_ID: ${{ matrix.scenario_id }} | ||
| TEST_SETUP_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }} | ||
|
|
||
| - name: Print Proxy Logs | ||
| if: always() | ||
| run: cat /tmp/mitmproxy.log | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| [submodule "tests_integration/definitions/conda-forge-pinning/resources/feedstock"] | ||
| path = tests_integration/definitions/conda-forge-pinning/resources/feedstock | ||
| url = https://github.com/conda-forge/conda-forge-pinning-feedstock.git | ||
| [submodule "tests_integration/definitions/pydantic/resources/feedstock"] | ||
| path = tests_integration/definitions/pydantic/resources/feedstock | ||
| url = https://github.com/conda-forge/pydantic-feedstock.git | ||
| [submodule "tests_integration/definitions/pydantic/resources/feedstock_v1"] | ||
| path = tests_integration/definitions/pydantic/resources/feedstock_v1 | ||
| url = https://github.com/conda-forge/pydantic-feedstock.git |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No
pushtrigger?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we add a push trigger (for main you mean?), we have even more integration test runs that block each other. Since every push to main should happen through the merge queue anyway, I don't see an advantage of adding a push trigger here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only asking because we do have it in
tests.yml. Maybe it's redundant there? @beckermrThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do bypass the merge queue sometimes, but that is after the integration tests run on the PR, so this should be fine.