Skip to content

Commit e3bc1ed

Browse files
lewisjaredlewisjared
committed
Merge remote-tracking branch 'origin/main' into execution-refactor
* origin/main: (89 commits) Do not list duplicate entries from the CLI (#210) Add changelog Match all facet values when using `FacetFilter` with `keep=False` (#209) Disable pushing container images from forks (#207) Nicer error message when specifying wrong columns in `ref datasets list` (#203) add changelog and minor changes chore: Reenable skip on push chore: Fix integration test chore: Remove slow CI job chore: Fix output directory calculation in ilamb chore: Add python version to the names fix: Sanitise directory names test: Fetch ILAMB data test: Use a non-hidden directory test: Track the output directories as artifacts chore: Remove unused comment docs: changelog chore: changelog test: Add test coverage feat: Add a timeout parameter to the solve command ...
2 parents 7b462c6 + 16caf3f commit e3bc1ed

57 files changed

Lines changed: 896 additions & 159 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci-integration.yaml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: Integration tests
2+
3+
on:
4+
# Allow manual triggering of this workflow
5+
workflow_dispatch:
6+
# Run on each push to main and tagged version
7+
push:
8+
branches: [main]
9+
tags: ['v*']
10+
# Runs every day at 2:15am (UTC) (~ midday in AEST)
11+
schedule:
12+
- cron: '2 15 * * *'
13+
14+
jobs:
15+
tests:
16+
# These tests are also run on each push to main and tagged version via the `ci` workflow
17+
if: github.event_name != 'push'
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
os: [ "ubuntu-latest" ]
22+
python-version: [ "3.10", "3.11", "3.12", "3.13" ]
23+
runs-on: "${{ matrix.os }}"
24+
defaults:
25+
run:
26+
shell: bash
27+
steps:
28+
- name: Check out repository
29+
uses: actions/checkout@v4
30+
- uses: ./.github/actions/setup
31+
with:
32+
python-version: ${{ matrix.python-version }}
33+
- name: Run tests
34+
run: |
35+
make fetch-test-data
36+
make test
37+
tests-slow:
38+
env:
39+
REF_TEST_OUTPUT: "test-outputs"
40+
strategy:
41+
fail-fast: false
42+
matrix:
43+
os: [ "ubuntu-latest" ]
44+
python-version: [ "3.10" ]
45+
runs-on: "${{ matrix.os }}"
46+
defaults:
47+
run:
48+
shell: bash
49+
steps:
50+
- name: Check out repository
51+
uses: actions/checkout@v4
52+
- uses: ./.github/actions/setup
53+
with:
54+
python-version: ${{ matrix.python-version }}
55+
- name: Run tests
56+
run: |
57+
make fetch-test-data
58+
uv run python scripts/fetch-ilamb-data.py ilamb.txt
59+
make test-integration-slow
60+
# Upload the scratch and results directories as artifacts
61+
- name: Upload scratch artifacts
62+
uses: actions/upload-artifact@v4
63+
if: always()
64+
with:
65+
name: integration-output-${{ matrix.python-version }}
66+
path: ${{ env.REF_TEST_OUTPUT }}
67+
retention-days: 7

‎.github/workflows/ci.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ jobs:
1818
run: make pre-commit
1919

2020
tests:
21+
env:
22+
REF_TEST_OUTPUT: "test-outputs"
2123
strategy:
2224
fail-fast: false
2325
matrix:
@@ -45,6 +47,13 @@ jobs:
4547
uses: codecov/codecov-action@v5
4648
env:
4749
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
50+
- name: Upload scratch artifacts
51+
uses: actions/upload-artifact@v4
52+
if: always()
53+
with:
54+
name: test-output-${{ matrix.python-version }}
55+
path: ${{ env.REF_TEST_OUTPUT }}
56+
retention-days: 7
4857

4958
imports-without-extras:
5059
strategy:

‎.github/workflows/containers.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: Build Containers
2+
3+
on:
4+
pull_request:
5+
workflow_dispatch:
6+
push:
7+
branches:
8+
- "main"
9+
tags:
10+
- "v*"
11+
12+
permissions:
13+
contents: read
14+
packages: write
15+
attestations: write
16+
id-token: write
17+
18+
jobs:
19+
ref:
20+
name: REF
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v4
24+
- uses: docker/setup-buildx-action@v3
25+
- uses: docker/login-action@v3
26+
with:
27+
registry: ghcr.io
28+
username: ${{ github.actor }}
29+
password: ${{ secrets.GITHUB_TOKEN }}
30+
- uses: docker/metadata-action@v5
31+
id: metadata
32+
with:
33+
images: ghcr.io/${{ github.repository_owner }}/ref
34+
tags: |
35+
type=ref,event=pr
36+
type=ref,event=tag
37+
# set latest tag for default branch
38+
type=raw,value=main,enable={{is_default_branch}}
39+
- uses: docker/build-push-action@v4
40+
id: push
41+
with:
42+
cache-from: type=gha
43+
cache-to: type=gha,mode=max
44+
context: .
45+
file: packages/ref/Dockerfile
46+
push: ${{ ! github.event.pull_request.head.repo.fork }}
47+
sbom: true
48+
tags: ${{ steps.metadata.outputs.tags }}
49+
labels: ${{ steps.metadata.outputs.labels }}
50+
- name: Attest to REF image
51+
uses: actions/attest-build-provenance@v2
52+
if: ${{ ! github.event.pull_request.head.repo.fork }}
53+
with:
54+
subject-name: ghcr.io/${{ github.repository_owner }}/ref
55+
subject-digest: ${{ steps.push.outputs.digest }}
56+
push-to-registry: true

‎Makefile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,11 @@ clean-sample-data: ## clean up the sample data
4444
build: clean ## build the packages to be deployed to PyPI
4545
cp LICENCE NOTICE packages/ref
4646
cp LICENCE NOTICE packages/ref-core
47+
cp LICENCE NOTICE packages/ref-celery
48+
cp LICENCE NOTICE packages/ref-metrics-example
49+
cp LICENCE NOTICE packages/ref-metrics-esmvaltool
50+
cp LICENCE NOTICE packages/ref-metrics-ilamb
51+
cp LICENCE NOTICE packages/ref-metrics-pmp
4752
uv build --package cmip_ref --no-sources
4853
uv build --package cmip_ref_core --no-sources
4954
uv build --package cmip_ref_celery --no-sources
@@ -102,6 +107,12 @@ test-metrics-pmp: ## run the tests
102107

103108
.PHONY: test-integration
104109
test-integration: ## run the integration tests
110+
uv run \
111+
pytest tests -m "not slow" \
112+
-r a -v
113+
114+
.PHONY: test-integration-slow
115+
test-integration-slow: ## run the integration tests, including the slow tests which may take a while
105116
uv run \
106117
pytest tests \
107118
-r a -v

‎changelog/133.fix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fixed the validation error when 'attributes' value is a dict

‎changelog/156.feature.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Build and publish container images to [Github Container Registry](https://github.com/Climate-REF/climate-ref/pkgs/container/ref)

‎changelog/166.improvement.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Add activity and institute to ESMValTool recipes to allow running with models
2+
and experiments that are not in the CMIP6 controlled vocabulary.

‎changelog/170.fix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Enforce the use of relative paths when copying files after an execution. This resolves an issue where files were not being copied to the correct location causing failures in PMP.

‎changelog/173.feature.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Enable more variability modes for PMP modes of variability metrics

‎changelog/174.fix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
If no obs4mips-compliant reference dataset is found in specified directory, give a meaningful error message.

0 commit comments

Comments
 (0)