Use GitHub actions for unit testing (#745)

peanutfun · web-flow · commit d8eaf55a7d7b · 2023-09-20T08:58:09.000+02:00
* Add GitHub Actions and Dockerfile for test environment * Fix env var query in github workflow * Revert "Fix env var query in github workflow" This reverts commit dc37cec. * Try specifying Docker tag with variables * Hardcode Docker Hub namespace * Conditionally run jobs and try to fix unittest job * Fix syntax error in github workflow * Do not call make in github workflow * Try fixing the shell inside Docker container for github workflow * Call micromamba directly in github workflow * Use 'micromamba install' because 'update' does not update pip deps * Split GitHub workflows * Consolidate GitHub Workflows * Try fixing syntax for GitHub workflow expression * Fix typo in docker workflow * Fix list input for docker.yml workflow * Run all tests in GitHub workflow * Do not use 'defaults' in workflow * Fix stray 'defaults' in workflow * Rename testing steps * Do not use multiprocessing for integration tests * Revert climada.conf to 9302bf2 * Revert "Do not use multiprocessing for integration tests" This reverts commit 9be8632. The command does not actually use multiprocessing but makes coverage aware of multiprocessing used by the called tests. * Skip failing integration tests for now * Add action to publish test results * Run pipeline on branches and pull requests * Grant permissions from top-level job * Use other XML test reports for displaying results * Add new workflow using micromamba directly * Add data and notebook tests to new workflow * Separately build conda environment This avoids building a new environment for every job. * Add integration tests and separate test result reporting * Fix missing 'runs-on' * ci: Add ipython to extra requirements * ci: Fix publish results * Fix permissions. * Fix path to artifacts. * Remove all but the mamba pipelines * Run test matrix with different Python versions Remove version pip for Python in environment specs. * Fix Python version parsing in CI * Add linting job to CI * Skip integration tests in GitHub CI * Use custom conda environment cache key in CI * Add matchers and custom linting step to CI * Ignore pylint errors in CI * ci: Debug matcher path * ci: Lint with ruff * ci: Fix Python target version for ruff * ci: Add permissions for publishing checks to test job * Revert "ci: Lint with ruff" This reverts commits 41bb4c2 and 60ee28f. * Update CI configuration * Use new micromamba action. * Do not lint. * Upload coverage reports as artifacts. * Report test results for every Python version individually. * Stop testing Python 3.11 due to incompatibilities with dataclass * Revert "Stop testing Python 3.11 due to incompatibilities with dataclass" This reverts commit 6ed83e6. * Set environment caching for GitHub workflow * Remove unused files * Fix mutable default values in ImpactFreqCurve * Fix Nightlight and LitPop tests * Move file checks to integration tests. They require file downloads. * Fix assertion for almost equal arrays. * Fix accuracy in default Emanuel impact func test * Improve names in Github CI * Disallow Python 3.11 for Climada The unsequa module throws segmentation faults while unit testing. * Try testing Python v3.11 again * Improve GitHub Actions documentation * Add introduction to GitHub Actions to the docs. * Add information to the CI guide. * Improve comments in the ci.yml workflow definition.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,70 @@
+name: GitHub CI
+
+# Execute this for every push
+on: [push]
+
+# Use bash explicitly for being able to enter the conda environment
+defaults:
+  run:
+    shell: bash -l {0}
+
+jobs:
+  build-and-test:
+    name: Build Env, Install, Unit Tests
+    runs-on: ubuntu-latest
+    permissions:
+      # For publishing results
+      checks: write
+
+    # Run this test for different Python versions
+    strategy:
+      # Do not abort other tests if only a single one fails
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11"]
+
+    steps:
+      -
+        name: Checkout Repo
+        uses: actions/checkout@v3
+      -
+        # Store the current date to use it as cache key for the environment
+        name: Get current date
+        id: date
+        run: echo "date=$(date +%Y-%m-%d)" >> "${GITHUB_OUTPUT}"
+      -
+        name: Create Environment with Mamba
+        uses: mamba-org/setup-micromamba@v1
+        with:
+          environment-name: climada_env_${{ matrix.python-version }}
+          environment-file: requirements/env_climada.yml
+          create-args: >-
+            python=${{ matrix.python-version }}
+            make
+          init-shell: >-
+            bash
+          # Persist environment for branch, Python version, single day
+          cache-environment-key: env-${{ github.ref }}-${{ matrix.python-version }}-${{ steps.date.outputs.date }}
+      -
+        name: Install CLIMADA
+        run: |
+          python -m pip install ".[test]"
+      -
+        name: Run Unit Tests
+        run: |
+          make unit_test
+      -
+        name: Publish Test Results
+        uses: EnricoMi/publish-unit-test-result-action@v2
+        if: always()
+        with:
+          junit_files: tests_xml/tests.xml
+          check_name: "Unit Test Results Python ${{ matrix.python-version }}"
+          comment_mode: "off"
+      -
+        name: Upload Coverage Reports
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: coverage-report-unittests-py${{ matrix.python-version }}
+          path: coverage/
diff --git a/climada/engine/impact.py b/climada/engine/impact.py
@@ -21,7 +21,7 @@
 
 __all__ = ['ImpactFreqCurve', 'Impact']
 
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 import logging
 import copy
 import csv
@@ -1785,10 +1785,10 @@ class ImpactFreqCurve():
     """Impact exceedence frequency curve.
     """
 
-    return_per : np.array = np.array([])
+    return_per : np.ndarray = field(default_factory=lambda: np.empty(0))
     """return period"""
 
-    impact : np.array = np.array([])
+    impact : np.ndarray = field(default_factory=lambda: np.empty(0))
     """impact exceeding frequency"""
 
     unit : str = ''
diff --git a/climada/entity/exposures/test/test_litpop.py b/climada/entity/exposures/test/test_litpop.py
@@ -317,7 +317,7 @@ def test_gridpoints_core_calc_offsets_exp_rescale(self):
         self.assertEqual(result_array.shape, results_check.shape)
         self.assertAlmostEqual(result_array.sum(), tot)
         self.assertEqual(result_array[1,2], results_check[1,2])
-        np.testing.assert_array_almost_equal_nulp(result_array, results_check)
+        np.testing.assert_allclose(result_array, results_check)
 
     def test_grp_read_pass(self):
         """test _grp_read() to pass and return either dict with admin1 values or None"""
diff --git a/climada/entity/exposures/test/test_nightlight.py b/climada/entity/exposures/test/test_nightlight.py
@@ -56,22 +56,6 @@ def test_required_files(self):
         self.assertRaises(ValueError, nightlight.get_required_nl_files,
                           (-90, 90))
 
-    def test_check_files_exist(self):
-        """Test check_nightlight_local_file_exists"""
-        # If invalid directory is supplied it has to fail
-        try:
-            nightlight.check_nl_local_file_exists(
-                np.ones(np.count_nonzero(BM_FILENAMES)), 'Invalid/path')[0]
-            raise Exception("if the path is not valid, check_nl_local_file_exists should fail")
-        except ValueError:
-            pass
-        files_exist = nightlight.check_nl_local_file_exists(
-            np.ones(np.count_nonzero(BM_FILENAMES)), SYSTEM_DIR)
-        self.assertTrue(
-            files_exist.sum() > 0,
-            f'{files_exist} {BM_FILENAMES}'
-        )
-
     def test_download_nightlight_files(self):
         """Test check_nightlight_local_file_exists"""
         # Not the same length of arguments
@@ -118,42 +102,6 @@ def test_get_required_nl_files(self):
         bool = np.array_equal(np.array([0, 0, 0, 0, 0, 0, 1, 0]), req_files)
         self.assertTrue(bool)
 
-    def test_check_nl_local_file_exists(self):
-        """ Test that an array with the correct number of already existing files
-            is produced, the LOGGER messages logged and the ValueError raised. """
-
-        # check logger messages by giving a to short req_file
-        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='WARNING') as cm:
-            nightlight.check_nl_local_file_exists(required_files = np.array([0, 0, 1, 1]))
-        self.assertIn('The parameter \'required_files\' was too short and is ignored',
-                      cm.output[0])
-
-        # check logger message: not all files are available
-        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='DEBUG') as cm:
-            nightlight.check_nl_local_file_exists()
-        self.assertIn('Not all satellite files available. Found ', cm.output[0])
-        self.assertIn(f' out of 8 required files in {Path(SYSTEM_DIR)}', cm.output[0])
-
-        # check logger message: no files found in checkpath
-        check_path = Path('climada/entity/exposures')
-        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='INFO') as cm:
-            # using a random path where no files are stored
-            nightlight.check_nl_local_file_exists(check_path=check_path)
-        self.assertIn(f'No satellite files found locally in {check_path}',
-                      cm.output[0])
-
-        # test raises with wrong path
-        check_path = Path('/random/wrong/path')
-        with self.assertRaises(ValueError) as cm:
-            nightlight.check_nl_local_file_exists(check_path=check_path)
-        self.assertEqual(f'The given path does not exist: {check_path}',
-                         str(cm.exception))
-
-        # test that files_exist is correct
-        files_exist = nightlight.check_nl_local_file_exists()
-        self.assertGreaterEqual(int(sum(files_exist)), 3)
-        self.assertLessEqual(int(sum(files_exist)), 8)
-
 # Execute Tests
 if __name__ == "__main__":
     TESTS = unittest.TestLoader().loadTestsFromTestCase(TestNightLight)
diff --git a/climada/entity/impact_funcs/test/test_tc.py b/climada/entity/impact_funcs/test/test_tc.py
@@ -39,21 +39,30 @@ def test_default_values_pass(self):
         self.assertTrue(np.array_equal(imp_fun.intensity, np.arange(0, 121, 5)))
         self.assertTrue(np.array_equal(imp_fun.paa, np.ones((25,))))
         self.assertTrue(np.array_equal(imp_fun.mdd[0:6], np.zeros((6,))))
-        self.assertTrue(np.array_equal(imp_fun.mdd[6:10],
-                        np.array([0.0006753419543492556, 0.006790495604105169,
-                                  0.02425254393374475, 0.05758706257339458])))
-        self.assertTrue(np.array_equal(imp_fun.mdd[10:15],
-                        np.array([0.10870556455111065, 0.1761433569521351,
-                                  0.2553983618763961, 0.34033822528795565,
-                                  0.4249447743109498])))
-        self.assertTrue(np.array_equal(imp_fun.mdd[15:20],
-                        np.array([0.5045777092933046, 0.576424302849412,
-                                  0.6393091739184916, 0.6932203123193963,
-                                  0.7388256596555696])))
-        self.assertTrue(np.array_equal(imp_fun.mdd[20:25],
-                        np.array([0.777104531116526, 0.8091124649261859,
-                                  0.8358522190681132, 0.8582150905529946,
-                                  0.8769633232141456])))
+        np.testing.assert_allclose(
+            imp_fun.mdd[6:25],
+            [
+                0.0006753419543492556,
+                0.006790495604105169,
+                0.02425254393374475,
+                0.05758706257339458,
+                0.10870556455111065,
+                0.1761433569521351,
+                0.2553983618763961,
+                0.34033822528795565,
+                0.4249447743109498,
+                0.5045777092933046,
+                0.576424302849412,
+                0.6393091739184916,
+                0.6932203123193963,
+                0.7388256596555696,
+                0.777104531116526,
+                0.8091124649261859,
+                0.8358522190681132,
+                0.8582150905529946,
+                0.8769633232141456,
+            ],
+        )
 
     def test_values_pass(self):
         """Compute mdr interpolating values."""
diff --git a/climada/test/test_nightlight.py b/climada/test/test_nightlight.py
@@ -254,6 +254,58 @@ def test_untar_noaa_stable_nighlight(self):
         self.assertIn('found more than one potential intensity file in', cm.output[0])
         path_tar.unlink()
 
+    def test_check_nl_local_file_exists(self):
+        """ Test that an array with the correct number of already existing files
+            is produced, the LOGGER messages logged and the ValueError raised. """
+
+        # check logger messages by giving a to short req_file
+        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='WARNING') as cm:
+            nightlight.check_nl_local_file_exists(required_files = np.array([0, 0, 1, 1]))
+        self.assertIn('The parameter \'required_files\' was too short and is ignored',
+                      cm.output[0])
+
+        # check logger message: not all files are available
+        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='DEBUG') as cm:
+            nightlight.check_nl_local_file_exists()
+        self.assertIn('Not all satellite files available. Found ', cm.output[0])
+        self.assertIn(f' out of 8 required files in {Path(SYSTEM_DIR)}', cm.output[0])
+
+        # check logger message: no files found in checkpath
+        check_path = Path('climada/entity/exposures')
+        with self.assertLogs('climada.entity.exposures.litpop.nightlight', level='INFO') as cm:
+            # using a random path where no files are stored
+            nightlight.check_nl_local_file_exists(check_path=check_path)
+        self.assertIn(f'No satellite files found locally in {check_path}',
+                      cm.output[0])
+
+        # test raises with wrong path
+        check_path = Path('/random/wrong/path')
+        with self.assertRaises(ValueError) as cm:
+            nightlight.check_nl_local_file_exists(check_path=check_path)
+        self.assertEqual(f'The given path does not exist: {check_path}',
+                         str(cm.exception))
+
+        # test that files_exist is correct
+        files_exist = nightlight.check_nl_local_file_exists()
+        self.assertGreaterEqual(int(sum(files_exist)), 3)
+        self.assertLessEqual(int(sum(files_exist)), 8)
+
+    def test_check_files_exist(self):
+        """Test check_nightlight_local_file_exists"""
+        # If invalid directory is supplied it has to fail
+        try:
+            nightlight.check_nl_local_file_exists(
+                np.ones(np.count_nonzero(BM_FILENAMES)), 'Invalid/path')[0]
+            raise Exception("if the path is not valid, check_nl_local_file_exists should fail")
+        except ValueError:
+            pass
+        files_exist = nightlight.check_nl_local_file_exists(
+            np.ones(np.count_nonzero(BM_FILENAMES)), SYSTEM_DIR)
+        self.assertTrue(
+            files_exist.sum() > 0,
+            f'{files_exist} {BM_FILENAMES}'
+        )
+
 # Execute Tests
 if __name__ == "__main__":
     TESTS = unittest.TestLoader().loadTestsFromTestCase(TestNightlight)
diff --git a/doc/guide/Guide_Continuous_Integration_and_Testing.ipynb b/doc/guide/Guide_Continuous_Integration_and_Testing.ipynb
@@ -299,7 +299,12 @@
     "\n",
     "- All tests must pass before submitting a pull request.\n",
     "- Integration tests don't run on feature branches in Jenkins, therefore developers are requested to run them locally.\n",
-    "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed."
+    "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed.\n",
+    "\n",
+    "#### GitHub Actions\n",
+    "\n",
+    "We adopted test automation via GitHub Actions in an experimental state.\n",
+    "See [GitHub Actions CI](github-actions.rst) for details."
    ]
   },
   {
diff --git a/doc/guide/github-actions.rst b/doc/guide/github-actions.rst
@@ -0,0 +1,29 @@
+=================
+GitHub Actions CI
+=================
+
+CLIMADA has been using a private Jenkins instance for automated testing (Continuous Integration, CI), see :doc:`Guide_Continuous_Integration_and_Testing`.
+We recently adopted `GitHub Actions <https://docs.github.com/en/actions>`_ for automated unit testing.
+GitHub Actions is a service provided by GitHub, which lets you configure CI/CD pipelines based on YAML configuration files.
+GitHub provides servers which ample computational resources to create software environments, install software, test it, and deploy it.
+See the `GitHub Actions Overview <https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions>`_ for a technical introduction, and the `Workflow Syntax <https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions>`_ for a reference of the pipeline definitions.
+
+The CI results for each pull request can be inspected in the "Checks" tab.
+For GitHub Actions, users can inspect the logs of every step for every job.
+
+.. note::
+
+   As of CLIMADA v4.0, the default CI technology remains Jenkins.
+   GitHub Actions CI is currently considered experimental for CLIMADA development.
+
+---------------------
+Unit Testing Pipeline
+---------------------
+
+This pipeline is defined by the ``.github/workflows/ci.yml`` file.
+It contains a single job which will create a CLIMADA environment with Mamba for multiple Python versions, install CLIMADA, run the unit tests, and report the test coverage as well as the simplified test results.
+The job has a `strategy <https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategy>`_ which runs it for multiple times for different Python versions.
+This way, we make sure that CLIMADA is compatible with all currently supported versions of Python.
+
+The coverage reports in HTML format will be uploaded as job artifacts and can be downloaded as ZIP files.
+The test results are simple testing summaries that will appear as individual checks/jobs after the respective job completed.
diff --git a/doc/index.rst b/doc/index.rst
@@ -103,6 +103,7 @@ Jump right in:
    Performance and Best Practices <guide/Guide_Py_Performance>
    Coding Conventions <guide/Guide_Miscellaneous>
    Building the Documentation <README>
+   guide/github-actions
 
 
 .. toctree::
diff --git a/requirements/env_climada.yml b/requirements/env_climada.yml
@@ -26,7 +26,7 @@ dependencies:
   - pycountry>=22.3
   - pyepsg>=0.4
   - pytables>=3.7
-  - python=3.9
+  - python>=3.9,<3.12
   - pyxlsb>=1.0
   - rasterio>=1.3
   - requests>=2.31
diff --git a/setup.py b/setup.py
@@ -57,6 +57,8 @@
 
     keywords='climate adaptation',
 
+    python_requires=">=3.9,<3.12",
+
     install_requires=[
         'bottleneck',
         'cartopy',

Original file line number	Diff line number	Diff line change
`@@ -299,7 +299,12 @@`
`299`	`299`	`"\n",`
`300`	`300`	`"- All tests must pass before submitting a pull request.\n",`
`301`	`301`	`"- Integration tests don't run on feature branches in Jenkins, therefore developers are requested to run them locally.\n",`
`302`		`- "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed."`
	`302`	`+ "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed.\n",`
	`303`	`+ "\n",`
	`304`	`+ "#### GitHub Actions\n",`
	`305`	`+ "\n",`
	`306`	`+ "We adopted test automation via GitHub Actions in an experimental state.\n",`
	`307`	`+ "See [GitHub Actions CI](github-actions.rst) for details."`
`303`	`308`	`]`
`304`	`309`	`},`
`305`	`310`	`{`