Merge branch 'calibrate-impact-functions' into cross-calibrate-impact-functions

Author: peanutfun

AUTHORS.md

@@ -29,4 +29,6 @@
 * Raphael Portmann
 * Nicolas Colombi
 * Leonie Villiger
-* Timo Schmid
+* Timo Schmid
+* Kam Lam Yeung
+* Sarah Hülsen

CHANGELOG.md

@@ -10,18 +10,41 @@ Code freeze date: YYYY-MM-DD
 
 ### Dependency Changes
 
+Added:
+
+- `pyproj` >=3.5
+- `pyarrow` >=14.0
+- `numexpr` >=2.8
+
+Removed:
+
+- `proj` (in favor of `pyproj`)
+
 ### Added
 
 - `climada.util.calibrate` module for calibrating impact functions [#692](https://github.com/CLIMADA-project/climada_python/pull/692)
+- Convenience method `api_client.Client.get_dataset_file`, combining `get_dataset_info` and `download_dataset`, returning a single file objet. [#821](https://github.com/CLIMADA-project/climada_python/pull/821)
+- Read and Write methods to and from csv files for the `DiscRates` class. [#818](ttps://github.com/CLIMADA-project/climada_python/pull/818)
 
 ### Changed
 
+- Update Developer and Installation Guides for easier accessibility by new developers. [808](https://github.com/CLIMADA-project/climada_python/pull/808)
+- Add `shapes` argument to `geo_im_from_array` to allow flexible turning on/off of plotting coastline in `plot_intensity`. [#805](https://github.com/CLIMADA-project/climada_python/pull/805)
 - Update `CONTRIBUTING.md` to better explain types of contributions to this repository [#797](https://github.com/CLIMADA-project/climada_python/pull/797)
 - The default tile layer in Exposures maps is not Stamen Terrain anymore, but [CartoDB Positron](https://github.com/CartoDB/basemap-styles). Affected methods are `climada.engine.Impact.plot_basemap_eai_exposure`,`climada.engine.Impact.plot_basemap_impact_exposure` and `climada.entity.Exposures.plot_basemap`. [#798](https://github.com/CLIMADA-project/climada_python/pull/798)
+- Recommend using Mamba instead of Conda for installing CLIMADA [#809](https://github.com/CLIMADA-project/climada_python/pull/809)
+- `Hazard.from_xarray_raster` now allows arbitrary values as 'event' coordinates [#837](https://github.com/CLIMADA-project/climada_python/pull/837)
+- `climada.test.get_test_file` now compares the version of the requested test dataset with the version of climada itself and selects the most appropriate dataset. In this way a test file can be updated without the need of changing the code of the unittest. [#822](https://github.com/CLIMADA-project/climada_python/pull/822)
+- Explicitly require `pyproj` instead of `proj` (the latter is now implicitly required) [#845](https://github.com/CLIMADA-project/climada_python/pull/845)
 
 ### Fixed
 
+- `Hazard.from_xarray_raster` now stores strings as default values for `Hazard.event_name` [#795](https://github.com/CLIMADA-project/climada_python/pull/795)
 - Fix the dist_approx util function when used with method="geosphere" and log=True and points that are very close. [#792](https://github.com/CLIMADA-project/climada_python/pull/792)
+- `climada.util.yearsets.sample_from_poisson`: fix a bug ([#819](https://github.com/CLIMADA-project/climada_python/issues/819)) and inconsistency that occurs when lambda events per year (`lam`) are set to 1. [[#823](https://github.com/CLIMADA-project/climada_python/pull/823)]
+- In the TropCyclone class in the Holland model 2008 and 2010 implementation, a doublecounting of translational velocity is removed [#833](https://github.com/CLIMADA-project/climada_python/pull/833)
+- `climada.util.test.test_finance` and `climada.test.test_engine` updated to recent input data from worldbank [#841](https://github.com/CLIMADA-project/climada_python/pull/841)
+- Set `nodefaults` in Conda environment specs because `defaults` are not compatible with conda-forge [#845](https://github.com/CLIMADA-project/climada_python/pull/845)
 
 ### Deprecated
 

README.md

@@ -21,13 +21,14 @@ This is the Python (3.9+) version of CLIMADA - please see [here](https://github.
 ## Getting started
 
 CLIMADA runs on Windows, macOS and Linux.
-The released versions of the CLIMADA core can be installed directly through Anaconda:
+The released versions of CLIMADA are available from [conda-forge](https://anaconda.org/conda-forge/climada).
+Use the [Mamba](https://mamba.readthedocs.io/en/latest/) package manager to install it:
 
 ```shell
-conda install -c conda-forge climada
+mamba install -c conda-forge climada
 ```
 
-It is **highly recommended** to install CLIMADA into a **separate** Anaconda environment.
+It is **highly recommended** to install CLIMADA into a **separate** Conda environment.
 See the [installation guide](https://climada-python.readthedocs.io/en/latest/guide/install.html) for further information.
 
 Follow the [tutorials](https://climada-python.readthedocs.io/en/stable/tutorial/1_main_climada.html) in a Jupyter Notebook to see what can be done with CLIMADA and how.

climada/engine/test/test_cost_benefit.py

@@ -35,7 +35,7 @@
 from climada.test import get_test_file
 
 
-HAZ_TEST_MAT = get_test_file('atl_prob_no_name')
+HAZ_TEST_MAT = get_test_file('atl_prob_no_name', file_format='matlab')
 ENT_TEST_MAT = get_test_file('demo_today', file_format='MAT-file')
 
 

climada/engine/unsequa/test/test_unsequa.py

@@ -40,12 +40,9 @@
                                     TEST_UNC_OUTPUT_IMPACT, TEST_UNC_OUTPUT_COSTBEN)
 from climada.util.api_client import Client
 
-apiclient = Client()
-ds = apiclient.get_dataset_info(name=TEST_UNC_OUTPUT_IMPACT, status='test_dataset')
-_target_dir, [test_unc_output_impact] = apiclient.download_dataset(ds)
 
-ds = apiclient.get_dataset_info(name=TEST_UNC_OUTPUT_COSTBEN, status='test_dataset')
-_target_dir, [test_unc_output_costben] = apiclient.download_dataset(ds)
+test_unc_output_impact = Client().get_dataset_file(name=TEST_UNC_OUTPUT_IMPACT, status='test_dataset')
+test_unc_output_costben = Client().get_dataset_file(name=TEST_UNC_OUTPUT_COSTBEN, status='test_dataset')
 
 
 def impf_dem(x_paa=1, x_mdd=1):

climada/entity/disc_rates/base.py

@@ -259,9 +259,11 @@ def from_mat(cls, file_name, var_names=None):
         return cls(years=years, rates=rates)
 
     def read_mat(self, *args, **kwargs):
-        """This function is deprecated, use DiscRates.from_mats instead."""
-        LOGGER.warning("The use of DiscRates.read_mats is deprecated."
-                       "Use DiscRates.from_mats instead.")
+        """This function is deprecated, use ``DiscRates.from_mat`` instead."""
+        LOGGER.warning(
+            "The use of DiscRates.read_mat is deprecated."
+            "Use DiscRates.from_mat instead."
+        )
         self.__dict__ = DiscRates.from_mat(*args, **kwargs).__dict__
 
     @classmethod
@@ -307,8 +309,7 @@ def read_excel(self, *args, **kwargs):
         """This function is deprecated, use DiscRates.from_excel instead."""
         LOGGER.warning("The use of DiscRates.read_excel is deprecated."
                        "Use DiscRates.from_excel instead.")
-        self.__dict__ = DiscRates.from_mat(*args, **kwargs).__dict__
-
+        self.__dict__ = DiscRates.from_excel(*args, **kwargs).__dict__
 
     def write_excel(self, file_name, var_names=None):
         """
@@ -341,3 +342,68 @@ def write_excel(self, file_name, var_names=None):
             disc_ws.write(i_yr, 0, disc_yr)
             disc_ws.write(i_yr, 1, disc_rt)
         disc_wb.close()
+
+    @classmethod
+    def from_csv(
+        cls, file_name, year_column="year", disc_column="discount_rate", **kwargs
+    ):
+        """
+        Read DiscRate from a csv file following template and store variables.
+
+        Parameters
+        ----------
+        file_name: str
+            filename including path and extension
+        year_column: str, optional
+            name of the column that contains the years,
+            Default: "year"
+        disc_column: str, optional
+            name of the column that contains the discount rates,
+            Default: "discount_rate"
+        **kwargs:
+            any additional arguments, e.g., `sep`, `delimiter`, `head`,
+            are forwarded to ``pandas.read_csv``
+
+        Returns
+        -------
+        climada.entity.DiscRates :
+            The disc rates from the csv file
+        """
+        dfr = pd.read_csv(file_name, **kwargs)
+        try:
+            years = dfr[year_column].values.astype(int, copy=False)
+            rates = dfr[disc_column].values
+        except KeyError as err:
+            raise ValueError(
+                f"missing column in csv file ({year_column} or {disc_column})"
+            ) from err
+
+        return cls(years=years, rates=rates)
+
+    def write_csv(
+        self, file_name, year_column="year", disc_column="discount_rate", **kwargs
+    ):
+        """
+        Write DiscRate to a csv file following template and store variables.
+
+        Parameters
+        ----------
+        file_name: str
+            filename including path and extension
+        year_column: str, optional
+            name of the column that contains the years,
+            Default: "year"
+        disc_column: str, optional
+            name of the column that contains the discount rates,
+            Default: "discount_rate"
+        **kwargs:
+            any additional arguments, e.g., `sep`, `delimiter`, `head`,
+            are forwarded to ``pandas.read_csv``
+        """
+        dfr = pd.DataFrame(
+            {
+                year_column: self.years,
+                disc_column: self.rates,
+            }
+        )
+        dfr.to_csv(file_name, **kwargs)

climada/entity/disc_rates/test/test_base.py

@@ -21,6 +21,8 @@
 import unittest
 import numpy as np
 import copy
+from pathlib import Path
+from tempfile import TemporaryDirectory
 
 from climada import CONFIG
 from climada.entity.disc_rates.base import DiscRates
@@ -216,23 +218,46 @@ def test_demo_file_pass(self):
         self.assertEqual(disc_rate.rates.max(), 0.02)
 
 
-class TestWriter(unittest.TestCase):
-    """Test excel reader for discount rates"""
+class TestWriteRead(unittest.TestCase):
+    """Test file write read cycle for discount rates"""
+
+    @classmethod
+    def setUpClass(cls):
+        cls._td = TemporaryDirectory()
+        cls.tempdir = Path(cls._td.name)
+
+    @classmethod
+    def tearDownClass(cls):
+        cls._td.cleanup()
 
-    def test_write_read_pass(self):
+    def test_write_read_excel_pass(self):
         """Read demo excel file."""
         years = np.arange(1950, 2150)
         rates = np.ones(years.size) * 0.03
         disc_rate = DiscRates(years=years, rates=rates)
 
-        file_name = CONFIG.disc_rates.test_data.dir().joinpath('test_disc.xlsx')
+        file_name = self.tempdir.joinpath('test_disc.xlsx')
         disc_rate.write_excel(file_name)
 
         disc_read = DiscRates.from_excel(file_name)
 
         self.assertTrue(np.array_equal(disc_read.years, disc_rate.years))
         self.assertTrue(np.array_equal(disc_read.rates, disc_rate.rates))
 
+    def test_write_read_csv_pass(self):
+        """Write and read csv file."""
+        years = np.arange(1950, 2150)
+        rates = np.ones(years.size) * 0.03
+        disc_rate = DiscRates(years=years, rates=rates)
+
+        file_name = self.tempdir.joinpath('test_disc.csv')
+        disc_rate.write_csv(file_name)
+
+        disc_read = DiscRates.from_csv(file_name)
+
+        self.assertTrue(np.array_equal(disc_read.years, disc_rate.years))
+        self.assertTrue(np.array_equal(disc_read.rates, disc_rate.rates))
+
 
 # Execute Tests
 if __name__ == "__main__":
@@ -243,5 +268,5 @@ def test_write_read_pass(self):
     TESTS.addTests(unittest.TestLoader().loadTestsFromTestCase(TestNetPresValue))
     TESTS.addTests(unittest.TestLoader().loadTestsFromTestCase(TestReaderExcel))
     TESTS.addTests(unittest.TestLoader().loadTestsFromTestCase(TestReaderMat))
-    TESTS.addTests(unittest.TestLoader().loadTestsFromTestCase(TestWriter))
+    TESTS.addTests(unittest.TestLoader().loadTestsFromTestCase(TestWriteRead))
     unittest.TextTestRunner(verbosity=2).run(TESTS)

climada/entity/measures/test/test_base.py

@@ -33,13 +33,13 @@
 from climada.entity.measures.measure_set import MeasureSet
 from climada.entity.measures.base import Measure, IMPF_ID_FACT
 from climada.util.constants import EXP_DEMO_H5, HAZ_DEMO_H5
+from climada.test import get_test_file
 import climada.util.coordinates as u_coord
-import climada.hazard.test as hazard_test
 import climada.entity.exposures.test as exposures_test
 
 DATA_DIR = CONFIG.measures.test_data.dir()
 
-HAZ_TEST_MAT = Path(hazard_test.__file__).parent / 'data' / 'atl_prob_no_name.mat'
+HAZ_TEST_MAT = get_test_file('atl_prob_no_name', file_format='matlab')
 ENT_TEST_MAT = Path(exposures_test.__file__).parent / 'data' / 'demo_today.mat'
 
 class TestApply(unittest.TestCase):

climada/hazard/base.py

@@ -463,12 +463,13 @@ def from_xarray_raster(
     ):
         """Read raster-like data from an xarray Dataset
 
-        This method reads data that can be interpreted using three coordinates for event,
-        latitude, and longitude. The data and the coordinates themselves may be organized
-        in arbitrary dimensions in the Dataset (e.g. three dimensions 'year', 'month',
-        'day' for the coordinate 'event'). The three coordinates to be read can be
-        specified via the ``coordinate_vars`` parameter. See Notes and Examples if you
-        want to load single-event data that does not contain an event dimension.
+        This method reads data that can be interpreted using three coordinates: event,
+        latitude, and longitude. The names of the coordinates to be read from the
+        dataset can be specified via the ``coordinate_vars`` parameter. The data and the
+        coordinates themselves may be organized in arbitrary dimensions (e.g. two
+        dimensions 'year' and 'altitude' for the coordinate 'event').  See Notes and
+        Examples if you want to load single-event data that does not contain an event
+        dimension.
 
         The only required data is the intensity. For all other data, this method can
         supply sensible default values. By default, this method will try to find these
@@ -513,12 +514,14 @@ def from_xarray_raster(
 
             Default values are:
 
-            * ``date``: The ``event`` coordinate interpreted as date
+            * ``date``: The ``event`` coordinate interpreted as date or ordinal, or
+              ones if that fails (which will issue a warning).
             * ``fraction``: ``None``, which results in a value of 1.0 everywhere, see
               :py:meth:`Hazard.__init__` for details.
             * ``hazard_type``: Empty string
             * ``frequency``: 1.0 for every event
-            * ``event_name``: String representation of the event time
+            * ``event_name``: String representation of the event date or empty strings
+              if that fails (which will issue a warning).
             * ``event_id``: Consecutive integers starting at 1 and increasing with time
         crs : str, optional
             Identifier for the coordinate reference system of the coordinates. Defaults
@@ -553,13 +556,16 @@ def from_xarray_raster(
           and Examples) before loading the Dataset as Hazard.
         * Single-valued data for variables ``frequency``. ``event_name``, and
           ``event_date`` will be broadcast to every event.
+        * The ``event`` coordinate may take arbitrary values. In case these values
+          cannot be interpreted as dates or date ordinals, the default values for
+          ``Hazard.date`` and ``Hazard.event_name`` are used, see the
+          ``data_vars``` parameter documentation above.
         * To avoid confusion in the call signature, several parameters are keyword-only
           arguments.
         * The attributes ``Hazard.haz_type`` and ``Hazard.unit`` currently cannot be
           read from the Dataset. Use the method parameters to set these attributes.
         * This method does not read coordinate system metadata. Use the ``crs`` parameter
           to set a custom coordinate system identifier.
-        * This method **does not** read lazily. Single data arrays must fit into memory.
 
         Examples
         --------
@@ -802,14 +808,48 @@ def strict_positive_int_accessor(array: xr.DataArray) -> np.ndarray:
                 raise ValueError(f"'{array.name}' data must be larger than zero")
             return array.values
 
-        def date_to_ordinal_accessor(array: xr.DataArray) -> np.ndarray:
+        def date_to_ordinal_accessor(
+            array: xr.DataArray, strict: bool = True
+        ) -> np.ndarray:
             """Take a DataArray and transform it into ordinals"""
-            if np.issubdtype(array.dtype, np.integer):
-                # Assume that data is ordinals
-                return strict_positive_int_accessor(array)
+            try:
+                if np.issubdtype(array.dtype, np.integer):
+                    # Assume that data is ordinals
+                    return strict_positive_int_accessor(array)
+
+                # Try transforming to ordinals
+                return np.array(u_dt.datetime64_to_ordinal(array.values))
+
+            # Handle access errors
+            except (ValueError, TypeError) as err:
+                if strict:
+                    raise err
+
+                LOGGER.warning(
+                    "Failed to read values of '%s' as dates or ordinals. Hazard.date "
+                    "will be ones only",
+                    array.name,
+                )
+                return np.ones(array.shape)
+
+        def year_month_day_accessor(
+            array: xr.DataArray, strict: bool = True
+        ) -> np.ndarray:
+            """Take an array and return am array of YYYY-MM-DD strings"""
+            try:
+                return array.dt.strftime("%Y-%m-%d").values
+
+            # Handle access errors
+            except (ValueError, TypeError) as err:
+                if strict:
+                    raise err
 
-            # Try transforming to ordinals
-            return np.array(u_dt.datetime64_to_ordinal(array.values))
+                LOGGER.warning(
+                    "Failed to read values of '%s' as dates. Hazard.event_name will be "
+                    "empty strings",
+                    array.name,
+                )
+                return np.full(array.shape, "")
 
         def maybe_repeat(values: np.ndarray, times: int) -> np.ndarray:
             """Return the array or repeat a single-valued array
@@ -840,8 +880,12 @@ def maybe_repeat(values: np.ndarray, times: int) -> np.ndarray:
                     None,
                     np.ones(num_events),
                     np.array(range(num_events), dtype=int) + 1,
-                    list(data[coords["event"]].values),
-                    np.array(u_dt.datetime64_to_ordinal(data[coords["event"]].values)),
+                    list(
+                        year_month_day_accessor(
+                            data[coords["event"]], strict=False
+                        ).flat
+                    ),
+                    date_to_ordinal_accessor(data[coords["event"]], strict=False),
                 ],
                 # The accessor for the data in the Dataset
                 accessor=[