Merge branch 'develop' into feature/review_api_tutorial

Author: emanuel-schmid

.github/pull_request_template.md

@@ -19,10 +19,10 @@ This PR fixes #
 ### PR Reviewer Checklist
 
 - [ ] Read the [Contribution Guide][contrib]
-- [ ] [CLIMADA Reviewer Checklist](https://climada-python.readthedocs.io/en/latest/guide/Guide_Reviewer_Checklist.html) passed
+- [ ] [CLIMADA Reviewer Checklist](https://climada-python.readthedocs.io/en/latest/development/Guide_Review.html) passed
 - [ ] [Tests][testing] passing
 - [ ] No new [linter issues][linter]
 
 [contrib]: https://github.com/CLIMADA-project/climada_python/blob/main/CONTRIBUTING.md
-[testing]: https://climada-python.readthedocs.io/en/latest/guide/Guide_Continuous_Integration_and_Testing.html
-[linter]: https://climada-python.readthedocs.io/en/stable/guide/Guide_Continuous_Integration_and_Testing.html#3.C.--Static-Code-Analysis
+[testing]: https://climada-python.readthedocs.io/en/latest/development/Guide_continuous_integration_GitHub_actions.html
+[linter]: https://climada-python.readthedocs.io/en/latest/development/Guide_continuous_integration_GitHub_actions.html#static-code-analysis

.github/workflows/ci.yml

@@ -47,7 +47,7 @@ jobs:
       -
         name: Install CLIMADA
         run: |
-          python -m pip install ".[test]"
+          python -m pip install "./[dev]"
       -
         name: Run Unit Tests
         run: |

.zenodo.json

@@ -119,6 +119,9 @@
     },
     {
       "name": "Valentin Gebhart"
+    },
+    {
+      "name": "Dahyann Araya"
     }
   ]
 }

AUTHORS.md

@@ -36,3 +36,4 @@
 * Luca Severino
 * Samuel Juhel
 * Valentin Gebhart
+* Dahyann Araya

CHANGELOG.md

@@ -10,24 +10,41 @@ Code freeze date: YYYY-MM-DD
 
 ### Dependency Changes
 
+Added:
+
+- `fiona` >=1.10
+
+Updated:
+
+- `geopandas` >=0.14,<1.0 → >=0.14
+- `pandas` >=2.1,<2.2 → >=2.1
+
 Removed:
 
 - `pandas-datareader`
 
 ### Added
+
 - Added optional parameter to `geo_im_from_array`, `plot_from_gdf`, `plot_rp_imp`, `plot_rp_intensity`,
 `plot_intensity`, `plot_fraction`, `_event_plot` to mask plotting when regions are too far from data points [#1047](https://github.com/CLIMADA-project/climada_python/pull/1047). To recreate previous plots (no masking), the parameter can be set to None.
 - Added instructions to install Climada petals on Euler cluster in `doc.guide.Guide_Euler.ipynb` [#1029](https://github.com/CLIMADA-project/climada_python/pull/1029)
-
+- Added util methods to handle crs coordinates consistently: `is_geo_coords`, `check_if_geo_coords`, `get_crs_unit`, `estimate_matching_threshold`, `degree_to_km`, and `km_to_degree` [#1080](https://github.com/CLIMADA-project/climada_python/pull/1080)
 - `ImpactFunc` and `ImpactFuncSet` now support equality comparisons via `==` [#1027](https://github.com/CLIMADA-project/climada_python/pull/1027)
 
 ### Changed
+- Changed default distance threshold for nearest neighbor matching in `util.coordinates.match_coordinates` from a fixed value of 100km to twice the highest resolution of the coords_to_assign [#1080](https://github.com/CLIMADA-project/climada_python/pull/1080).
 - Changed the default mask_distance in `util.plot.geo_im_from_array` to 0.03 to avoid white gaps in gridded hazard data with comparably low resolution (>80 centroids per axis) [#1073](https://github.com/CLIMADA-project/climada_python/pull/1073)
 - Increased speed of `util.plot.add_shapes` by avoiding for loops, substantially speeding up `Hazard.plot_intensity` and other functions. [#1073](https://github.com/CLIMADA-project/climada_python/pull/1073)
+- Update `util.coordinates.match_centroids`, `util.coordinates.match_coordinates`, so that they also
+accept coordinates that are not defined in degree. [#1080](https://github.com/CLIMADA-project/climada_python/pull/1080)
+- Implement cheap test to check that input coordinates at least seem geographic for functions that require
+geographic coordinates as input (e.g. `util.coordinates.dist_to_coast`, `util.coordinates.coord_on_land`, `util.coordinates.lon_normalize`, `util.coordinates.lon_bounds`). [#1080](https://github.com/CLIMADA-project/climada_python/pull/1080)
 - `Hazard.local_exceedance_intensity`, `Hazard.local_return_period` and `Impact.local_exceedance_impact`, `Impact.local_return_period`, using the `climada.util.interpolation` module: New default (no binning), binning on decimals, and faster implementation [#1012](https://github.com/CLIMADA-project/climada_python/pull/1012)
 - World Bank indicator data is now downloaded directly from their API via the function `download_world_bank_indicator`, instead of relying on the `pandas-datareader` package [#1033](https://github.com/CLIMADA-project/climada_python/pull/1033)
 - `Exposures.write_hdf5` pickles geometry data in WKB format, which is faster and more sustainable. [#1051](https://github.com/CLIMADA-project/climada_python/pull/1051)
 - The online documentation has been completely overhauled, now uses PyData theme: [#977](https://github.com/CLIMADA-project/climada_python/pull/977)
+- Add `climada.hazard.xarray` module with helper structures for reading Hazard objects from `xarray` data [#1063](https://github.com/CLIMADA-project/climada_python/pull/1063)
+- The output of the `impact_yearset` was changed to only contain attributes corresponding to the yearly impact set. The application of the correction factor and the frequency of the resulting yearly impact object are corrected. [#1075](https://github.com/CLIMADA-project/climada_python/pull/1075)
 
 ### Fixed
 
@@ -36,9 +53,13 @@ Removed:
 
 ### Deprecated
 
+- `Hazard.from_xarray_raster_file`. Use `Hazard.from_xarray_raster` and pass the file path as `data` argument [#1063](https://github.com/CLIMADA-project/climada_python/pull/1063)
+
 ### Removed
 
 - `climada.util.interpolation.round_to_sig_digits` [#1012](https://github.com/CLIMADA-project/climada_python/pull/1012)
+- `intensity_thres` of `Hazard`, `StormEurope`, and `TropCyclones` object [#1065](https://github.com/CLIMADA-project/climada_python/pull/1065)
+- Deprecated method `climada.hazard.trop_cyclone.trop_cyclone.TropCyclone.set_from_tracks` [#1065](https://github.com/CLIMADA-project/climada_python/pull/1065)
 
 ## 6.0.1
 
@@ -89,8 +110,12 @@ Removed:
 
 ### Added
 
+- `climada.entity.impact_funcs.trop_cyclone.ImpfSetTropCyclone.get_impf_id_regions_per_countries` function [#1034](https://github.com/CLIMADA-project/climada_python/pull/1034)
+- `climada.hazard.tc_tracks.BasinBoundsStorm` Enum class `climada.hazard.tc_tracks.subset_by_basin` function [#1031](https://github.com/CLIMADA-project/climada_python/pull/1031)
 - `climada.hazard.tc_tracks.TCTracks.subset_years` function [#1023](https://github.com/CLIMADA-project/climada_python/pull/1023)
-- `climada.hazard.tc_tracks.TCTracks.from_FAST` function, add Australia basin (AU) [#993](https://github.com/CLIMADA-project/climada_python/pull/993)
+-`climada.hazard.tc_tracks.compute_track_density` function, `climada.hazard.tc_tracks.compute_genesis_density` function, `climada.hazard.plot.plot_track_density` function
+ [#1003](https://github.com/CLIMADA-project/climada_python/pull/1003)
+-`climada.hazard.tc_tracks.TCTracks.from_FAST` function, add Australia basin (AU) [#993](https://github.com/CLIMADA-project/climada_python/pull/993)
 - Add `osm-flex` package to CLIMADA core [#981](https://github.com/CLIMADA-project/climada_python/pull/981)
 - `doc.tutorial.climada_entity_Exposures_osm.ipynb` tutorial explaining how to use `osm-flex` with CLIMADA
 - `climada.util.coordinates.bounding_box_global` function [#980](https://github.com/CLIMADA-project/climada_python/pull/980)
@@ -208,6 +233,7 @@ CLIMADA tutorials. [#872](https://github.com/CLIMADA-project/climada_python/pull
 - `Impact.write_hdf5` now throws an error if `event_name` is does not contain strings exclusively [#894](https://github.com/CLIMADA-project/climada_python/pull/894)
 - Split `climada.hazard.trop_cyclone` module into smaller submodules without affecting module usage [#911](https://github.com/CLIMADA-project/climada_python/pull/911)
 - `yearly_steps` parameter of `TropCyclone.apply_climate_scenario_knu` has been made explicit [#991](https://github.com/CLIMADA-project/climada_python/pull/991)
+- `Hazard.write_hdf5` writes centroids as x,y columns (or as wkb in case of polygons) at a compression level of 9, not as pickled `Shapely` objects anymore, which reduces the size of the files significantly.
 
 ### Fixed
 

climada/engine/impact_calc.py

@@ -92,6 +92,12 @@ def impact(
             Default: True
         assign_centroids : bool, optional
             indicates whether centroids are assigned to the self.exposures object.
+            The default centroids assignment uses Euclidean distance and a threshold equal to
+            twice the highest resolution (smallest distance between any two nearest neighbours)
+            of the centroids (assuming a regular grid).
+            Recommendation: assign the centroids directly with control
+            over the distance metric and the distance threshold using
+            `exposure.assign_centroids(hazard)`. In this case, set assign_centroids to ``False``.
             Centroids assignment is an expensive operation; set this to ``False`` to save
             computation time if the hazards' centroids are already assigned to the exposures
             object.
@@ -114,10 +120,20 @@ def impact(
             >>> imp = impcalc.impact(insured=True)
             >>> imp.aai_agg
 
+            >>> haz = Hazard.from_hdf5(HAZ_DEMO_H5)  # Set hazard
+            >>> impfset = ImpactFuncSet.from_excel(ENT_TEMPLATE_XLS)
+            >>> exp = Exposures(pd.read_excel(ENT_TEMPLATE_XLS))
+            >>> #Adjust threshold and distance to centroids/exposure resolution
+            >>> exp.assign_centroids(hazard, threshold=1.5, distance='euclidean')
+            >>> impcalc = ImpactCal(exp, impfset, haz)
+            >>> imp = impcalc.impact(insured=True, assign_centroids=False)
+            >>> imp.aai_agg
+
         See also
         --------
         apply_deductible_to_mat : apply deductible to impact matrix
         apply_cover_to_mat : apply cover to impact matrix
+        climada.entity.exposures.assign_centroids : assign centroids to exposures explicitly
         """
         # TODO: consider refactoring, making use of Exposures.hazard_impf
         # check for compatibility of exposures and hazard type

climada/engine/impact_data.py

@@ -774,7 +774,7 @@ def clean_emdat_df(
         df_data["Start Day"] = np.array(day_list, dtype="int")
         df_data["Start Year"] = np.array(year_list, dtype="int")
         for var in ["Disaster Subtype", "Disaster Type", "Country"]:
-            df_data[VARNAMES_EMDAT[target_version][var]].fillna("None", inplace=True)
+            df_data.fillna({VARNAMES_EMDAT[target_version][var]: "None"}, inplace=True)
 
     # (3) Filter by countries, year range, and disaster type
     # (3.1) Countries:

climada/engine/test/test_impact_calc.py

@@ -210,6 +210,28 @@ def test_calc_impact_TC_pass(self):
         self.assertAlmostEqual(6.570532945599105e11, impact.tot_value)
         self.assertAlmostEqual(6.512201157564421e09 * x, impact.aai_agg, 5)
 
+    def test_calc_impact_TC_change_crs_pass(self):
+        """Test compute impact"""
+        crs = "EPSG:4087"
+        exp = ENT.exposures.copy()
+        exp.to_crs(crs)
+        haz = deepcopy(HAZ)
+        haz.centroids.to_crs(crs)
+        icalc = ImpactCalc(exp, ENT.impact_funcs, haz)
+        impact = icalc.impact()
+        self.assertEqual(icalc.n_events, len(impact.at_event))
+        self.assertEqual(0, impact.at_event[0])
+        self.assertEqual(0, impact.at_event[7225])
+        self.assertAlmostEqual(1.472482938320243e08, impact.at_event[13809], delta=1)
+        self.assertAlmostEqual(7.076504723057620e10, impact.at_event[12147], delta=1)
+        self.assertEqual(0, impact.at_event[14449])
+        self.assertEqual(icalc.n_exp_pnt, len(impact.eai_exp))
+        self.assertAlmostEqual(1.518553670803242e08, impact.eai_exp[0], delta=1)
+        self.assertAlmostEqual(1.373490457046383e08, impact.eai_exp[25], 6)
+        self.assertAlmostEqual(1.066837260150042e08, impact.eai_exp[49], 6)
+        self.assertAlmostEqual(6.570532945599105e11, impact.tot_value)
+        self.assertAlmostEqual(6.512201157564421e09, impact.aai_agg, 5)
+
     def test_calc_impact_RF_pass(self):
         haz = Hazard.from_hdf5(get_test_file("test_hazard_US_flood_random_locations"))
         exp = Exposures.from_hdf5(
@@ -574,10 +596,13 @@ def test_single_exp_zero_mdr(self):
         eai_exp = np.array([aai_agg])
         at_event = np.array([imp_evt, 0])
         exp.set_geometry_points()
+        exp.assign_centroids(haz, threshold=1)
         impf_tc = ImpfTropCyclone.from_emanuel_usa()
         impf_set = ImpactFuncSet([impf_tc])
         impf_set.check()
-        imp = ImpactCalc(exp, impf_set, haz).impact(save_mat=True)
+        imp = ImpactCalc(exp, impf_set, haz).impact(
+            assign_centroids=False, save_mat=True
+        )
         check_impact(self, imp, haz, exp, aai_agg, eai_exp, at_event, at_event)
 
 

climada/engine/unsequa/calc_cost_benefit.py

@@ -398,7 +398,7 @@ def _map_costben_calc(
             ent_future=ent_fut,
             save_imp=False,
             assign_centroids=False,
-            **cost_benefit_kwargs
+            **cost_benefit_kwargs,
         )
         # Extract from climada.impact the chosen metrics
         uncertainty_values.append(

climada/engine/unsequa/test/test_unsequa.py

@@ -650,7 +650,7 @@ def test_calc_sensitivity_all_pass(self):
         }
 
         def test_sensitivity_method(
-            exp_unc, impf_unc, haz_unc, sensitivity_method, param_dict
+            exp_unc, impf_unc, haz_unc, sensitivity_method, param_dict, places
         ):
             """Function to test each seaprate sensitivity method"""
             unc_calc = CalcImpact(exp_unc, impf_unc, haz_unc)
@@ -681,7 +681,7 @@ def test_sensitivity_method(
             self.assertAlmostEqual(
                 param_dict["test_si_value"][0],
                 unc_data.aai_agg_sens_df["aai_agg"][param_dict["test_si_value"][1]],
-                places=5,
+                places=places,
             )
 
             self.assertEqual(unc_data.aai_agg_unc_df.size, unc_data.n_samples)
@@ -695,7 +695,12 @@ def test_sensitivity_method(
         # loop over each method and do test
         for sensitivity_method, method_params in test_dict.items():
             test_sensitivity_method(
-                exp_unc, impf_unc, haz_unc, sensitivity_method, method_params
+                exp_unc,
+                impf_unc,
+                haz_unc,
+                sensitivity_method,
+                method_params,
+                places=2 if sensitivity_method == "rbd_fast" else 5,
             )