Merge pull request #26 from alliander-opensource/feature/pre-release-improvements

Feature/pre release improvements

Author: JelmarVersleijen

CONTRIBUTING.md

@@ -13,10 +13,11 @@ We'd love to accept your patches and contributions to this project. There are ju
 Contribution does not necessarily mean committing code to the repository.
 We recognize different levels of contributions as shown below in increasing order of dedication.
 
-1. Use and test the project. Give feedback on the user experience or suggest new features.
-2. Report bugs or security vulnerabilities.
-3. Fix bugs.
-4. Improve the project by developing new features.
+1. Use and test the project. Give feedback on the user experience.
+2. Suggest or prioritize new features to be developed, based on substantiated input from a DSO.
+3. Report bugs or security vulnerabilities.
+4. Fix bugs.
+5. Improve the project by developing new features.
 
 ## Filing bugs, security vulnerabilities or feature requests
 
@@ -79,12 +80,12 @@ Contributions should be submitted as GitHub pull requests. See [Creating a pull
 Follow this process for a code change and pull request:
 
 1. Fork the repository.
-1. Make your change in a feature/description_of_your_change branch.
-1. Run the tests.
-1. Run the [pre-commit hooks](https://pre-commit.com/) to check for code style and other issues.
-1. Submit a pull request.
-1. Pull requests will be reviewed by one of the maintainers who may discuss, offer constructive feedback, request changes, or approve the work.
-1. Upon receiving the sign-off from one of the maintainers will merge it for you.
+2. Make your change in a feature/description_of_your_change branch.
+3. Run the tests.
+4. Run the [pre-commit hooks](https://pre-commit.com/) to check for code style and other issues.
+5. Submit a pull request.
+6. Pull requests will be reviewed by one of the maintainers who may discuss, offer constructive feedback, request changes, or approve the work.
+7. Upon receiving the sign-off from one of the maintainers will merge it for you.
 
 ## Attribution
 

README.md

@@ -8,7 +8,7 @@ SPDX-License-Identifier: Apache-2.0
 
 This repository shares research on software for automatic placement of electricity cables using a combination of geo-information and graph theory.
 
-The utility network needs to be expanded due to the energy transition. Finding a location for new infrastructure is no easy feat considering the amount of involved design criteria. 
+The utility network needs to be expanded due to the energy transition. Finding a location for new infrastructure is no easy feat considering the amount of involved design criteria.
 This research includes the creation of a software package for automatic placement of utility network using a combination of geo-information and graph theory.
 
 This research is being carried out at Alliander, a Dutch DSO, as part of [Jelmar Versleijen](https://research.wur.nl/en/persons/jelmar-versleijen)'s PhD with [Wagening University](https://www.wur.nl/en.htm). [Read more about research at Alliander](https://www.alliander.com/nl/alliander-en-open-research/).
@@ -57,12 +57,14 @@ Run tests using pytest:
 ```bash
 poetry run python -m pytest tests/
 ```
+For quick experimentation, weights of existing criteria can be changed. This can be done by editing the `mcda_presets.py` file.
 
-Expanding criteria included in the `mcda_presets.py` can be done by:
+Adding new criteria can be done by:
 
-1. Adding a new class to the `criteria` folder.
-2. Implementing the `get_suitability` method.
-3. Adding the new class to the `mcda_presets.py` file. Set the group and weight of the new class.
+1. Add vector data to the case study geopackage in the `data/examples` folder.
+2. Adding a new entry to `mcda_presets.py` according to the pydantic model `RasterPresetCriteria`.
+3. Adding a new class to the `utility_route_planner/mcda/vector_preprocessing` folder.
+4. Implementing the `specific_preprocess` method. This commonly consists out reclassifying a dominant attribute.
 
 # Support
 
@@ -96,4 +98,4 @@ The software is largely dependent on data. Data is incorporated in the example f
 
 Citing
 -------
-t.b.d.
+t.b.d.

main.py

@@ -50,7 +50,11 @@ def run_mcda_lcpa(
 
     logger.info(f"Route CPU time: {(time.process_time_ns() - start_cpu_time) / 1e9:.2f} seconds.")
     route_evaluation_metrics = RouteEvaluationMetrics(
-        lcpa_engine.lcpa_result, path_suitability_raster, human_designed_route, project_area_geometry
+        lcpa_engine.lcpa_result,
+        path_suitability_raster,
+        human_designed_route,
+        project_area_geometry,
+        mcda_engine.processed_vector_metrics,
     )
     route_evaluation_metrics.get_route_evaluation_metrics()
 

tests/integration/mcda/mcda_vector_base_test.py

@@ -2,12 +2,14 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-from unittest.mock import Mock
+from unittest.mock import Mock, MagicMock
 
 import geopandas as gpd
 import numpy as np
+import pandas as pd
 
 import pytest
+import shapely
 
 from settings import Config
 from utility_route_planner.models.mcda.exceptions import InvalidSuitabilityValue, UnassignedValueFoundDuringReclassify
@@ -111,3 +113,100 @@ def test_missing_values(input_which_should_raise):
     wrong_input = input_which_should_raise
     with pytest.raises(UnassignedValueFoundDuringReclassify):
         validate_values_to_reclassify(values_to_reclassify, wrong_input)
+
+
+class DummyPreprocessor(VectorPreprocessorBase):
+    criterion = "dummy_criterion"
+
+    def specific_preprocess(self, prepared_data, criterion):
+        pass
+
+
+class TestGetMetrics:
+    @pytest.fixture
+    def get_dummy_criterion(self):
+        return MagicMock(spec=RasterPresetCriteria)
+
+    def test_get_statistics_invalid_column(self, get_dummy_criterion):
+        dummy_criterion = get_dummy_criterion
+        dummy_criterion.columns_to_reclassify = ["missing_col"]
+        dummy_criterion.layer_names = ["layer1"]
+
+        gdf = gpd.GeoDataFrame({"geometry": [None], "col1": ["A"], "suitability_value": [1]})
+        preprocessor = DummyPreprocessor()
+        with pytest.raises(ValueError):
+            preprocessor.get_statistics(get_dummy_criterion, gdf)
+
+    def test_get_statistics_no_reclassify(self, get_dummy_criterion):
+        mock_criterion = get_dummy_criterion
+        mock_criterion.columns_to_reclassify = []
+        mock_criterion.layer_names = ["layer1"]
+        gdf = gpd.GeoDataFrame({"geometry": [], "suitability_value": []})
+
+        preprocessor = DummyPreprocessor()
+        result = preprocessor.get_statistics(mock_criterion, gdf)
+        assert "criterion" in result.columns
+        assert "weight_key" in result.columns
+        assert "area_m2" in result.columns
+        assert "suitability_value" in result.columns
+
+    def test_get_statistics_one_reclassify_column(self, get_dummy_criterion):
+        dummy_criterion = get_dummy_criterion
+        dummy_criterion.columns_to_reclassify = ["col1"]
+        dummy_criterion.layer_names = ["layer1"]
+        gdf = gpd.GeoDataFrame(
+            {
+                "geometry": [shapely.Point(0, 0).buffer(10), shapely.Point(0, 0).buffer(10)],
+                "col1": ["A", "B"],
+                "suitability_value": [1, 2],
+            }
+        )
+        preprocessor = DummyPreprocessor()
+        result = preprocessor.get_statistics(dummy_criterion, gdf)
+        expected_df = pd.DataFrame(
+            {
+                "criterion": ["dummy_criterion", "dummy_criterion"],
+                "weight_key": ["A", "B"],
+                "suitability_value": [1, 2],
+                "area_m2": [313.6548490545941, 313.6548490545941],
+            }
+        )
+        pd.testing.assert_frame_equal(result.reset_index(drop=True), expected_df, check_dtype=False)
+
+    def test_get_statistics_two_reclassify_columns(self, get_dummy_criterion):
+        dummy_criterion = get_dummy_criterion
+        get_dummy_criterion.columns_to_reclassify = ["col1", "col2"]
+        get_dummy_criterion.layer_names = ["layer1"]
+        example_polygon1 = shapely.Polygon([(0, 0), (0, 10), (10, 10), (10, 0), (0, 0)])
+        example_polygon2 = shapely.Polygon([(5, 5), (5, 15), (15, 15), (15, 5), (5, 5)])
+        gdf = gpd.GeoDataFrame(
+            {
+                "geometry": [example_polygon1, example_polygon1, example_polygon2],
+                "col1": ["A", "B", "B"],
+                "col2": ["X", "Y", "Y"],
+                "suitability_value": [1, 2, 2],
+            }
+        )
+        preprocessor = DummyPreprocessor()
+        result = preprocessor.get_statistics(dummy_criterion, gdf)
+
+        expected_df = pd.DataFrame(
+            {
+                "criterion": ["dummy_criterion", "dummy_criterion"],
+                "weight_key": ["A: X", "B: Y"],
+                "suitability_value": [1, 2],
+                "area_m2": [100, 175.0],
+            }
+        )
+        pd.testing.assert_frame_equal(result.reset_index(drop=True), expected_df, check_dtype=False)
+
+    def test_get_statistics_three_reclassify_columns_raises(self, get_dummy_criterion):
+        dummy_criterion = get_dummy_criterion
+        dummy_criterion.columns_to_reclassify = ["col1", "col2", "col3"]
+        dummy_criterion.layer_names = ["layer1"]
+        gdf = gpd.GeoDataFrame(
+            {"geometry": [None], "col1": ["A"], "col2": ["B"], "col3": ["C"], "suitability_value": [1]}
+        )
+        preprocessor = DummyPreprocessor()
+        with pytest.raises(ValueError):
+            preprocessor.get_statistics(dummy_criterion, gdf)

tests/integration/mcda/mcda_vector_test.py

@@ -935,6 +935,118 @@ def test_small_above_ground_obstacles(self):
         assert "niet-bgt" not in reclassified_gdf["bgt-type"]
         assert "waardeOnbekend" not in reclassified_gdf["plus-type"]
 
+        reclassified_gdfscheiding_only = SmallAboveGroundObstacles._set_suitability_values(
+            [bgt_bak_bord_kast_paal_put_straatmeubilair], weight_values
+        )
+        pd.testing.assert_series_equal(
+            reclassified_gdfscheiding_only["suitability_value"],
+            pd.Series(
+                [
+                    8,
+                    9,
+                    10,
+                    11,
+                    12,
+                    13,
+                    14,
+                    15,
+                    16,
+                    17,
+                    18,
+                    19,
+                    20,
+                    21,
+                    22,
+                    23,
+                    24,
+                    25,
+                    26,
+                    27,
+                    28,
+                    29,
+                    30,
+                    31,
+                    32,
+                    33,
+                    34,
+                    35,
+                    36,
+                    37,
+                    38,
+                    39,
+                    40,
+                    41,
+                    42,
+                    43,
+                    44,
+                    44,
+                    45,
+                    46,
+                    47,
+                    48,
+                    49,
+                    50,
+                    51,
+                    52,
+                    53,
+                    54,
+                    55,
+                    56,
+                    57,
+                    58,
+                    59,
+                    59,
+                    60,
+                    61,
+                    62,
+                    63,
+                    64,
+                    65,
+                    66,
+                    67,
+                    68,
+                    69,
+                    70,
+                    71,
+                    72,
+                    72,
+                    73,
+                    74,
+                    75,
+                    76,
+                    77,
+                    78,
+                    79,
+                    80,
+                    81,
+                    82,
+                    83,
+                    84,
+                    85,
+                ]
+            ),
+            check_names=False,
+            check_exact=True,
+            check_dtype=False,
+            check_index=False,
+        )
+
+        assert "waardeOnbekend" not in reclassified_gdfscheiding_only["plus-type"]
+
+        reclassified_gdf_others_only = SmallAboveGroundObstacles._set_suitability_values(
+            [bgt_scheiding_1, bgt_scheiding_2], weight_values
+        )
+        pd.testing.assert_series_equal(
+            reclassified_gdf_others_only["suitability_value"],
+            pd.Series([1, 2, 3, 4, 5, 7]),
+            check_names=False,
+            check_exact=True,
+            check_dtype=False,
+            check_index=False,
+        )
+
+        assert "niet-bgt" not in reclassified_gdf_others_only["bgt-type"]
+
     def test_vegetation_object(self):
         weight_values = {
             # plus_type
@@ -1190,3 +1302,17 @@ def test_existing_substations(self):
             check_index=False,
         )
         assert reclassified_gdf.geom_type.unique().tolist() == ["Polygon"]
+
+
+def test_get_vector_metrics_benchmark():
+    mcda_engine = McdaCostSurfaceEngine(
+        Config.RASTER_PRESET_NAME_BENCHMARK, Config.PYTEST_PATH_GEOPACKAGE_MCDA, shapely.Point(1, 1).buffer(100)
+    )
+    df1 = pd.DataFrame({"area_m2": [1.2, 2.5, 3.7]})
+    df2 = pd.DataFrame({"area_m2": [4.1, 5.9]})
+    present_weight_dfs = [df1, df2]
+    n_total_weights = mcda_engine.get_vector_metrics(present_weight_dfs)
+
+    assert n_total_weights == 195  # As per the benchmark preset configuration
+    df_expected = pd.DataFrame({"area_m2": [1, 2, 4, 4, 6]})
+    pd.testing.assert_frame_equal(df_expected.reset_index(drop=True), pd.DataFrame({"area_m2": [1, 2, 4, 4, 6]}))

tests/integration/mcda_lcpa_main_test.py

@@ -48,6 +48,7 @@ def test_mcda_lcpa_chain_pytest_files(self, utility_route_sketch):
         write_results_to_geopackage(Config.PATH_GEOPACKAGE_LCPA_OUTPUT, lcpa_engine.lcpa_result, "utility_route_result")
 
 
+@pytest.mark.skip(reason="Benchmark cases are skipped in normal test runs to save time.")
 @pytest.mark.parametrize(
     "path_geopackage, layer_name_project_area, layer_name_utility_route_human_designed",
     [

utility_route_planner/models/mcda/load_mcda_preset.py

@@ -30,9 +30,12 @@ class RasterPresetCriteria(pydantic.BaseModel):
     layer_names: list = pydantic.Field(description="Layer names in the geopackage which will be handled.")
     preprocessing_function: VectorPreprocessorBase
     group: str = pydantic.Field(description="Determines how the criteria is handled.")
+    columns_to_reclassify: typing.Optional[list] = pydantic.Field(
+        default=[], description="List of dominant attributes to reclassify."
+    )
     weight_values: dict = pydantic.Field(..., description="Contains values for defining how important the layer is.")
     geometry_values: typing.Optional[dict] = pydantic.Field(
-        default=None,
+        default={},
         description="Contains values for optional computational geometry steps, e.g., buffer.",
     )