Edge case clustering issues

[trevorb1]

Version Checks (indicate both or one)

• This bug exists on the master branch of PyPSA-USA.

• This bug exists on the develop branch of PyPSA-USA.

The Issue

I am getting a clustering error on the simplification step when running Wisconsin and Ohio independently at reeds_zone resolution. This is quite similar to the issue discussed in Issue #646.

Steps To Reproduce

Run this config:

# docs :
scenario:
  interconnect: [eastern] #"usa|texas|western|eastern"
  clusters: [7a] # or 7, 7m, 6, 6a, 6m
  simpl: [70]
  opts: [3h]
  ll: [v1.0]
  sector: "" # G
  planning_horizons: [2030] 
foresight: 'perfect' # myopic, perfect

model_topology:
  transmission_network: 'reeds' # [reeds, tamu]
  topological_boundaries: 'reeds_zone' # [county, reeds_zone, state]
  interface_transmission_limits: false
  include: # mixed zone types not supported
    # reeds_zone: []
    reeds_state: ['WI'] # or "OH"
    # reeds_ba: []
  aggregate: # eligible keys: [reeds_zone, trans_reg]
    # trans_grp: []
    # reeds_zone: []

Error Message

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: '1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3053, in get_loc
    return self._engine.get_loc(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1132, in __getitem__
    result = self._get_value(key)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1237, in _get_value
    loc = self.index.get_loc(label)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3055, in get_loc
    raise KeyError(key) from err
KeyError: ('p75', '1')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: '1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3220, in _get_loc_level
    return (self._engine.get_loc(key), None)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmpbb5f37nj.simplify_network.py", line 292, in <module>
    clustering = clustering_for_n_clusters(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 334, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 314, in busmap_for_n_clusters
    .apply(busmap_for_country, include_groups=False)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1819, in apply
    return self._python_apply_general(f, self._obj_with_exclusions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1885, in _python_apply_general
    values, mutated = self._grouper.apply_groupwise(f, data, self.axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 919, in apply_groupwise
    res = f(group)
          ^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 290, in busmap_for_country
    n_clusters_per_region[x.name],
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1142, in __getitem__
    return self._get_values_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1210, in _get_values_tuple
    indexer, new_index = self.index.get_loc_level(key)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3150, in get_loc_level
    loc, mi = self._get_loc_level(key, level=level)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3222, in _get_loc_level
    raise KeyError(key) from err
KeyError: ('p75', '1')

Anything else?

The best I can tell is it is an issue with the subnetworks be totally empty. For example, in WI, the subnetwork ("p75", "1") does not exist in the data structure.

When I inspect it, the sub_network, it consists of two buses (25682 and 20889 in this case)

However, these buses are only connected to each other, and have generators, but no loads attached to them.

---

My best guess is since this is an isolated sub_network with no load, something odd is happening in the busmap clustering? But I don't know the clustering details well enough off the top of my head to make a guess more insightful then that haha.

[trevorb1]

Pretty hacky solution, but to at least get it to run you can just delete the subnetwork from the network, and then re-determine the network topology. This is a band-aid solution tho, and doesnt really address why this error is happening.

        for sub_network in n.buses.sub_network.unique():
            buses = n.buses[n.buses.sub_network == sub_network]
            loads = n.loads[n.loads.bus.isin(buses.index)]
            if loads.empty:
                logger.warning(f"Removing sub_network {sub_network}")
                gens = n.generators[n.generators.bus.isin(buses.index)]
                lines = n.lines[n.lines.bus0.isin(buses.index) | n.lines.bus1.isin(buses.index)]
                n.mremove("Bus", buses.index)
                n.mremove("Generator", gens.index)
                n.mremove("Line", lines.index)
        n.determine_network_topology()

[trevorb1]

As @ktehranchi suggested, changing the weighting-strategy in the clustering.simplify_network config may resolve this issue. I was running with demand-capacity but did not try population.

pypsa-usa/workflow/repo_data/config/config.default.yaml

Lines 158 to 163 in 105edcb

	clustering:
	simplify_network:
	weighting_strategy: demand-capacity # choose from: [population, demand-capacity]
	to_substations: false # network is simplified to nodes with positive or negative power injection (i.e. substations or offwind connections)
	algorithm: kmeans # choose from: [hac, kmeans]
	feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.