Implementing transmission nodes through .csv files requires workaround

[anedelcu2002]

What happened?

I have been working on modeling the heating system for a neighborhood with Calliope. Because of the resolution and scale of the model (up to tens of thousands of nodes), supply and demand techs are defined in the .yaml config file, while nodes, links, and transmission techs are defined procedurally in separate .csv files. I tried to mimic the implementation for the national example in the library files, but putting the transmission nodes in the .csv without any technology or parameter value breaks the model reading operation. The workaround I settled on was applying a heat demand tech to each transmission node and giving it a value of zero. Connecting these nodes works normally.

Which operating systems have you used?

• macOS
• Windows
• Linux

Version

v0.7.0

Relevant log output

ERROR: Analysis failed with exception:
ValueError: not enough values to unpack (expected 2, got 1)

Error summary saved to: outputs\scenario_summary.json
Traceback (most recent call last):
  File "C:\Users\alexn\Documents\GitHub\SC-G5\delft_calliope\run_analysis.py", line 384, in <module>
    model, results = main(CONFIG)
                     ^^^^^^^^^^^^
  File "C:\Users\alexn\Documents\GitHub\SC-G5\delft_calliope\run_analysis.py", line 272, in main
    model = create_scenario_model(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexn\Documents\GitHub\SC-G5\delft_calliope\functions\model\create_scenario_model.py", line 256, in create_scenario_model
    model = calliope.read_yaml(temp_yaml_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexn\.conda\envs\calliope\Lib\site-packages\calliope\model.py", line 79, in read_yaml
    return read_dict(
           ^^^^^^^^^^
  File "C:\Users\alexn\.conda\envs\calliope\Lib\site-packages\calliope\model.py", line 131, in read_dict
    model_data_factory.clean()
  File "C:\Users\alexn\.conda\envs\calliope\Lib\site-packages\calliope\preprocess\model_data.py", line 127, in clean
    self.add_link_distances()
  File "C:\Users\alexn\.conda\envs\calliope\Lib\site-packages\calliope\preprocess\model_data.py", line 338, in add_link_distances
    node1, node2 = tech_def.where(tech_def).dropna("nodes").nodes.values
    ^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

[anedelcu2002]

I also tried to pass the data through the Pandas dataframes to Calliope directly rather than write to and read from .csv files, but that also failed with similar error messages (probably related to my not understanding how the nested data structure with carriers and such works in Calliope). I'd be happy to try and replicate that as well but it would require some additional effort.

[brynpickering]

@anedelcu2002 can you try following the approach taken in the national scale example model that's defined purely from data tables in the test directory. That is, your transmission links would be defined in CSV as:

techs,link_to,link_from
region1_to_region1_1,region1,region1_1
region1_to_region1_2,region1,region1_2
region1_to_region1_3,region1,region1_3
region1_to_region2,region1,region2

Tens of thousands of nodes sounds like a recipe for disaster, though, for any energy system modelling tool to be able to handle!

[anedelcu2002]

@brynpickering thanks for the quick answer! The problem is not with the techs, which I indeed define like this, but with the nodes themselves. Here is how two demand nodes and four transmission nodes are defined:

D0503100000034482,demand_LQ_heat,sink_use_equals,,344.9139014056572
D0503100000036714,demand_LQ_heat,sink_use_equals,,231.6430026628996
LQHtransmission1,demand_LQ_heat,sink_use_equals,,0.0
LQHtransmission2,demand_LQ_heat,sink_use_equals,,0.0
LQHtransmission3,demand_LQ_heat,sink_use_equals,,0.0
LQHtransmission4,demand_LQ_heat,sink_use_equals,,0.0

If I do everything the exact same way but with the transmission nodes lacking the tech or the other parameters, I get the error above. The full data table structure for an example run is available here.

RE the number of nodes: you are completely right, I'm truly testing the limits of Calliope and reaching some pretty nasty bottlenecks in model reading and building (sometimes up to 30 minutes to build a scenario!) but this also results in very high-definition neighborhood-level models based on real-life distribution grid topologies!

[brynpickering]

@anedelcu2002 I understand the issue now. Let me investigate it a bit!