Refactor to use Tables in most places

[abelsiqueira]

Currently, we have info scattered in different places:

• Input data (time-independent and profiles) as Table (assuming reading from ESDL and InfluxDB)
• Graph structure
• Time-independent data as MetaGraph (i.e., graph[object].property)
• Profiles as MetaGraph (i.e., graph[object].profiles)
• Clustered profiles as Table
• Representative periods as Table
• Representative periods as Struct
• Mapping of year-partition and representative periods at Table
• Partition definition (implicit) as Table
• Partition definition (explicit) as MetaGraph (i.e., graph[object].partitions)
• Constraints partition as Dict (computed from explicit partition definition)
• Base periods as Table (after Add long and short storage asset constraints #498 we'll know more)
• Flows variable as Table
• Constraints partitions with incoming and outgoing flows as Table
• Solution (value of variables) as Struct and MetaGraph

PS. Table means CSV/DataFrames/DB

The low-level model creation from CSVs looks like the following:

graph, rep_periods, base_periods = create_graph_and_representative_periods_from_csv_folder(dir)
cons_partitions = compute_constraints_partitions(graph, rep_periods)
dataframes = construct_dataframes(graph, rep_periods, cons_partitions))
model = create_model(graph, rep_periods, dataframes, base_periods)

The CSV folder has 8 data sources:

• 2 for time-invariant properties
• 2 for clustered profile data
• 2 for partition data
• 1 for rep period definition
• 1 for mapping of year partition and rp_periods

To avoid data duplication, and simplify the process, it might be better to move/keep most of the data to Table format.

The main questions/changes are the following:

• Remove or reduce MetaGraph
  • It currently houses the time-independent data, profiles, partitions, and solution
  • It is used extensively, since most data is only available here
• Remove intermediate steps between partition spec and dataframes
  • Currently goes from CSV to MetaGraph, then some parts are used for cons_partitions which gets translated into the dataframes.
  • We would need to store the assets partitions somewhere, as they currently only exist in the MetaGraph.
• Store Solution as DataFrames
  • Currently, it partly requires using existing DataFrames (see save_solution_to_file).
• Store all variables as DataFrames
  • The linearised variables flows and storage_level have data frames associated. The storage_level shares a data frame with the constraints partitioning.
  • The investment could also have a dedicated DataFrame. If the solution moves to DataFrames, this can just be another column.

Related issues:

• Add the constraints for short- and long-storage in the model #456
• Create a diagram of the workflow #415
• Build the input workflow #289
• Define scenario specification workflow #414

cc. @clizbe @suvayu @datejada

[suvayu]

Agree with standardising all internal data structures to be one kind. For the input workflow, I'm assuming every pipeline in the end produces a DataFrame that can be passed on to TEM. As for intermediate data structures in the workflow, I suspect they will always be tabular, so if in Julia they are DataFrames, and if in DuckDB they will be tables.

[abelsiqueira]

Here's an draft DB schema-ish/workflow-ish visualisation of a DataFrames-centric approach.