Define the workflow

[clizbe]

Description

What do we need from Toolbox/ESDL and what's our pipeline going to be?

• What are all the pieces of the pipeline?
• How much of the pipeline can SpineToolbox cover?
• How much of the pipeline can ESDL/EDR cover?
• What is left uncovered?
• Do we need to change the input of the model to "match" SpineToolbox?

Related Issues

Primary #289
See also discussion #56

Blocking #94, #106, #105, #118, #88, #89, #115, #36

[clizbe]

(draft - feel free to edit)

Pipeline	SpineToolbox	ESDL	Tulipa	Other
1. Reference/Store Raw Data Sources	Plug-in	YES	-	-
- MIDDEN (EDR or Energy.nl)	Reference	Reference	-	-
- Profiles (CoreRES for instance)	Reference	Reference	-	-
- Networks (ENTSO-E, TenneT)	Reference	Reference	-	-
- Others?	"	Reference	-	SQL
2. Preprocessing Steps	Plug-in	-	Some	-
3. Specifying scenarios	YES	Partly	Possibly	Other file
4. More pre-processing according to scenarios?	Plug-in	-	Can	-
5. Conversion into model in Julia	Plug-in?	-	Possibly	-
6. Running model	-	-	YES	-
7. Output results to somewhere	New SQL db	Tulipa DB	-	-
8. UI for graphs and analyses	Simple	MapEditor?	-	ESM UI, others
9. Share results with other models	-	YES	-	-

[clizbe]

The more I look at this, the more I think we should use SpineToolbox to integrate everything and the EDR as just one of several data sources. And maybe also be able to put results back into ESDL to share results with other models and use the MapEditor etc for analysis (depending on squeeze-to-juice ratio).

Mostly when looking at the SpineToolbox column - it really highlights that SpineToolbox is an integrator. Really, all of these pieces are different tools that have just a bit of overlap. We should decide where the interfaces are.

What I'm a bit concerned about is the whole system still working in 5 years. Seems a bit complex.

[clizbe]

Assigned myself although this is a group effort.

[clizbe]

Discussion:
Suvayu recommends not using SpineToolbox because it has its own limitations. It can be added as a bonus later.
ESDL is usable - we just need to figure out the other parts of the process and how we want it to flow.

[clizbe]

WHAT WE WANT

• Build the network once (in a while)
  • Use draft networks to build new networks
• Sufficient flexibility for ad-hoc code for experimentation
• Definition of temporal stuff
• Definition of scenarios (what is included here?)
  • Scope: just model or parts of pipeline (which parts?)
• Definition of solver specifications
• Be able to mix data sources (ESDL + ENTSO-E for example)
• Self-hosted Tulipa database (in case sources change/vanish, & reduce re-pulling/processing data)
• Export ESDL to simplified representation that is compatible with Tulipa

[clizbe]

Incorporated into issue