How to smartly manage model compilation?

[clizbe]

Compiling the model takes a lot of time (Julia thing) with subsequent runs going faster.
(Becoming a problem in Spine, so let's think about this early.)

How are we dealing with this in the workflow?
Is the stable version of Tulipa something that compiles once and then can take any data through it?
Or will the scenario define a model that needs precompiling before doing multiple runs?

---

@suvayu's thoughts:
I think this request needs to be separated according to use case. For example, if you changed an input dataset, naively, you have to rerun. However if you say "I'm doing a sensitivity study, and my changes are only limited to X" then theoretically the repetitions need not start from scratch. But I think that's a very advanced feature which requires deep technical research. AFAIU, this is in @g-moralesespana and @datejada's wishlist (GUSS in GAMS). But there could be simpler use cases between these two extremes.

That said, I'm not sure whether this would fall under the purview or pipeline/workflow or model building. My hunch is, it'll depend on the use case.

@datejada @g-moralesespana @gnawin @abelsiqueira

[clizbe]

Maybe the model needs to recognize when scenario changes require recompilation? Is this something Julia can do automatically?

[suvayu]

Maybe this is a relevant discussion: spine-tools/SpineOpt.jl#885

[suvayu]

I'm semi convinced this is a difficult problem, and any implementation would have to be on our side (not JuMP)

[clizbe]

Maybe relevant from @datejada's notes from JuMP Con:

"There is a new cool package for JuMP called ParametricOptInterface.jl, which will make our life easier to implement the rolling horizon feature in Tulipa (@g-moralesespana it effectively updates parameters on a model without rebuilding the model). I talked to Joaquim (one of the leading developers). I will contribute with an example in their repo on how to set a rolling horizon type of application, which we can take as a starting point for Tulipa."