Deserialize and run task graphs

[jl-wynen]

I experimented with deserializing JSON task graphs back into sciline.TaskGraph in https://gist.github.com/jl-wynen/d72588744e2813a46dd05521be4d86ba The approach works but has some non-trivial components.

This is not part of the goals of #92 and #124 but it is good to know that it is possible.

Should we make this part of Sciline? It could be handled by the package itself or, if we don't want to distribute it with the package, it could be a script that lives in the Sciline repo.

[SimonHeybrock]

A useful intermediate goal, even if we cannot fully deserialize (e.g., to lack of params, or full code serialization), would be the ability to recreate a TaskGraph for the purpose of visualizing it. This would allow for understanding visually what the computation did.

I think having a deserialize routine alongside serialize probably makes sense in Sciline.

[jl-wynen]

What is your opinion on trying to deserialize keys? And about tracking the installed packages?

[SimonHeybrock]

• Tracking packages and versions seems like a must-have (medium term).
• Deserializing keys does not appear to be fundamental, since the keys are just a way Sciline assembles the task graph, but we already have one when we serialize. In fact, any sort of key-handling will in general not work anymore with the new ideas floated in A new approach for "parameter tables" #141 and Add graph with map and reduce operations. cyclebane#1, since it will allow for transforming task graphs in a way that does not adhere to sciline.Pipeline's way of connecting provider outputs with provider inputs.

[jl-wynen]

In fact, any sort of key-handling will in general not work anymore with the new ideas floated in #141 and scipp/cyclebane#1

Good point. So even the solution in the prototype won't work. I guess we could use the edge id as a key when we only care about running the computation. When it comes to drawing the task graph, we need the key be based on the original key in order to display a meaningful name. But we can wait fand see what the solution in cyclebane looks like.