PyPSA-Earth dockerization

[ekatef]

It might be worth to create a docked image of PyPSA-Earth. The idea has been discussed a few times with the main output that a deeper investigation and some experiments might be beneficial to better understand suitability of the dockerization approach.

An example of Docker implementation is available #726 (thanks @joconnor-ecaa and @Tomkourou)

Trying to collect pros and cons as they have appeared throughout previous discusstions.

Advantages

1. Containers are very handy to standardise software distribution while Docker is a current standard de facto among all container types
2. Docker isolates an application process from the OS thus minimising efforts to get the model run including installing all the dependencies required by the workflow (mind JDK!)

Disadvantages

1. A need to install Docker itself
2. Additionally maintenance efforts needed to create the docker image including registry management and considerations related to security features

[ekatef]

@Tomkourou, @mnm-matin, @pz-max, @davide-f would be grateful for your comments and corrections on the topic

[Tomkourou]

Thanks for starting this discussion @ekatef. As far as I can tell from my experimentation with Docker you actually won't have to build OS-specific images, Docker will do that for you by default.

You will however need to add pypsa-earth onto a registry and manage it there. This would certainly mean additional work.

The Docker image can also be used to install other dependencies out of the box such as openjdk. I also added an empty gurobi.lic to my container and set it up so that gurobipy already knows where to look for it once someone adds their credentials into it. Overall I think it's a convenient way to install and use pypsa-earth.

In the end, I think it would be useful to hear from you on new user experience with installing pypsa-earth. How often do you need to help people, how often is environment management a bottleneck. If it happens a lot, perhaps Docker could help ease this issue. If not, then it might not be worth implementing it.

[ekatef]

Thanks for starting this discussion @ekatef. As far as I can tell from my experimentation with Docker you actually won't have to build OS-specific images, Docker will do that for you by default.

You will however need to add pypsa-earth onto a registry and manage it there. This would certainly mean additional work.

The Docker image can also be used to install other dependencies out of the box such as openjdk. I also added an empty gurobi.lic to my container and set it up so that gurobipy already knows where to look for it once someone adds their credentials into it. Overall I think it's a convenient way to install and use pypsa-earth.

In the end, I think it would be useful to hear from you on new user experience with installing pypsa-earth. How often do you need to help people, how often is environment management a bottleneck. If it happens a lot, perhaps Docker could help ease this issue. If not, then it might not be worth implementing it.

@Tomkourou thanks a lot for docker implementations and comments! The post updated.

Absolutely, installation of PyPSA-Earth can be quite a challenge for new users, especially on Windows. We have been discussing using docker images to facilitate it, and my feeling is that it could work very well.

[pz-max]

@ekatef I think we have test implementation here: #726 (thanks @Tomkourou)
It looks quite convenient.

Several things need to be tested/ considered:

• Documentation on the docker installation procedure especially with solvers like gurobi and anything else that is required for the use
• Docker for PyPSA-Earth development. How easy is it to debug dependencies with docker?
• What about reproducibility? Can I run an old docker image easily (probably yes?)
• Write and run a test with and without the docker image
• Who will continuously maintain the docker image? (saw the above PR has an automatic docker update workflow)

For the last point, we can do that together if it really helps a lot of people. So far the environment installation seems stable. As docker can be useful for Kubernetes stuff, I think we could have both solutions in the beginning.

--- Add
@ekatef yes, splitting dev "main" and release sounds good. But I think we could also stick for the main at the moment.
In good Python projects such as "pandas" they have in the main a Dockerfile as well. Additionally, they have a branch for each release. I think that's required if people want to stay to one particular version e.g. 2.0.x and keep this maintained without moving to another version...? So all this suggest that having a Dockerfile in the main should not cause problems

[ekatef]

@ekatef I think we have test implementation here: #726 (thanks @Tomkourou) It looks quite convenient.

Several things need to be tested/ considered:

• Documentation on the docker installation procedure especially with solvers like gurobi and anything else that is required for the use
• Docker for PyPSA-Earth development. How easy is it to debug dependencies with docker?
• What about reproducibility? Can I run an old docker image easily (probably yes?)
• Write and run a test with and without the docker image
• Who will continuously maintain the docker image? (saw the above PR has an automatic docker update workflow)

For the last point, we can do that together if it really helps a lot of people. So far the environment installation seems stable. As docker can be useful for Kubernetes stuff, I think we could have both solutions in the beginning.

@pz-max great points! 🙂 Could we probably split release and development branches, and use a docked image to distribute a stable release? I remember, you have added a citation which explains such an approach very clearly. If you think it's appropriate, I'd be happy to copy these consideration there