Performance profiling

[ekatef]

Some performance improvements seem to be desirable to build a global PyPSA-Earth model (see #445 and #543 for details). Profiling may be very helpful to address performance optimisation effectively.

Among Python profilers cProfiler is currently the most handy one. It belongs to the standard Python toolkit is written in C and hence is more efficient as compared with pure Pythonic profiler.

A convenient way to profile scripts being a part of Snakemake workflow is to apply a decorator function, e.g. adding to the workflow something like this:

import cProfile

def profile(func):
    """Decorator for run function profile"""
    def wrapper(*args, **kwargs):
        profile_filename = func.__name__ + '.prof'
        profiler = cProfile.Profile()
        result = profiler.runcall(func, *args, **kwargs)
        profiler.dump_stats(profile_filename)
        return result
    return wrapper
...
@profile
def clean_data(
    input_files, ...

That will save the profiling results as a .prof file which can be analysed further with performance analysis tools. The following visualisation tools have been tested:

1. Snakeviz (pip install snakeviz)

snakeviz clean_data.prof

2. tuna (pip install tuna)

tuna clean_data.prof

3. flameprof (pip install flameprof)

flameprof --cpu clean_data.prof > requests.svg

4. grpof2dot (pip install gprof2dot)

gprof2dot  --node-label "total-time" -n 0.001 -f pstats clean_data.prof | dot -Tpng -o clean_osm_graph2.png

Currently my impession is that snakeviz is the most convenient from UX point of view. However, tuna and flameprof may address some specific performance tracing issues more precisely. The workflow graphs of may be helpful to understand the workflow sub-stages better.