Threading for I/O operations

[sharon92]

I am working on a relative medium sized project with dimensions of 9 x 1200 x 1000 - 110 stress periods
Reading and writing the input external binary files like RIV, RCH, and more take a considerable amount of time.

Profiling with cProfile points out that most of the time is spent reading the files sequentially.

Is asynchronous data loading in plans?

Some tests:

here m is a loaded Modflow 6 object
m.get_package("rch").recharge.array

this takes a wall time of about 42 seconds

Threaded approach:

from concurrent.futures import ThreadPoolExecutor

def get_recharge_data(i):
    return m.get_package("rch").recharge.get_data(i)

with ThreadPoolExecutor() as executor:
    rch = list(executor.map(get_recharge_data, range(self.m.nper)))

returns with a wall time of 24 seconds,

Not much but still an improvement. Takes about 57% of the time from the built-in approach.

Would love to contribute to these performance improvements if wished for.

[wpbonelli]

@sharon92 thanks for your comment and for volunteering. File IO is a big bottleneck at the moment. We have this in mind for flopy 4.x which is in early development.

Lazy IO (via e.g. dask) is definitely on the agenda. Concurrent and/or parallel IO on separate files is much easier to reason about than on the same file, as in your example, but maybe in certain cases we could consider that too.

Once we are a bit closer to releasing 4.x your input would be much appreciated. MF6 internals are getting a rewrite so I would not advise diving too deeply into the current codebase right now.

[sharon92]

Thanks for the reply, after I posted this I checked that the ThreadPoolExecutor actually didnt make any improvement. Just loading the data sequentially with the recharge.get_data method was faster than recharge.array so I wanted to delete the post but you can only close it I think.

Thanks! really looking forward to 4.x, yes I have realised you can go into a rabbit hole when trying to find out more about the underlying functions in flopy :)