Killing pybamm job after a certain time

[isaacbasil]

Hi,

I am trying to run an optimisation problem for a custom model in pybamm, which I believe is not suited to pybop, so I am coding it manually. The problem I am facing is that for certain input parameters, the Casadi solver seems to get stuck, showing a message such as:

At t = 2.5034e-07 and h = 1.36424e-13, the corrector convergence failed repeatedly or with |h| = hmin

and not advancing. I want to kill this solver job if it is still hanging after a certain time. I tried doing this with signal.alarm(timeout) but this doesn't successfully kill it, I guess because of the C++ backend that the solver uses. I have also tried doing it with the multiprocessing package, which does successfully kill the solver, but this prevents me from doing the optimisation in parallel, because daemonic processes cannot spawn children.

Does anyone have any recommendations on how I could kill the solver after a certain time? I can't see any obvious inputs to the solvers that would do this.

[valentinsulzer]

Not a direct answer but we recommend using the IDAKLUSolver instead of the Casadi solver, this is now the default in the latest release

[isaacbasil]

thanks, I have just tried running it with IDAKLU solver and it does seem to be performing better (despite still getting stuck eventually).

[aabills]

I've used this package with some success for this: https://pypi.org/project/func-timeout/

It uses a separate thread to call the function (in this case the model), and kills the thread if things go over the timeout period

[isaacbasil]

Thanks very much - I just tried with that. It definitely seems to be fixing some problems and successfully timing out on some of the solver iterations (compared to before where it would never time out). For some reason it still seems to be getting stuck in certain situations, so I'm not sure why that is.

[aabills]

Yeah it's not perfect, I never figured out why

[isaacbasil]

no problem thanks anyway it's definitely helping