Bug: simulations are crashing when the number of sweeps is large

[danlessa]

On a Mac M1, Sonoma 14.2.1, cadCAD will crash if the number of sweeps is too large (more than 500 for a specific simulation)

Console output:

                  ___________    ____
  ________ __ ___/ / ____/   |  / __ \
 / ___/ __` / __  / /   / /| | / / / /
/ /__/ /_/ / /_/ / /___/ ___ |/ /_/ /
\___/\__,_/\__,_/\____/_/  |_/_____/
by cadCAD

cadCAD Version: 0.5.1
Execution Mode: local_proc
Simulation Dimensions:
Entire Simulation: (Models, Unique Timesteps, Params, Total Runs, Sub-States) = (1, 2, 36, 1200, 13)
     Simulation 0: (Timesteps, Params, Runs, Sub-States) = (2, 36, 1200, 13)
Initializing configurations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1200/1200 [00:03<00:00, 374.84it/s]
Execution Method: parallelize_simulations
Execution Mode: parallelized
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/danlessa/repos/bsci/aztec-gddt/aztec_gddt/__main__.py", line 27, in <module>
    main()
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/aztec_gddt/__main__.py", line 20, in main
    df = psuu_exploratory_run()
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/aztec_gddt/experiment.py", line 249, in psuu_exploratory_run
    sim_df = sim_run(*sim_args)
             ^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/aztec_gddt/utils/sim_run.py", line 55, in sim_run
    (records, tensor_field, _) = executor.execute()
                                 ^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/cadCAD/engine/__init__.py", line 214, in execute
    simulations_results = self.exec_method(
                          ^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/cadCAD/engine/execution.py", line 105, in parallelize_simulations
    results = flatten(list(map(lambda params: process_executor(params), new_params)))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/cadCAD/engine/execution.py", line 105, in <lambda>
    results = flatten(list(map(lambda params: process_executor(params), new_params)))
                                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/cadCAD/engine/execution.py", line 96, in process_executor
    with ProcessPool(processes=len_configs_structs) as pp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/pathos/multiprocessing.py", line 122, in __init__
    self._serve()
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/pathos/multiprocessing.py", line 134, in _serve
    _pool = Pool(nodes, **self._kwds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/pool.py", line 215, in __init__
    self._repopulate_pool()
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/context.py", line 281, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/danlessa/repos/bsci/aztec-gddt/venv/lib/python3.11/site-packages/multiprocess/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
               ^^^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

[linear]

CORE-105 Bug: simulations are crashing when the number of sweeps is large