[Bug Report] `StorageMonitor` exits prematurely when new futures are added to a running monitor

[neilSchroeder]

original report

I'm attempting to use lithops with cubed to cloud process a very large zarr store.

The graph for this looks it has 5 steps. The middle three steps have about ~3300 jobs that lithops attempts to submit to aws lambda. The first couple steps seem to work fine and all jobs successfully submit and run, but then on the third step it successfully submits 500 jobs (the limit I've set), all of which finish successfully, and then it hangs and fails to submit any more jobs just repetitively informing me that it's waiting for 2800 more jobs to run.

Is this a known issue? Is there something I can do to fix this?

I'm running on python 3.12 and am happy to provide more details as necessary.

---

Bug Report: StorageMonitor exits prematurely when new futures are added to a running monitor

When using a FunctionExecutor to run multiple sequential map() calls within the same executor context, the StorageMonitor can exit prematurely, leaving newly added futures unmonitored. This causes jobs to hang indefinitely at "Waiting for 0% of N function activations to complete".

Environment

• Lithops: 3.6.2
• Python: 3.12.11
• Backend: AWS Lambda (us-west-2)
• Storage: AWS S3

Analysis

This bug appears to be a race condition in lithops.monitor involving the interaction of StorageMonitor.run, Monitor._all_ready(), and Monitor.add_futures().

The race conditions seems to go like this:

1. Job N completes: All futures in self.futures done
2. _all_ready() returns True: The while loop condition becomes false
3. Job N+1 starts: add_futures() is called to add new futures to self.futures
4. Monitor exits: The while loop exits before checking the newly added futures
5. Result: The new futures from Job N+1 are never monitored

Debug Logs

2025-11-29 16:36:49,038 [DEBUG] monitor.py:147 -- ExecutorID a31010-20 - Pending: 0 - Running: 0 - Done: 3328
... [a bunch of invoker/wait/futures/etc. outputs here]...

# the very next monitor statements say:
2025-11-29 16:37:00,166 [DEBUG] monitor.py:147 -- ExecutorID a31010-20 - Pending: 3226 - Running: 54 - Done: 3376 # note that 3376 is more than than the number pending so the monitor exits
2025-11-29 16:37:00,175 [DEBUG] monitor.py:481 -- ExecutorID a31010-20 - Storage job monitor finished
...
2025-11-29 16:37:39,674 [INFO] wait.py:105 -- ExecutorID a31010-20 - Waiting for 0% of 3280 function activations to complete
# ^^ Hangs here forever - monitor has exited, 3280 tasks never processed

Second potential bug:

JobMonitor.is_alive() calls self.monitor.is_alive() but does not return anything. Additionally, no other monitor appears to have an is_alive definition. This function will always return None resulting in some strange behavior since None is falsey and may be contributing to the issue above by causing incorrect decisions about whether or not to create or start a new monitor.

[neilSchroeder]

cubed-dev/cubed#841

A separate discussion on the same topic is happening at the discussion above.

[neilSchroeder]

I made a local patch to lithops to fix the problem. I don't think this is necessarily a good production solution, but it did fix the issues I was having.

1. Fixed JobMonitor.is_alive() (line 523) - Missing return statement:

Before (bug):

def is_alive(self):
    self.monitor.is_alive()  # Returns None!

After (fixed):

def is_alive(self):
    return self.monitor.is_alive()

2. Fixed StorageMonitor.run() race condition - Require 3 consecutive _all_ready() checks before exiting:

Before: Exits immediately when _all_ready() returns True

while not self._all_ready():
    ...

After: Requires 3 consecutive True checks (~3 seconds window)

consecutive_all_ready = 0
while self.should_run:
    ...
    if self._all_ready():
        consecutive_all_ready += 1
        if consecutive_all_ready >= 3:
            break
    else:
        consecutive_all_ready = 0

[neilSchroeder]

A more robust solution is to make sure the monitor never exits on its own and only exits when explicitly told to.

while self.should_run:
    ...
    # No exit condition - just keep polling until done

The monitor thread stays alive slightly longer than strictly necessary, but for me the reliability gain is worth the trade-off

[tomwhite]

Thanks @neilSchroeder for digging in to possible fixes for these bugs!

@JosepSampe - do you have any thoughts about these?

1. Fixed JobMonitor.is_alive() (line 523) - Missing return statement:

FIW it looks like this was introduced in #1441 which was released in Lithops 3.6.1.

[JosepSampe]

@tomwhite Yep, there's a missing return statement there. Thanks @neilSchroeder for digging into it. I'll submit a fix.

[tomwhite]

Thanks for the fix @JosepSampe! What do you think the best solution for the race condition is?