openevolve does not correctly clean up when there are failures that lead to retries

[theahura]

Recently was running an experiment with openevolve and our compute cluster was, unbenownst to me, down. This lead to some interesting openevolve behavior -- it would fail, retry a bunch of times based on the config, and then instead of actually exiting when all the retries failed, it would...keep going?

Here's an example of the output logs:

2025-10-11 21:36:42,862 - ERROR - All 1 attempts failed with error: Connection error.                                                                                                                                                                                                                                                                                                     
2025-10-11 21:36:42,862 - DEBUG - Raising connection error                                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,862 - ERROR - All 1 attempts failed with error: Connection error.                                                                                                                                                                                                                                                                                                     
2025-10-11 21:36:42,862 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,863 - ERROR - LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                                                
2025-10-11 21:36:42,864 - WARNING - Iteration 147 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,864 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,864 - WARNING - Iteration 148 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,865 - WARNING - Iteration 149 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,865 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,865 - WARNING - Iteration 150 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,865 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,865 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,866 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,866 - WARNING - Iteration 151 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,866 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,866 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,866 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,866 - WARNING - Iteration 152 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,866 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,866 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,867 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,867 - WARNING - Iteration 153 error: LLM generation failed: Connection error.                                                                                                                                                                                                                                                                                         
2025-10-11 21:36:42,867 - DEBUG - Using selector: EpollSelector                                                                                                                                                                                                                                                                                                                           
2025-10-11 21:36:42,867 - DEBUG - Sampled parent 3b530fc4-8c86-4afe-809a-2773e5d8d917 and 0 
inspirations from island 0                                                                                                                                                                                                                                                                    
2025-10-11 21:36:42,867 - INFO - Sampled model: gpt-oss-120b                                                                                                                                                                                                                                                                                                                              
2025-10-11 21:36:42,867 - WARNING - Iteration 154 error: LLM generation failed: Connection error.     

Is there any kind of error handling that I am missing here? It seems like this is all happening inside the openevolve run function, so I can't figure out how to stop this behavior without monkeypatching the library.

More generally, I can imagine all kinds of issues with our server that I'd want openevolve to robustly fail for, e.g. if openevolve hits a max token limit or times out our server

[theahura]

Here is how I am currently patching openevolve locally:

 72                 if not hasattr(self, '_consecutive_failures'):
 73                     self._consecutive_failures = 0
 74                     self._max_consecutive_failures = 3
 75
 76                 def _is_connection_error(msg):
 77                     msg_lower = str(msg).lower()
 78                     patterns = ["connection", "network", "unreachable", "timed out",
 79                                 "getaddrinfo failed", "failed to connect", "llm generation failed"]
 80                     return any(p in msg_lower for p in patterns)
 81
 82                 if result.error and _is_connection_error(result.error):
 83                     self._consecutive_failures += 1
 84                     logger.error(
 85                         f"🚨 Connection error in iteration {completed_iteration} "
 86                         f"({self._consecutive_failures}/{self._max_consecutive_failures}): {result.error}"
 87                     )
 88                     if self._consecutive_failures >= self._max_consecutive_failures:
 89                         logger.error(
 90                             f"💥 FATAL: {self._consecutive_failures} consecutive connection errors. "
 91                             "This usually means the LLM server URL is incorrect or unreachable. "
 92                             "Halting execution."
 93                         )
 94                         self.shutdown_event.set()
 95                         break
 96                 elif not result.error or not _is_connection_error(result.error):
 97                     # Reset counter on successful iteration or non-connection error
 98                     if hasattr(self, '_consecutive_failures'):
 99                         self._consecutive_failures p 0
100

added to process_paralle.py

This essentially kills the entire process if it gets 3 consecutive failures, which, with retries, is 3 * num-retries. Could add more patterns in that pattern list too, thats obviously a bit of a kludge. You could make that part of the config, maybe some kind of 'fast-fail-list'?

[codelion]

There may be intermittent errors in long running processes so the design choice here was to retry and let it continue with next iteration. If we are consistently getting failures from the LLM inference layer then there isn't much that can be done. Currently it will proceed to completion but there won't be any useful outputs.

[theahura]

You can imagine how this is problematic though. One issue that we run into is sometimes our openevolve jobs get long enough where they start running into max token errors or backend timeouts. In these cases, we eat tons of resources only for the job to not make any progress at all. On a shared cluster, that usage of resources is costly because we are running compute alloc that could go to someone else.

In the code snippet above, it checks for intermittent failures N times, and if there is ever a success, it rewrites the count. So in our case, we have our openai config retry count set to 5, and this set to 3. Which means it will only fail if we get 15 consecutive LLM connection failures. That's a lot!

Part of why I think this could be configurable is bc it does not seem all that hard to implement a check where you do something like 'if this config value is set, add smarter fast fail options, otherwise just do the default thing'. We can have the cake you're describing and eat it too