openevolve checkpoint cannot save the proper number of evolved programs

[zigzagcai]

Hi developers,

I meet one issue when I try to integrate openevolve into our workload: In the checkpoint_{ckpt_interval} folder, I found the number of evolved programs not equal to the expected number.

[codelion]

Do you have the folder and the logs for the run? At times if the checkpointed iteration fails due to some error there won't be a checkpoint folder but I think otherwise all the programs should be present in it.

[zigzagcai]

Hi @codelion , we have the logs for the checkpoint part. From the logs we can find the checkpoint_{ckpt_num} not equal to the expected saved program number.

2025-09-22 14:56:48,570 - openevolve.database - INFO - Saved database with 3 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_0
2025-09-22 15:07:11,241 - openevolve.database - INFO - Saved database with 5 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_3
2025-09-22 15:12:55,221 - openevolve.database - INFO - Saved database with 6 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_6
2025-09-22 15:18:54,362 - openevolve.database - INFO - Saved database with 9 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_9
2025-09-22 15:42:55,058 - openevolve.database - INFO - Saved database with 15 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_12
2025-09-22 15:46:13,357 - openevolve.database - INFO - Saved database with 16 programs to openevolve_output/Matmul_KernelGen/checkpoints/checkpoint_15

I find that the _save_checkpoint function is defined here, and the checkpoint function is called here and here.

Is this due to lack of status synchronization before _save_checkpoint , resulting in the mismatch number of saved checkpointed programs?

[codelion]

The checkpoint numbers (0, 3, 6, 9, 12, 15) represent iteration numbers, but the program count in each checkpoint can differ from the iteration number.

We only store the successful programs in the database so if there is an error (see https://github.com/codelion/openevolve/blob/748602849767c9fd4f484c04ff28796bd03cbf4a/openevolve/process_parallel.py#L475) we do not add the program to database.

When we do migrations we copy the program (see https://github.com/codelion/openevolve/blob/748602849767c9fd4f484c04ff28796bd03cbf4a/openevolve/database.py#L1562) which can increase the total count of programs in database.

In the logs you can see if there are errors that lead to programs not been created or migrations that cause programs to be copied.

[zigzagcai]

The checkpoint numbers (0, 3, 6, 9, 12, 15) represent iteration numbers, but the program count in each checkpoint can differ from the iteration number.

We only store the successful programs in the database so if there is an error (see

openevolve/openevolve/process_parallel.py

Line 475 in 7486028

if result.error:
) we do not add the program to database.
When we do migrations we copy the program (see

openevolve/openevolve/database.py

Line 1562 in 7486028

self.programs[migrant_copy.id] = migrant_copy
) which can increase the total count of programs in database.
In the logs you can see if there are errors that lead to programs not been created or migrations that cause programs to be copied.

Thank you for your invaluable response @codelion !
I will check with it.