LLM Reflection/GEPA

[stevenshinechen]

Is there a way to do LLM reflection or GEPA?

I'd like to build a prompt/database of 'learnings' through an LLM that can then be fed in as part of the prompt for the next evolution

Currently there is the LLM evaluator but it seems to only take in current_program, is there a way to feed in any errors/error artifacts and then ensure it appears as part of the next prompt?

[codelion]

Great questions! OpenEvolve already has most of the features you're looking for:

Programs can return debugging data via the EvaluationResult class:

from openevolve.evaluation_result import EvaluationResult

return EvaluationResult(
    metrics={"score": 0.85},
    artifacts={"errors": error_log, "insights": learning_data}
)

Parent program artifacts are automatically passed to the LLM in subsequent iterations. They appear as "Last Execution Output" sections in prompts, giving the LLM context about previous errors and learnings.

There are two options for llm-only evaluation:

• Hybrid mode: Set use_llm_feedback: true in config to add LLM evaluation alongside your evaluator
• Pure LLM: Create a custom evaluator that only calls LLM

For your use case:

1. Enable artifacts in config: include_artifacts: true
2. Return evaluation errors/insights as artifacts in your evaluator
3. The system automatically feeds these to future prompts
4. Check out https://github.com/codelion/openevolve/tree/main/examples/llm_prompt_optimization - it shows LLM-based evaluation with iterative learning

The "prompt/database of learnings" you want is already built-in through the artifact system! Each iteration gets context from previous evaluations.

[stevenshinechen]

Thanks for the help!

Currently I call an LLM API separately and pass it as a message in the error artifacts (which I suppose is similar to your 'Pure LLM' evaluator)

But ideally I'd want the insights to be stored in a database somewhere or modify the system prompt.
The LLM prompt optimization example you showed only does prompt optimization whereas it would be easier to use if we can both evolve code and the prompt at the same time, e.g. it would be nice to have a MAP Elites database for both programs and the prompt

For the artifacts, does the LLM see all previous artifacts? You mention that 'each iteration gets context from previous evaluations' is there a cut off for how many iterations it goes back?

[codelion]

For the artifacts, does the LLM see all previous artifacts? You mention that 'each iteration gets context from previous evaluations' is there a cut off for how many iterations it goes back

It sees the previous top programs and artifacts you can control it with the top program and diverse program config values.

Having the prompts evolve along with the programs would be great, it is not implemented yet, there is another open issue about it as well - #170