About user prompt construction hard code and evoluation history

[FunMelon]

(Non-native English speakers, pool English)

Custom prompt

I want to perform some custom tasks based on the current framework, so I conducted a deep analysis of the user prompt construction method in the current program and found that it is not flexible enough:

1. Some fragments are still hard coded, such as "## Last Execution Output" and "## Diverse Programs", which is not flexible for custom prompt.
2. The interface methods in the fragments.json file have not been fully utilized.
3. Inspiration programs type are several fixed type like "Object-oriented approach", "NumPy-based implementation" ......

This tree topology diagram is my analysis of the current user prompt(diff template)：

1. Yellow block represents files that can be rewritten.
2. Red block represents hard coded in this files.
3. Green block represents interface in the fragments.json.
   



Suggestions for evoluation history

The currently provided programs for evoluation history include the following types, and I believe their effectiveness is questionable：


Previous Attempts

displaying the recent changes (It could be a problem of my understanding)

• The number is determined by the parameter num_top_degrams, which I think is unreasonable.
• Is it reasonable to display the program with the highest fitness after sorting instead of the nearest program, and is it redundant with the top program function?
• If the displayed changes exceed one line, the details of the changes will be hidden, which I think is useless for prompt.
• I think displaying changes in scores is more indicative than simply displaying scores
• Displaying the child nodes (mutated results) of the current node to evolve can be very helpful and can avoid generating the same result multiple times

Top Performing Programs

It is unreasonable to include 'Performs well on' in all metrics for the top program

Diverse Programs

Considering that the top program is not sufficient, this part may not necessarily exist, so its section is hard coded into a string instead of using a template?

Inspiration Programs

• The current sampling rules for Inspiration Programs are inconsistent between parallel and serial methods, and their effectiveness is worth exploring.
• Supporting rewritable inspiration should be more beneficial
  

What I want to do in the future?



1. Separate all hard coded parts and unify them into rewritable files, add similar tree topology diagram to this repository to illustration the construction of prompt.
2. Make significant rewrites to the 'evolution history' section, may overturn current evolutionary thinking.
3. Change more parts such as the Program structure and Inspiration type to partially rewritable to adapt to more flexible tasks, generalize the current framework from hard programming tasks to more general programming interface tasks, such as adjusting a machine learning configuration file to maximize the effectiveness of machine learning.
   
   @codelion What is your opinion on my above suggestion? If you support, I will implement my idea and submit a PR for you later.

[codelion]

I agree the prompt structure and templates can be improved further. It will be good to compare before and after the changes to see what impact the changes have on a few problems or a benchmark like AlgoTune. This will help us figure out if the changes are needed.

[FunMelon]

I agree the prompt structure and templates can be improved further. It will be good to compare before and after the changes to see what impact the changes have on a few problems or a benchmark like AlgoTune. This will help us figure out if the changes are needed.

Ok, I will first template the current prompt construction without altering the final prompt result. Changing the evolutionary logic will be a more complex and verifiable task. I will conduct comparative experiments and share them with everyone until I come up with convincing experimental results.