Merge branch 'main' into feat/MLX-kernel-optimization

Author: codelion

.github/instructions/sidebar-node-logic.instructions.md

@@ -0,0 +1,12 @@
+---
+applyTo: 'scripts/**/*.js'
+---
+- In this program, a dataset with nodes and edges is visualized in different graphs and lists. These view modes are selectable in tabs.
+- Nodes are parametrized with meta data including program ID, island number, generation nunmber, parent ID (which is used to determine the edge connections), a metric dataset with flexible keys, a code string, a dict with prompts and more. All data except program ID are optional.
+- A sidebar shows detailed node information. Its format is the same across all view modes.
+- The sidebar in this program is designed to show up dynamically when a node is selected in one of the graphs or lists. It appears on hover of the node and hides when the node is not hovered anymore.
+- A single node can be selected to turn it "sticky". When a node is sticky, its information remains visible in the sidebar and the sidebar remains open until the user clicks in the background. Hovering another node will not change the sidebar content if a node is already sticky.
+- The selected node is highlighted with a red border and synchronized across all graphs and lists. I.e., clicking a node in a list will also highlight it in the graphs.
+
+- A select box #highlight-select configures a filter logic that allows to highlight multiple nodes. Nodes are highlighted with a blue shadow in the graphs and lists.
+- A select box #metric-select shows the available metrics (determined dynamically from the dataset), and the selected metric may be used in the graph creation, filter and sorting logic.

.github/workflows/python-lint.yml

@@ -13,5 +13,5 @@ jobs:
       - uses: psf/black@stable
         with:
           options: "--check --verbose"
-          src: "./openevolve ./tests ./examples"
+          src: "./openevolve ./tests ./examples ./scripts"
           use_pyproject: true

Makefile

@@ -33,7 +33,7 @@ install: venv
 # Run Black code formatting
 .PHONY: lint
 lint: venv
-	$(PYTHON) -m black openevolve examples tests
+	$(PYTHON) -m black openevolve examples tests scripts
 
 # Run tests using the virtual environment
 .PHONY: test
@@ -50,7 +50,7 @@ docker-build:
 docker-run:
 	docker run --rm -v $(PROJECT_DIR):/app --network="host" $(DOCKER_IMAGE) examples/function_minimization/initial_program.py examples/function_minimization/evaluator.py --config examples/function_minimization/config.yaml --iterations 1000
 
-# Run the lm-eval benchmark
-.PHONY: lm-eval
-lm-eval:
-	$(PYTHON) scripts/lm_eval/lm-eval.py
+# Run the visualization script
+.PHONY: visualizer
+visualizer:
+	$(PYTHON) scripts/visualizer.py --path examples/

README.md

@@ -128,6 +128,32 @@ diff -u checkpoints/checkpoint_10/best_program.py checkpoints/checkpoint_20/best
 # Compare metrics
 cat checkpoints/checkpoint_*/best_program_info.json | grep -A 10 metrics
 ```
+
+### Visualizing the evolution tree
+
+The script in `scripts/visualize.py` allows you to visualize the evolution tree and display it in your webbrowser. The script watches live for the newest checkpoint directory in the examples/ folder structure and updates the graph. Alternatively, you can also provide a specific checkpoint folder with the `--path` parameter.
+
+```bash
+# Install requirements
+pip install -r scripts/requirements.txt
+
+# Start the visualization web server and have it watch the examples/ folder
+python scripts/visualizer.py
+
+# Start the visualization web server with a specific checkpoint
+python scripts/visualizer.py --path examples/function_minimization/openevolve_output/checkpoints/checkpoint_100/
+```
+
+In the visualization UI, you can
+- see the branching of your program evolution in a network visualization, with node radius chosen by the program fitness (= the currently selected metric),
+- see the parent-child relationship of nodes and click through them in the sidebar (use the yellow locator icon in the sidebar to center the node in the graph),
+- select the metric of interest (with the available metric choices depending on your data set),
+- highlight nodes, for example the top score (for the chosen metric) or the MAP-elites members,
+- click nodes to see their code and prompts (if available from the checkpoint data) in a sidebar,
+- in the "Performance" tab, see their selected metric score vs generation in a graph
+
+![OpenEvolve Visualizer](openevolve-visualizer.png)
+
 ### Docker
 
 You can also install and execute via Docker:

openevolve-visualizer.png

@@ -40,7 +40,6 @@ class LLMConfig(LLMModelConfig):
 
     # API configuration
     api_base: str = "https://api.openai.com/v1"
-    name: str = "gpt-4o"
 
     # Generation parameters
     system_message: Optional[str] = "system_message"
@@ -60,10 +59,10 @@ class LLMConfig(LLMModelConfig):
     evaluator_models: List[LLMModelConfig] = field(default_factory=lambda: [])
 
     # Backwardes compatibility with primary_model(_weight) options
-    primary_model: str = "gemini-2.0-flash-lite"
-    primary_model_weight: float = 0.8
-    secondary_model: str = "gemini-2.0-flash"
-    secondary_model_weight: float = 0.2
+    primary_model: str = None
+    primary_model_weight: float = None
+    secondary_model: str = None
+    secondary_model_weight: float = None
 
     def __post_init__(self):
         """Post-initialization to set up model configurations"""

openevolve/config.py

@@ -107,4 +107,8 @@ async def _call_api(self, params: Dict[str, Any]) -> str:
         response = await loop.run_in_executor(
             None, lambda: self.client.chat.completions.create(**params)
         )
+        # Logging of system prompt, user message and response content
+        logger = logging.getLogger(__name__)
+        logger.debug(f"API parameters: {params}")
+        logger.debug(f"API response: {response.choices[0].message.content}")
         return response.choices[0].message.content

openevolve/llm/openai.py

@@ -264,12 +264,12 @@ def _format_evolution_history(
 
                 # Only compare if both values are numeric
                 if isinstance(prog_value, (int, float)) and isinstance(parent_value, (int, float)):
-                    if prog_value >= parent_value:
+                    if prog_value > parent_value:
                         numeric_comparisons_improved.append(True)
                     else:
                         numeric_comparisons_improved.append(False)
 
-                    if prog_value <= parent_value:
+                    if prog_value < parent_value:
                         numeric_comparisons_regressed.append(True)
                     else:
                         numeric_comparisons_regressed.append(False)

openevolve/prompt/sampler.py

@@ -17,6 +17,7 @@ dependencies = [
     "pyyaml>=6.0",
     "numpy>=1.22.0",
     "tqdm>=4.64.0",
+    "flask",
 ]
 
 [project.optional-dependencies]

pyproject.toml

@@ -0,0 +1 @@
+flask