Skip to content

Commit 9b9796d

Browse files
safooraySafoora Yousefi
andauthored
Read me updates + docstr pipeline (#170)
Co-authored-by: Safoora Yousefi <[email protected]>
1 parent 866f746 commit 9b9796d

File tree

10 files changed

+156
-44
lines changed

10 files changed

+156
-44
lines changed

README.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,18 @@
33
<a href='https://arxiv.org/abs/2409.10566'><img src=https://img.shields.io/badge/arXiv-2409.10566-b31b1b.svg></a>
44
<a href='https://arxiv.org/pdf/2504.00294'><img src=https://img.shields.io/badge/arXiv-2504.00294-b31b1b.svg></a>
55
<a href='https://huggingface.co/datasets/microsoft/Eureka-Bench-Logs/tree/main'><img src=https://huggingface.co/front/assets/huggingface_logo-noborder.svg width="16">Eureka Evaluation Logs</a>
6-
<a href='https://microsoft.github.io/eureka-ml-insights'><img src=docs/figures/github.png width="16">Project Website</a>
6+
<a href='https://microsoft.github.io/eureka-ml-insights'><img src=readme_docs/figures/github.png width="16">Project Website</a>
77
</p>
88

99
This repository contains the code for the Eureka ML Insights framework. The framework is designed to help researchers and practitioners run reproducible evaluations of generative models using a variety of benchmarks and metrics efficiently. The framework allows the user to define custom pipelines for data processing, inference, and evaluation, and provides a set of pre-defined evaluation pipelines for key benchmarks.
1010

1111
## 📰 News
1212

13-
- **[2025/5/20]**: We have uploaded logs from all experiment reported in our papers on [HuggingFace](https://huggingface.co/datasets/microsoft/Eureka-Bench-Logs/tree/main).
14-
- **[2025/4/29]**: New blog post out [Eureka Inference-Time Scaling Insights: Where We Stand and What Lies Ahead](https://www.microsoft.com/en-us/research/articles/eureka-inference-time-scaling-insights-where-we-stand-and-what-lies-ahead/)
15-
- **[2025/3/31]**: We have a new paper out [Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead](https://arxiv.org/abs/2504.00294)
16-
- **[2024/9/17]**: New blog post out [Eureka: Evaluating and understanding progress in AI](https://aka.ms/eureka-ml-insights-blog)
17-
- **[2024/9/17]**: New paper out [Eureka: Evaluating and Understanding Large Foundation Models](https://arxiv.org/abs/2409.10566)
13+
- **[2025/5/20]**: <img src=https://huggingface.co/front/assets/huggingface_logo-noborder.svg width="16"> We have uploaded logs from all experiment reported in our papers on [HuggingFace](https://huggingface.co/datasets/microsoft/Eureka-Bench-Logs/tree/main).
14+
- **[2025/4/29]**: <img src=readme_docs/figures/msr_blog.png width="16"> New blog post out [Eureka Inference-Time Scaling Insights: Where We Stand and What Lies Ahead](https://www.microsoft.com/en-us/research/articles/eureka-inference-time-scaling-insights-where-we-stand-and-what-lies-ahead/)
15+
- **[2025/3/31]**: <img src=readme_docs/figures/arxiv_logo.svg width="16"> We have a new technical report out [Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead](https://arxiv.org/abs/2504.00294)
16+
- **[2024/9/17]**: <img src=readme_docs/figures/msr_blog.png width="16"> New blog post out [Eureka: Evaluating and understanding progress in AI](https://aka.ms/eureka-ml-insights-blog)
17+
- **[2024/9/17]**: <img src=readme_docs/figures/arxiv_logo.svg width="16"> New technical report out [Eureka: Evaluating and Understanding Large Foundation Models](https://arxiv.org/abs/2409.10566)
1818
## Table of Contents
1919
- [Eureka ML Insights Framework](#eureka-ml-insights-framework)
2020
- [📰 News](#-news)
@@ -98,7 +98,7 @@ The results of the experiment will be saved in a directory under `logs/FlenQA_Ex
9898
For other available experiment pipelines and model configurations, see the `eureka_ml_insights/user_configs` and `eureka_ml_insights/configs` directories, respectively. In [model_configs.py](eureka_ml_insights/configs/model_configs.py) you can configure the model classes to use your API keys, Key Vault urls, endpoints, and other model-specific configurations.
9999

100100
## 🗺️ Overview of Experiment Pipelines
101-
![Components](./docs/figures/transparent_uml.png)
101+
![Components](./readme_docs/figures/transparent_uml.png)
102102
Experiment pipelines define the sequence of components that are run to process data, run inference, and evaluate the model outputs. You can find examples of experiment pipeline configurations in the `user_configs` directory. To create a new experiment configuration, you need to define a class that inherits from `ExperimentConfig` and implements the `configure_pipeline` method. In the `configure_pipeline` method you define the Pipeline config (arrangement of Components) for your Experiment. Once your class is ready, add it to `user_configs/__init__.py` import list.
103103

104104

docs/figures/Benchmarks.png

-280 KB
Binary file not shown.

docs/figures/huggingface_logo-noborder.svg

Lines changed: 0 additions & 37 deletions
This file was deleted.
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
You are given the contents of a python file between the <module_content> </module_content> tags. This module contains classes and functions. Your task is write docstrings for the module, classes and functions in Google style.
2+
If there already exist docstrings, you should rewrite them to be in Google style.
3+
If there are no docstrings, you should write them in Google style.
4+
For dataclasses, since there is no explicit init method, make sure to add attribute docstrings to the class docstring. Note that if a dataclass inherits from other dataclasses, make sure to add all attributes of the base classes as well as new attributes to the docstring.
5+
Rewrite the whole file with no changes to the code. Only the docstrings should be changed.
6+
7+
<module_content>
8+
{{ file_content }}
9+
</module_content>
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
"""
2+
This module defines transformations for reading and writing file content
3+
and an experiment pipeline configuration for generating docstrings.
4+
"""
5+
6+
import os
7+
from dataclasses import dataclass
8+
from typing import Any
9+
10+
from eureka_ml_insights.configs import (
11+
DataSetConfig,
12+
ExperimentConfig,
13+
InferenceConfig,
14+
ModelConfig,
15+
PipelineConfig,
16+
PromptProcessingConfig,
17+
)
18+
from eureka_ml_insights.core import DataProcessing, Inference, PromptProcessing
19+
from eureka_ml_insights.data_utils import (
20+
DataReader,
21+
DFTransformBase,
22+
MMDataLoader,
23+
)
24+
25+
26+
@dataclass
27+
class FileReaderTransform(DFTransformBase):
28+
"""
29+
A transformation that reads file content from a specified file path column.
30+
"""
31+
32+
file_path_column: str = "file_path"
33+
34+
def transform(self, df):
35+
"""
36+
Transform the DataFrame by reading content based on file paths.
37+
38+
Args:
39+
df (pandas.DataFrame): The input DataFrame with the column specified
40+
by file_path_column.
41+
42+
Returns:
43+
pandas.DataFrame: The DataFrame with a new column 'file_content'.
44+
"""
45+
# Implement the logic to read files from the specified column
46+
df["file_content"] = df[self.file_path_column].apply(lambda x: open(x).read())
47+
return df
48+
49+
50+
class FileWriterTransform(DFTransformBase):
51+
"""
52+
A transformation that writes file content to a specified file path.
53+
"""
54+
55+
file_path_column: str = "file_path"
56+
file_content_column: str = "model_output"
57+
58+
def transform(self, df):
59+
"""
60+
Transforms the DataFrame by writing file content to disk.
61+
62+
This method replaces certain path elements, creates output directories if needed,
63+
and writes the content from file_content_column to files specified by file_path_column.
64+
65+
Args:
66+
df (pandas.DataFrame): The input DataFrame containing file path and content columns.
67+
68+
Returns:
69+
pandas.DataFrame: The original DataFrame after writing the content to disk.
70+
"""
71+
for index, row in df.iterrows():
72+
with open(row[self.file_path_column], "w") as f:
73+
f.write(row[self.file_content_column])
74+
return df
75+
76+
77+
class DOCSTR_PIPELINE(ExperimentConfig):
78+
"""
79+
An experiment configuration for a docstring generation pipeline.
80+
"""
81+
82+
def configure_pipeline(
83+
self, model_config: ModelConfig, resume_from: str = None, **kwargs: dict[str, Any]
84+
) -> PipelineConfig:
85+
"""
86+
Configure the pipeline components for docstring generation.
87+
88+
Args:
89+
model_config (ModelConfig): Configuration for the model.
90+
resume_from (str, optional): Path to a checkpoint to resume from. Defaults to None.
91+
**kwargs (dict[str, Any]): Additional keyword arguments.
92+
93+
Returns:
94+
PipelineConfig: The configured pipeline consisting of data processing,
95+
inference, and post-processing components.
96+
"""
97+
98+
# input file should be a csv file with a column 'file_path' containing paths to Python files you want to add docstrings to.
99+
input_file_path = kwargs.get("input_file_path", os.path.join(os.path.dirname(__file__), "../python_file_list.csv"))
100+
101+
# Configure the data processing component.
102+
data_processing_comp = PromptProcessingConfig(
103+
component_type=PromptProcessing,
104+
prompt_template_path=os.path.join(os.path.dirname(__file__), "../prompt_templates/doc_str.jinja"),
105+
data_reader_config=DataSetConfig(
106+
DataReader,
107+
{
108+
"path": input_file_path,
109+
"format": ".csv",
110+
"header": 0,
111+
"index_col": None,
112+
"transform": FileReaderTransform(file_path_column="file_path"),
113+
},
114+
),
115+
output_dir=os.path.join(self.log_dir, "data_processing_output"),
116+
)
117+
inference_comp = InferenceConfig(
118+
component_type=Inference,
119+
model_config=model_config,
120+
data_loader_config=DataSetConfig(
121+
MMDataLoader,
122+
{"path": os.path.join(data_processing_comp.output_dir, "transformed_data.jsonl")},
123+
),
124+
output_dir=os.path.join(self.log_dir, "inference_output"),
125+
resume_from=resume_from,
126+
max_concurrent=5,
127+
)
128+
post_process_comp = PromptProcessingConfig(
129+
component_type=DataProcessing,
130+
data_reader_config=DataSetConfig(
131+
DataReader,
132+
{
133+
"path": os.path.join(inference_comp.output_dir, "inference_result.jsonl"),
134+
"transform": FileWriterTransform(),
135+
},
136+
),
137+
output_dir=os.path.join(self.log_dir, "data_post_processing_output"),
138+
)
139+
return PipelineConfig([data_processing_comp, inference_comp, post_process_comp])

readme_docs/figures/arxiv_logo.svg

Lines changed: 1 addition & 0 deletions
Loading
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)