Skip to content

Commit 545eed4

Browse files
committed
update readme
1 parent ba94a96 commit 545eed4

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# 🧩 LLM Structured Output Benchmarks
2-
2+
Benchmark various LLM Structured Output frameworks: Instructor, Mirascope, Langchain, LlamaIndex, Fructose, Marvin, Outlines, etc on tasks like multi-label classification, named entity recognition, synthetic data generation, etc.
33

44
## 🏆 Benchmark Results [2024-06-22]
55
1. Multi-label classification
@@ -57,9 +57,9 @@ The easiest way to create a new framework is to reference the `./frameworks/inst
5757
- `sample_rows` (int): Number of rows to sample from the source data. Useful for testing on a smaller subset of data. Default is $0$ which uses all rows in source_data_pickle_path for the benchmarking. Obtained from the `init_kwargs` in the `./config.yaml` file.
5858
- `response_model` (Any): The response model to be used. Internally passed by the benchmarking script.
5959
1. The class should define a `run` method that takes three arguments:
60-
1. `inputs`: a dictionary of `{"text": str}` where `str` is the text to be sent to the framework
61-
1. `n_runs`: number of times to repeat each text
62-
1. `expected_response`: Output expected from the framework
60+
- `inputs`: a dictionary of `{"text": str}` where `str` is the text to be sent to the framework
61+
- `n_runs`: number of times to repeat each text
62+
- `expected_response`: Output expected from the framework
6363
1. This `run` method should create another `run_experiment` function that takes `inputs` as argument, runs that input through the framework and returns the output.
6464
1. The `run_experiment` function should be annotated with the `@experiment` decorator from `frameworks.base` with `n_runs` and `expected_resposne` as arguments.
6565
1. The `run` method should call the `run_experiment` function and return the three outputs `predictions`, `percent_successful` and `accuracy`.

0 commit comments

Comments
 (0)