Skip to content

Commit 90bef2e

Browse files
authored
Update README.md
1 parent 3d4f970 commit 90bef2e

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@ response = client.chat.completions.create(
218218
| Plugin | Slug | Description |
219219
| ----------------------- | ------------------ | ---------------------------------------------------------------------------------------------- |
220220
| Router | `router` | Uses the [optillm-bert-uncased](https://huggingface.co/codelion/optillm-bert-uncased) model to route requests to different approaches based on the user prompt |
221+
| Chain-of-Code | `coc` | Implements a chain of code approach that combines CoT with code execution and LLM based code simulation |
221222
| Memory | `memory` | Implements a short term memory layer, enables you to use unbounded context length with any LLM |
222223
| Privacy | `privacy` | Anonymize PII data in request and deanonymize it back to original value in response |
223224
| Read URLs | `readurls` | Reads all URLs found in the request, fetches the content at the URL and adds it to the context |
@@ -290,6 +291,20 @@ Authorization: Bearer your_secret_api_key
290291
```
291292
## SOTA results on benchmarks with optillm
292293

294+
### coc-claude-3-5-sonnet-20241022 on AIME 2024 pass@1 (Nov 2024)
295+
296+
| Model | Score |
297+
|-------|-----:|
298+
| o1-mini | 56.67 |
299+
| coc-claude-3-5-sonnet-20241022 | 46.67 |
300+
| coc-gemini/gemini-exp-1121 | 46.67 |
301+
| o1-preview | 40.00 |
302+
| f1-preview | 40.00 |
303+
| gemini-exp-1114 | 36.67 |
304+
| claude-3-5-sonnet-20241022 | 20.00 |
305+
| gemini-1.5-pro-002 | 20.00 |
306+
| gemini-1.5-flash-002 | 16.67 |
307+
293308
### readurls&memory-gpt-4o-mini on Google FRAMES Benchmark (Oct 2024)
294309
| Model | Accuracy |
295310
| ----- | -------- |
@@ -324,6 +339,7 @@ called patchflows. We saw huge performance gains across all the supported patchf
324339

325340
## References
326341

342+
- [Chain of Code: Reasoning with a Language Model-Augmented Code Emulator](https://arxiv.org/abs/2312.04474) - [Implementation](https://github.com/codelion/optillm/blob/main/optillm/plugins/coc_plugin.py)
327343
- [Entropy Based Sampling and Parallel CoT Decoding](https://github.com/xjdr-alt/entropix) - [Implementation](https://github.com/codelion/optillm/blob/main/optillm/entropy_decoding.py)
328344
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://arxiv.org/abs/2409.12941) - [Evaluation script](https://github.com/codelion/optillm/blob/main/scripts/eval_frames_benchmark.py)
329345
- [Writing in the Margins: Better Inference Pattern for Long Context Retrieval](https://www.arxiv.org/abs/2408.14906) - [Inspired the implementation of the memory plugin](https://github.com/codelion/optillm/blob/main/optillm/plugins/memory_plugin.py)

0 commit comments

Comments
 (0)