Skip to content

Commit 20815b6

Browse files
authored
Update README.md
1 parent 9004242 commit 20815b6

File tree

1 file changed

+12
-5
lines changed

1 file changed

+12
-5
lines changed

README.md

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# optillm
22

3-
optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time.
3+
optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time. A good example of how to combine such techniques together is the [CePO approach](optillm/cepo) from Cerebras.
44

55
[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/codelion/optillm)
66
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1SpuUb8d9xAoTh32M-9wJsB50AOH54EaH?usp=sharing)
@@ -46,9 +46,16 @@ source .venv/bin/activate
4646
pip install -r requirements.txt
4747
```
4848

49-
Set up the `OPENAI_API_KEY` environment variable (for OpenAI)
50-
or the `AZURE_OPENAI_API_KEY`, `AZURE_API_VERSION` and `AZURE_API_BASE` environment variables (for Azure OpenAI)
51-
or the `AZURE_API_VERSION` and `AZURE_API_BASE` environment variables and login using `az login` for Azure OpenAI with managed identity (see [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity)).
49+
We support all major LLM providers and models for inference. You just need to set the correct environment variable and the proxy will pick the corresponding client.
50+
51+
| Provider | Required Environment Variables | Additional Notes |
52+
|----------|-------------------------------|------------------|
53+
| OptiLLM | `OPTILLM_API_KEY` | Uses the inbuilt local server for inference. Supports logprobs and decoding techniques like `cot_decoding` & `entropy_decoding` |
54+
| OpenAI | `OPENAI_API_KEY` | - |
55+
| Cerebras | `CEREBRAS_API_KEY` | - |
56+
| Azure OpenAI | `AZURE_OPENAI_API_KEY`<br>`AZURE_API_VERSION`<br>`AZURE_API_BASE` | - |
57+
| Azure OpenAI (Managed Identity) | `AZURE_API_VERSION`<br>`AZURE_API_BASE` | Login required using `az login`. See [docs for details](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity) |
58+
| LiteLLM | depends on the model | See [docs for details](https://docs.litellm.ai/docs/providers) |
5259

5360
You can then run the optillm proxy as follows.
5461

@@ -325,7 +332,7 @@ Authorization: Bearer your_secret_api_key
325332

326333
## SOTA results on benchmarks with optillm
327334

328-
### CePO on math and code benchmarks
335+
### CePO on math and code benchmarks (Jan 2025)
329336

330337
| Method | Math-L5 | MMLU-Pro (Math) | GPQA | CRUX | LiveCodeBench (pass@1) | Simple QA |
331338
| -------------------------: | :-----: | :-------------: | :--: | :--: | :--------------------: | :-------: |

0 commit comments

Comments
 (0)