|
1 | 1 | # optillm |
2 | 2 |
|
3 | | -optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time. |
| 3 | +optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time. A good example of how to combine such techniques together is the [CePO approach](optillm/cepo) from Cerebras. |
4 | 4 |
|
5 | 5 | [](https://huggingface.co/spaces/codelion/optillm) |
6 | 6 | [](https://colab.research.google.com/drive/1SpuUb8d9xAoTh32M-9wJsB50AOH54EaH?usp=sharing) |
@@ -46,9 +46,16 @@ source .venv/bin/activate |
46 | 46 | pip install -r requirements.txt |
47 | 47 | ``` |
48 | 48 |
|
49 | | -Set up the `OPENAI_API_KEY` environment variable (for OpenAI) |
50 | | -or the `AZURE_OPENAI_API_KEY`, `AZURE_API_VERSION` and `AZURE_API_BASE` environment variables (for Azure OpenAI) |
51 | | -or the `AZURE_API_VERSION` and `AZURE_API_BASE` environment variables and login using `az login` for Azure OpenAI with managed identity (see [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity)). |
| 49 | +We support all major LLM providers and models for inference. You just need to set the correct environment variable and the proxy will pick the corresponding client. |
| 50 | + |
| 51 | +| Provider | Required Environment Variables | Additional Notes | |
| 52 | +|----------|-------------------------------|------------------| |
| 53 | +| OptiLLM | `OPTILLM_API_KEY` | Uses the inbuilt local server for inference. Supports logprobs and decoding techniques like `cot_decoding` & `entropy_decoding` | |
| 54 | +| OpenAI | `OPENAI_API_KEY` | - | |
| 55 | +| Cerebras | `CEREBRAS_API_KEY` | - | |
| 56 | +| Azure OpenAI | `AZURE_OPENAI_API_KEY`<br>`AZURE_API_VERSION`<br>`AZURE_API_BASE` | - | |
| 57 | +| Azure OpenAI (Managed Identity) | `AZURE_API_VERSION`<br>`AZURE_API_BASE` | Login required using `az login`. See [docs for details](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity) | |
| 58 | +| LiteLLM | depends on the model | See [docs for details](https://docs.litellm.ai/docs/providers) | |
52 | 59 |
|
53 | 60 | You can then run the optillm proxy as follows. |
54 | 61 |
|
@@ -325,7 +332,7 @@ Authorization: Bearer your_secret_api_key |
325 | 332 |
|
326 | 333 | ## SOTA results on benchmarks with optillm |
327 | 334 |
|
328 | | -### CePO on math and code benchmarks |
| 335 | +### CePO on math and code benchmarks (Jan 2025) |
329 | 336 |
|
330 | 337 | | Method | Math-L5 | MMLU-Pro (Math) | GPQA | CRUX | LiveCodeBench (pass@1) | Simple QA | |
331 | 338 | | -------------------------: | :-----: | :-------------: | :--: | :--: | :--------------------: | :-------: | |
|
0 commit comments