✨ LiteLLM March 2025 Roadmap

[krrishdholakia]

Organizing Improvements in 8 areas:

1. Security

• [Bug]: Editting a key in the UI clears its user_id via /key/update (vuln?) #8394

2. Reliability & Performance

• Improve litellm python import speed [Feature]: Improve import speed #7605
• [Feature]: Fallback for only some errors #8789
• [Bug]: NO FALLBACK when streaming and [Bug]: litellm.InternalServerError: AnthropicException - Overloaded. Handle with litellm.InternalServerError. #6957
• [Feature]: aiohttp migration - 10-100x Higher RPS Master ticket #7544

3. LLM Translation

New Providers, models, endpoints

• [Feature]: Support Embeddings in Infinity #8764
• [Feature]: Add Huggingface Text Embeddings Inference (TEI) provider for /rerank  #8372
• [Feature]: Support Venice AI provider #9093
• Gemini Multimodal api support Gemini Multimodal Live API support #9018
• [Feature]: Support Mistral OCR #9051
• [Feature] MCP bridge support #7934
• VLLM Rerank endpoint support

Anthropic

• [Feature]: Invoke Claude Thinking when using OpenAI API #9022
• [Bug]: Claude-3.7 'thinking' parameter not support with litellm_proxy #8916
• Claude 3.7 extended thinking mode and structured output via LiteLLM not compatible #8901
• [Bug]: Add Bedrock Support for /v1/messages API  #9159

Vertex / Google AI Studio

• Vertex AI logprobs support: [Feature]: Support logprobs for Vertex AI gemini Models #9091
• [Bug]: Gemini Flash 2.0 implementation is not returning the logprobs #8890
• [Bug]: error 400 when passing tools=[] to gemini #9080
• Vertex AI API supports schemas not supported by Gemini API, LiteLLM should reflect this #8864
• [Bug]: BadRequestError when passing a GCS URI to Gemini models #8849
• [Feature]: Context Caching for Vertex AI #6898

Bedrock

• Bedrock Application Inference Profile support [Bug]: Litellm does not work with AWS Bedrock Application Inference profiles #8911
• [Bug]: Bedrock Embeddings model cohere.embed-multilingual-v3 does not work in the OpenAI client unless you specify encoding_format=None #9066

OpenAI

• [Bug]: index field not populated for a chunk with choices when calling tools using OpenAI (streaming mode, n>1) #8977
• [Bug]: max_completion_tokens is not supported for some gpt-4o deployments on Azure #8855
• Fix model info API for whisper + dall-e-3 [Bug]: Model info API with whisper and Dall-e #7584 (comment)
• [Bug]: Realtime websocket #8828

4. Spend Tracking / Budgeting

• Azure data zone cost tracking [Feature]: Add o3 Data Zone model prices #8692
• [Bug]: multi instance virtual key tpm rpm setting can not synchronization #9008
• [Bug]: Floating point rounding errors in x-litellm-response-cost header #8867
• [Feature]: Expand Spend Logs with Cached Usage details #8784
• [Bug]: Cost calculation for Gemini 2 Flash on Vertex raises ">128k tokens" error #8579
• [Bug]: Cannot track spend by tag with /audio/transcription endpoint #8439
• [Feature]: reset budget according to calender month #9098

5. Logging / Guardrails

• [Feature]: Add supported call types for langfuse #8936
• Arize AI project-based logging
• Cached / multimodal token are not passed through to Langfuse #8515
• [Bug]: Message Redaction on OTEL impacts logging output on s3 #8489 (v2 of redaction implementation when using multiple logging integrations)
• Azure content safety integration #2071
• Lakera V2 API support

6. Admin UI Improvements

• [feat] The All Up and Team Based Usage 'Usage' tab pages need to have a date selector with the usual dropdowns / custom querying (last 7, last 28, previous month, etc etc etc) (Like it already is in the Customer Usage page)
• [feat] The 'Select Key' in Customer Usage should be searchable, and filterable by user/team
• [bug] While the 'Internal Users' UI did get the sorting we talked about last time, it appears to only be sorting/filtering within the page and not the global query; i.e. one would expect that sorting by decreasing spend on page 1 would have the top users, but if that user didn't appear on page 1, it won't.
• [feat] API keys that users create without assigning them to teams cannot be re-assigned via the UI (They can via the API, but I'll ask about that again in a bit...)
• [feat] API Keys should be able to be filtered by User
• [feat] The 'API Keys' column in the Users page should forward to the Virtual Keys page filtered by that User
• [feat] In 'Virtual Keys' it'd be helpful to add a date-selector drop down to reset / view accurate spends (I think this is just the global/spend/report api )

7. General Improvements

• Return x-litellm-call-id in failure responses from litellm gateway
• [Feature]: litellm.supports_reasoning and drop_params working with reasoning #8765
• [Feature]: Allow config litellm.api_key and litellm.api_base by environment variables #9075
• [Bug]: router cache MOVED Error with Redis Cluster #8878

8. Helm Improvements

• https://github.com/BerriAI/litellm/issues?page=3

[CakeCrusher]

@krrishdholakia
#8977 solved by #9242
#8855 solved by #6376