Support for Batch Processing with Batch API on LiteLLM?

[rodrigobdz]

Does LiteLLM support Batch Processing?

Azure and OpenAI support Batch Processing with Batch API to reduce inference costs by 50%.

It is unclear from the LiteLLM docs whether this is supported with Batching Completion or whether it is a different feature only with a similar name.

I searched the issues and discussions, but given the naming confusion (and the similar naming), my search did not yield meaningful results.

[rodrigobdz]

Related https://github.com/orgs/langfuse/discussions/5831.

[krrishdholakia]

Hi @rodrigobdz the Batch API is supported - https://docs.litellm.ai/docs/batches

[rodrigobdz]

Thanks, @krrishdholakia! I need to test this, but that answers my question for now

[skucherlapati]

@krrishdholakia Am using python sdk for vertexai and I get the following error:

BadRequestError: litellm.BadRequestError: LiteLLM doesn't support vertex_ai for 'create_batch'. Only 'openai' is supported.

Code snippet:

retrieved_batch = await litellm.aretrieve_batch(
    batch_id=id, custom_llm_provider="vertex_ai"
)

Is this only supported with the proxy?

[krrishdholakia]

testing - will follow up @skucherlapati

[krrishdholakia]

Hi @skucherlapati wasn't able to get to this today. Will follow up tomorrow

cc: @ishaan-jaff if you're able to get to this before me

[joetam]

@krrishdholakia @ishaan-jaff will gemini support be added?

[achpalaman]

Hi @krrishdholakia, is batch processing also supported on Anthropic? If not, are there any plans to support it?

Thank you!

[timbmg]

Is there support for using batch processing in the Python SDK?