How do you pass non-OpenAI provider specific parameters from client when using Proxy LLM?

[hnykda]

When you use Proxy LLM, you can use openai default python client such as:

import openai
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

however, create has a specific set of arguments (no kwargs, for example), and therefore I don't see a way how we can pass provider specific parameters to some of the providers as one can do with barebone litellm python SDK, documented e.g. here.

Am I missing something?

[hnykda]

Oh, I found this:

so it's probably fine. We just shouldn't use the builtin openai client, but raw requests. https://docs.litellm.ai/docs/completion/provider_specific_params#proxy-usage

[ishaan-jaff]

no, you can pass this in extra_body when using the OpenAI python SDK too @hnykda

[hnykda]

Aha! I see. Is that documented anywhere?

[hnykda]

Also, how do you do it in practice?

create(model, messages, extra_body=kwargs)? is it actually safe to rely on openai create method not doing anything weird behind the scenes?

Additionally, I am not sure if I trust openai client to treat the response of e.g. anthropic citations untouched. Do you?