Files endpoints - How to configure the config.yaml

[Jacobh2]

Hey all!

I'm trying to understand how to setup the config.yaml file so that all the file related endpoints work as expected.

What I would wish is that users of the litellm setup that I have can use the OpenAI python client, set the base_url and api_key to the LiteLLM and simply use the methods as-is. Is it possible to do this so that users doesn't have to include the extra_body={"custom_llm_provider": "azure"} parameter?

Secondly: I am able to make the upload of files work when I set the config to the following:

files_settings:
  - custom_llm_provider: "azure"
    api_base: https://<my-subdomain>.openai.azure.com/
    api_key: <my-token>
    api_version: <version>

With this config, I'm able to run

curl http://localhost:4000/v1/files \
    -H "Authorization: Bearer sk-1234" \
    -F purpose="batch" \
    -F custom_llm_provider="azure" \
    -F file="@batch_test_data.jsonl"

and I get a proper output:

{
    "id": "file-a5dce1ffdf314938aa621a9c678b77ab",
    "bytes": 818,
    "created_at": 1737550680,
    "filename": "batch_test_data.jsonl",
    "object": "file",
    "purpose": "batch",
    "status": "pending",
    "status_details": null
}

but I have to provide the extra -F custom_llm_provider="azure" which I'd like to not have to.

Also, running

curl http://localhost:4000/v1/files \
    -H "Authorization: Bearer sk-1234" \
    -H "custom_llm_provider: azure"

or

curl http://localhost:4000/v1/files \
    -H "Authorization: Bearer sk-1234"

both give the same error:

{
    "error": {
        "message": "The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable",
        "type": "None",
        "param": "None",
        "code": "500"
    }
}

with the same above config: So it works to upload files, but not to fetch files. What is it I am missing? I have tried to find documentation on this but unable to find any.

Thank you! 🙌

[Jacobh2]

Looking through the code of LiteLLM, how the list files endpoint work, it still seems to try to fetch the custom_llm_provider from the request body as a fallback, which IMO is a bit strange consider it is a GET request so no body is to be expected to exist.

[racdev]

adding the following query parameter to the get request works for me (v1.60.0): provider=azure

[Jacobh2]

I would like to not have to ask my users to "know" that the files are ultimately uploaded to azure, it should be completely cloud agnostic for them, they should simply just use the api as if it was the openai one.

This is what I'd like to understand how I can configure.

[racdev]

Oh I agree, being able to set something like "default_custom_llm_provider" in the config would be great and make the experience seemless for users, but currently that's not possible as far as I can tell from looking through the code (unless you are using OpenAI which is kind of set as the default as far as I can tell). Overall I think the files/assistants/batch APIs etc aren't as well developed as the rest of the stack (e.g. no vector store APIs yet), but hopefully these will get a bit more attention.

[racdev]

Issue #8712 may be relevant here