how to enable context caching in vertex ai models within LiteLLM Router ?

[AbhishekRP2002]

here is my model list:

model_list_v2 = [
    # {
    #     "model_name": "sprouts-llm",
    #     "litellm_params": {
    #         "model": "vertex_ai/claude-3-5-sonnet@20240620",
    #         "temperature": 0,
    #         "timeout": 15,
    #         "vertex_project": "mlsprouts",
    #         "vertex_location": "us-east5",
    #         "custom_llm_provider": "vertex_ai",
    #         "tool_choice": "required",
    #     },
    #     "model_info": {"base_model": "vertex_ai/claude-3-5-sonnet@20240620"},
    # },
    {
        "model_name": "sprouts-llm",
        "litellm_params": {
            "model": "vertex_ai/gemini-1.5-pro-001",
            "temperature": 0,
            "timeout": 15,
            "vertex_project": "mlsprouts",
            "vertex_location": "us-central1",
            "tool_choice": "required",
            "custom_llm_provider": "vertex_ai",
        },
        "model_info": {"base_model": "gemini-1.5-pro-001"},
    },
    {
        "model_name": "sprouts-llm",
        "litellm_params": {
            "model": "vertex_ai/gemini-1.5-pro-002",
            "temperature": 0,
            "timeout": 15,
            "vertex_project": "mlsprouts",
            "vertex_location": "us-central1",
            "tool_choice": "required",
            "custom_llm_provider": "vertex_ai",
        },
        "model_info": {"base_model": "gemini-1.5-pro-002"},
    },
    {
        "model_name": "sprouts-llm",
        "litellm_params": {
            "model": "vertex_ai/gemini-1.5-flash-002",
            "temperature": 0,
            "timeout": 15,
            "vertex_project": "mlsprouts",
            "vertex_location": "us-central1",
            "tool_choice": "required",
            "custom_llm_provider": "vertex_ai",
        },
        "model_info": {"base_model": "gemini-1.5-flash-002"},
    },
    #  working - for azure/gpt-4o pass the tool_choice param here in the litellm_params
    {
        "model_name": "azure-gpt-4o",
        "litellm_params": {
            "model": "azure/gpt-4o-global-standard",
            "api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
            "api_base": os.environ.get("AZURE_OPENAI_ENDPOINT"),
            "api_version": "2024-08-01-preview",
            "tool_choice": {"type": "function", "function": {"name": "AISignal"}},
            "custom_llm_provider": "azure",
            "temperature": 0,
            "timeout": 15,
        },
        "model_info": {"base_model": "azure/global-standard/gpt-4o-2024-08-06"},
    },
    # working - same as above
    {
        "model_name": "azure-gpt-4o-mini",
        "litellm_params": {
            "model": "azure/gpt-4o-mini-global-standard",
            "api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
            "api_base": os.environ.get("AZURE_OPENAI_ENDPOINT"),
            "api_version": "2024-08-01-preview",
            "tool_choice": {"type": "function", "function": {"name": "AISignal"}},
            "custom_llm_provider": "azure",
            "temperature": 0,
            "timeout": 15,
        },
        "model_info": {"base_model": "azure/global-standard/gpt-4o-mini"},
    },
]

i am using LiteLLM router from LangChain, shown as follows:

litellm_router = Router(
            model_list=model_list_v2,
            enable_pre_call_checks=True,
            allowed_fails=1,
            cooldown_time=30,
            routing_strategy="usage-based-routing-v2",
            num_retries=3,
            fallbacks=[{"sprouts-llm": ["azure-gpt-4o", "azure-gpt-4o-mini"]}],
            content_policy_fallbacks=[
                {"sprouts-llm": ["azure-gpt-4o", "azure-gpt-4o-mini"]}
            ],
            set_verbose=True,
            debug_level="INFO",
        )
        model = ChatLiteLLMRouter(router=litellm_router)

i want to understand how i can enable context caching for gemini family models in this ?
any help would be really appreciated
cc: @krrishdholakia

[sammcj]

See also #6898

[AbhishekRP2002]

Thanks for responding, will take a look.