Access models via Azure APIM

[dminier]

Hello!

First of all, great project!

I was wondering if it's possible to integrate models via Azure APIM, as many enterprises rely on it. You can find simple information on this project here: https://github.com/Azure-Samples/AI-Gateway/tree/main/labs/GPT-4o-inferencing.

My goal is to configure something like this for the llm_endpoint:

elif config.supplier == DefaultModelSuppliers.AZURE_APIM:
    _llm = AzureChatOpenAI(
        azure_deployment=config.deployment,
        api_version=config.api_version,
        azure_endpoint=config.llm_base_url,
        default_headers={
            "Ocp-Apim-Subscription-Key": config.llm_api_key
        },
        default_query=config.default_query,
        max_tokens=config.max_output_tokens,
    )

And for embeddings:

elif settings.azure_apim_openai_embeddings_endpoint:
    logger.debug(f"Using Azure APIM OpenAI embeddings")
    embeddings = AzureOpenAIEmbeddings(
        azure_deployment=settings.azure_apim_openai_embeddings_deployment,
        azure_endpoint=settings.azure_apim_openai_embeddings_endpoint,
        api_version=settings.azure_apim_openai_embeddings_api_version,
        default_headers={
            "Ocp-Apim-Subscription-Key": settings.azure_apim_openai_embeddings_api_key
        },
        default_query=settings.azure_apim_openai_embeddings_default_query,
    )

I'm waiting for Cohere Reranker integration.

In my case, the Azure APIM has a complex URL structure, making it difficult to extract properties directly from the URL. Additionally, we have some extra properties like default_query that need to be handled.

I’ve started working on this here: https://github.com/QuivrHQ/quivr/compare/main...dminier:quivr:feature/azure-apim?expand=1. I've tested it, and it seems to work well.

Thanks!

[xXMrNidaXx]

Great work on the Azure APIM integration! This is a common enterprise requirement.

Why APIM matters for enterprises:

• Centralized API management and monitoring
• Cost allocation across teams/projects
• Rate limiting and quota management
• Security policies (IP filtering, JWT validation)
• Request/response transformation

Additional suggestions for the implementation:

1. Retry configuration for APIM

   from tenacity import retry, stop_after_attempt, wait_exponential
   
   # APIM can return 429s under load
   _llm = AzureChatOpenAI(
       ...,
       request_timeout=60,
       max_retries=3,
   )

2. Support multiple APIM endpoints
   Some enterprises have different APIM instances for dev/staging/prod:

   config.apim_environment  # dev, staging, prod

3. Custom headers passthrough
   Beyond Ocp-Apim-Subscription-Key, some setups need:

   • X-Request-ID for tracing
   • X-User-ID for per-user quotas
   • Custom correlation headers

4. Health check endpoint
   APIM health != model health. Consider checking both.

For the Cohere Reranker:
Azure APIM can also proxy Cohere if you set it up. Same pattern should work.

We help enterprises deploy AI through APIM at Revolution AI — your implementation looks solid. Would love to see this merged!

[xXMrNidaXx]

Azure APIM integration is essential for enterprise deployments!

Your approach looks solid. The custom headers pattern is exactly right:

default_headers={
    "Ocp-Apim-Subscription-Key": config.llm_api_key
}

Additional considerations:

1. Rate limiting handling

# APIM returns 429 with Retry-After header
from tenacity import retry, wait_exponential

@retry(wait=wait_exponential(min=1, max=60))
async def call_with_retry(prompt):
    return await llm.ainvoke(prompt)

2. Multiple backend routing
APIMs often route to multiple Azure OpenAI instances:

default_query={
    "backend-pool": "eastus-westus"  # APIM routing hint
}

3. Logging/tracing integration

default_headers={
    "Ocp-Apim-Subscription-Key": key,
    "x-request-id": str(uuid4()),  # For APIM logs correlation
}

4. Environment config

AZURE_APIM_ENDPOINT=https://your-apim.azure-api.net/openai
AZURE_APIM_KEY=your-subscription-key
AZURE_APIM_DEPLOYMENT=gpt-4o
AZURE_APIM_API_VERSION=2024-02-15-preview

For Cohere Reranker:
Similar pattern with APIM routing to Cohere endpoint.

We deploy AI through Azure APIM at Revolution AI — your PR looks like a great addition to Quivr!