Skip to content

Add OpenRouter Integration for Metrics Evaluation #2626

@Djalal-H

Description

@Djalal-H

I am experiencing a frustrating issue where trying to use OpenRouter models for metrics evaluation fails or falls back to OpenAI defaults (like gpt-4.1), even when configuring it via the CLI (deepeval set-openrouter) or environment variables.

Upon inspecting the source code (specifically the utility functions handling model initialization and validation), I noticed that OpenRouter is entirely missing from the core model routing logic. Specifically:

  1. OpenRouterModel is not imported from deepeval.models.
  2. There is no should_use_openrouter() function checking the environment/settings.
  3. initialize_model() does not include OpenRouter in its fallback/routing tree.
  4. Most importantly, is_native_model() does not include isinstance(model, OpenRouterModel). Because of this, metrics don't recognize OpenRouter as a native model, setting using_native_model = False and triggering unexpected behaviors or OpenAI API key errors.

Describe the solution you'd like
I would like OpenRouterModel to be fully integrated as a first-class citizen in the metrics utility code alongside Gemini, Anthropic, Ollama, etc.

Specifically, the following changes would resolve the issue:

  1. Import OpenRouterModel alongside the other models.
  2. Add a should_use_openrouter() helper checking for USE_OPENROUTER_MODEL flags.
  3. Update initialize_model() to return OpenRouterModel(model=model), True.
  4. Update is_native_model() to include or isinstance(model, OpenRouterModel).

Describe alternatives you've considered
Currently, the only workaround is heavily disguising the OpenRouter API key as an OPENAI_API_KEY and injecting a custom GPTModel instance into every single metric manually in Python code. This defeats the purpose of the CLI configuration and clutters the test files.

Additional context
Based on the provided codebase, here is a quick look at where the gap is happening in is_native_model:

def is_native_model(
    model: Optional[Union[str, DeepEvalBaseLLM]] = None,
) -> bool:
    if (
        isinstance(model, GPTModel)
        or isinstance(model, AnthropicModel)
        or isinstance(model, AzureOpenAIModel)
        or isinstance(model, OllamaModel)
        or isinstance(model, LocalModel)
        or isinstance(model, GeminiModel)
        or isinstance(model, AmazonBedrockModel)
        or isinstance(model, LiteLLMModel)
        or isinstance(model, KimiModel)
        or isinstance(model, GrokModel)
        or isinstance(model, DeepSeekModel)
        # Missing: or isinstance(model, OpenRouterModel)
    ):
        return True
    else:
        return False

Adding it here and in initialize_model() would make the OpenRouter integration work seamlessly!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions