BUG: nano-gpt provider naming conflicts with litellm integration - only custom non-standard naming works

[Gerkinfeltser]

Issue Summary

When configuring the nano-gpt provider through LLM-API-Key-Proxy with litellm, standard naming patterns fail. Only using a custom name that avoids "nano" and "gpt" combinations works reliably.

Test Configurations & Results

Attempt 1: NANOGPT (non-custom method)

• .env: NANOGPT (added via "add api key", not custom)
• opencode.jsonc model names tried:
  • nano-gpt (per litellm docs: https://docs.litellm.ai/docs/providers/nano-gpt)
    • Error: "An unexpected error occurred during the stream: 'nano-gpt'"
  • nanogpt
    • Error: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=nanogpt/deepseek-v3-0324\n Pass model as E.g. For 'Huggingface' inference endpoints pass in \completion(model='huggingface/starcacker',..)` Learn more: https://docs.litellm.ai/docs/providers`
  • nano_gpt
    • Error: "An unexpected error occurred during the stream: 'nano_gpt'"

Attempt 2: NANO_GPT (custom OpenAI-compatible)

• .env: NANO_GPT (added as custom OpenAI-compatible)
• opencode.jsonc model names tried:
  • nano-gpt, nano_gpt
    • Result: Initially worked, then became flakey (same litellm BadRequestError as above)

Working Solution

• .env: NG (custom OpenAI-compatible)
• opencode.jsonc: ng/<model> (e.g., ng/deepseek-v3-0324)
• Result: Works reliably

Expected Behavior

Per litellm documentation, using nano-gpt with NANOGPT or NANO_GPT should work correctly without requiring non-standard naming.

Hypothesis

litellm appears to intercept names containing "nano" and "gpt" in combination, attempting to route them through a built-in nano-gpt provider handler instead of treating them as custom OpenAI-compatible endpoints.

[mirrobot-agent]

@Gerkinfeltser Thank you for submitting this issue. I am now beginning my analysis and will report back shortly.

[b3nw]

The actual problem is a naming normalization failure in the proxy's environment variable lookup logic.
Exact Root Cause

In src/rotator_library/client.py:491:

api_base_env = f"{provider.upper()}_API_BASE"
if os.getenv(api_base_env):

The issue:

1. Users set environment variables following convention: NANO_GPT_API_BASE (underscores)
2. litellm documentation uses nano-gpt format (hyphens) in model names
3. When requests come in with nano-gpt/model-name, the code looks for NANO-GPT_API_BASE
4. Lookup fails → provider not detected as custom OpenAI-compatible → litellm receives model without required api_base parameter

Proposed Fix - Exact Lines

File: src/rotator_library/client.py

Current code (lines 485-498):
485|         provider = model.split("/")[0]
486| 
487|         # Handle custom OpenAI-compatible providers
488|         # Check if this is a custom provider by looking for API_BASE environment variable
489|         import os
490| 
491|         api_base_env = f"{provider.upper()}_API_BASE"
492|         if os.getenv(api_base_env):
493|             # For custom providers, tell LiteLLM to use openai provider with custom model name
494|             # This preserves original model name in logs but converts for LiteLLM
495|             kwargs = kwargs.copy()  # Don't modify original
496|             kwargs["model"] = f"openai/{model.split('/', 1)[1]}"
497|             kwargs["api_base"] = os.getenv(api_base_env).rstrip("/")
498|             kwargs["custom_llm_provider"] = "openai"

Replacement (lines 485-500):
485|         provider = model.split("/")[0]
486| 
487|         # Handle custom OpenAI-compatible providers
488|         # Check if this is a custom provider by looking for API_BASE environment variable
489|         import os
490| 
491|         # Normalize provider name: replace hyphens with underscores for env var lookup
492|         # This handles cases where users send "nano-gpt" but have "NANO_GPT_API_BASE"
493|         normalized_provider = provider.upper().replace("-", "_")
494|         api_base_env = f"{normalized_provider}_API_BASE"
495|         if os.getenv(api_base_env):
496|             # For custom providers, tell LiteLLM to use openai provider with custom model name
497|             # This preserves original model name in logs but converts for LiteLLM
498|             kwargs = kwargs.copy()  # Don't modify original
499|             kwargs["model"] = f"openai/{model.split('/', 1)[1]}"
500|             kwargs["api_base"] = os.getenv(api_base_env).rstrip("/")
501|             kwargs["custom_llm_provider"] = "openai"

Summary of changes:

• Line 491-493: Add normalization logic (3 new lines)
• Line 494: Changed from f"{provider.upper()}_API_BASE" to f"{normalized_provider}_API_BASE"
• Lines 495-501: Existing code unchanged but shifted down by 2 lines

This fix adds 3 lines total and changes 1 line. The core change is replacing provider.upper() with provider.upper().replace("-", "_") stored in a normalized_provider variable.

[Mirrowel]

@mirrobot-agent check this claim and issue against the dev branch. would this be the case, or are dev branch changes solving this?

[mirrobot-agent]

@Mirrowel, I'm checking the claim and the state of the dev branch regarding the nano-gpt provider naming issue. I'll compare the current implementation in dev with the proposed fix and report back.

[mirrobot-agent]

@Mirrowel, I have completed my investigation on the dev branch.

Conclusion: The claim made by @b3nw is correct, and the current changes in the dev branch do not solve this issue.

Analysis

The root cause is a naming normalization failure when mapping provider names (from model IDs) to environment variables.

1. Mismatch in Routing: In src/rotator_library/client.py, the provider is extracted from the model string (e.g., nano-gpt from nano-gpt/deepseek-v3). The code then looks up environment variables using f"{provider.upper()}_CUSTOM_API_BASE". For nano-gpt, this results in NANO-GPT_CUSTOM_API_BASE. Hyphens are invalid in environment variable names in many environments, and users naturally use underscores (NANO_GPT_...).
2. Mismatch in Discovery: When the proxy starts, it discovers API keys from the environment. NANO_GPT_API_KEY is registered under the provider nano_gpt (underscores preserved or converted to lowercase). However, when a request comes in for nano-gpt/..., the proxy looks for keys under nano-gpt. Since nano-gpt != nano_gpt, it fails to find any configured keys.
3. LiteLLM Requirement: LiteLLM strictly requires nano-gpt (with a hyphen) in the model string to route to its built-in handler. If the user tries to use nanogpt to match the proxy's current behavior, LiteLLM fails with a BadRequestError.

Findings in dev Branch

I examined src/rotator_library/client.py in the dev branch (around lines 780-797) and verified that it still uses provider.upper() directly without hyphen-to-underscore normalization:

# src/rotator_library/client.py:786 in dev
custom_api_base_env = f"{provider.upper()}_CUSTOM_API_BASE"

Recommendation

The fix proposed by @b3nw is correct and should be applied. Specifically, we should normalize the provider name by replacing hyphens with underscores whenever performing environment variable lookups or internal provider key lookups.

Interestingly, I found that this normalization is already used in other parts of the codebase (e.g., src/rotator_library/providers/provider_cache.py:568), which further supports applying it here for consistency.

I can implement this fix across the relevant files if you'd like.

This investigation was performed by an AI assistant.