Merge pull request #55 from Annotation-Garden/develop

Release v0.6.2-alpha: BYOK model selection and provider auto-clear

Author: neuromechanist

docs/manual-test-byok-model-selection.md

@@ -0,0 +1,162 @@
+# Manual Test Plan: BYOK Model Selection
+
+This document describes how to manually test the BYOK model/provider/temperature selection feature after PR #54 is merged.
+
+## Prerequisites
+
+1. Have a valid OpenRouter API key (get one at https://openrouter.ai)
+2. Have the HEDit CLI installed: `pip install hedit` or `pip install -e .`
+3. Know the API endpoint (e.g., `https://api.annotation.garden/hedit` or local `http://localhost:38427`)
+
+## Test 1: Request Body Model Selection (API)
+
+Test that model settings in the request body are used.
+
+```bash
+# Set your API key
+export OPENROUTER_KEY="sk-or-v1-your-key-here"
+
+# Test with custom model in request body
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -d '{
+    "description": "A red circle appears on the left side of the screen",
+    "model": "openai/gpt-4o-mini",
+    "temperature": 0.3
+  }'
+```
+
+**Expected**: Should use `gpt-4o-mini` model (verify in OpenRouter dashboard usage logs).
+
+## Test 2: Header-Based Model Selection (API)
+
+Test that model settings in headers are used as fallback.
+
+```bash
+# Test with custom model in headers
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -H "X-OpenRouter-Model: anthropic/claude-3-haiku-20240307" \
+  -H "X-OpenRouter-Temperature: 0.1" \
+  -d '{
+    "description": "A blue square fades in at the center"
+  }'
+```
+
+**Expected**: Should use `claude-3-haiku` model.
+
+## Test 3: Request Body Overrides Headers
+
+Test that request body has higher priority than headers.
+
+```bash
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -H "X-OpenRouter-Model: anthropic/claude-3-haiku-20240307" \
+  -d '{
+    "description": "A green triangle rotates",
+    "model": "openai/gpt-4o-mini"
+  }'
+```
+
+**Expected**: Should use `gpt-4o-mini` (body), NOT `claude-3-haiku` (header).
+
+## Test 4: CLI Model Selection
+
+Test the CLI with `--model` flag.
+
+```bash
+# Initialize with your key
+hedit init --api-key $OPENROUTER_KEY
+
+# Test with custom model
+hedit annotate "A loud beep sound plays" --model openai/gpt-4o-mini --temperature 0.2
+```
+
+**Expected**: Should use specified model.
+
+## Test 5: Image Annotation with Vision Model
+
+Test image annotation with custom vision model.
+
+```bash
+# Create a test image or use any image file
+curl -X POST https://api.annotation.garden/hedit/annotate-from-image \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -d "{
+    \"image\": \"data:image/png;base64,$(base64 -i test_image.png)\",
+    \"model\": \"openai/gpt-4o\",
+    \"vision_model\": \"openai/gpt-4o\",
+    \"temperature\": 0.3
+  }"
+```
+
+**Expected**: Should use specified vision model for description.
+
+## Test 6: Server Default Fallback
+
+Test that without BYOK, server uses its defaults (this should already work).
+
+```bash
+# Using server API key (if you have one)
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-server-api-key" \
+  -d '{
+    "description": "A warning message appears"
+  }'
+```
+
+**Expected**: Should use server's default model from environment variables.
+
+## Test 7: Temperature Range Validation
+
+Test that temperature validation works.
+
+```bash
+# Invalid temperature (should fail validation)
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -d '{
+    "description": "Test",
+    "temperature": 1.5
+  }'
+```
+
+**Expected**: Should return 422 validation error (temperature must be 0.0-1.0).
+
+## Test 8: Provider Selection
+
+Test provider preference (e.g., Cerebras for fast inference).
+
+```bash
+curl -X POST https://api.annotation.garden/hedit/annotate \
+  -H "Content-Type: application/json" \
+  -H "X-OpenRouter-Key: $OPENROUTER_KEY" \
+  -d '{
+    "description": "A participant presses a button",
+    "model": "openai/gpt-oss-120b",
+    "provider": "Cerebras"
+  }'
+```
+
+**Expected**: Should route through Cerebras provider (faster inference).
+
+## Verification
+
+For all tests, verify:
+1. The request succeeds (HTTP 200)
+2. Valid HED annotation is returned
+3. Check OpenRouter dashboard to confirm which model was used
+4. Response time may vary by model/provider
+
+## Notes
+
+- The model parameter overrides ALL agents (annotation, evaluation, assessment)
+- Per-agent model selection is not yet supported via API (future enhancement)
+- Invalid model names will result in OpenRouter errors

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "hedit"
-version = "0.6.1-alpha2"
+version = "0.6.2-alpha"
 description = "Multi-agent system for HED annotation generation and validation"
 readme = "README.md"
 requires-python = ">=3.12"

src/api/main.py

@@ -48,30 +48,60 @@
 _byok_config: dict = {}
 
 
-def create_byok_workflow(openrouter_key: str) -> HedAnnotationWorkflow:
+def create_byok_workflow(
+    openrouter_key: str,
+    model: str | None = None,
+    provider: str | None = None,
+    temperature: float | None = None,
+) -> HedAnnotationWorkflow:
     """Create a workflow instance using the user's OpenRouter key (BYOK mode).
 
     Args:
         openrouter_key: User's OpenRouter API key
+        model: Override model for all agents (uses server default if None)
+        provider: Override provider preference (uses server default if None)
+        temperature: Override LLM temperature (uses server default if None)
 
     Returns:
-        Configured HedAnnotationWorkflow using the user's key
+        Configured HedAnnotationWorkflow using the user's key and model settings
     """
     global _byok_config
 
     # Get configuration (cached from server startup)
-    llm_temperature = _byok_config.get("temperature", 0.1)
-    provider_preference = _byok_config.get("provider_preference")
     schema_dir = _byok_config.get("schema_dir")
     validator_path = _byok_config.get("validator_path")
     use_js_validator = _byok_config.get("use_js_validator", True)
 
-    # Get model configuration from headers or use defaults
-    annotation_model = get_model_name(os.getenv("ANNOTATION_MODEL", "openai/gpt-oss-120b"))
-    evaluation_model = get_model_name(os.getenv("EVALUATION_MODEL", "qwen/qwen3-235b-a22b-2507"))
-    assessment_model = get_model_name(os.getenv("ASSESSMENT_MODEL", "openai/gpt-oss-120b"))
+    # Use user-provided settings or fall back to server defaults
+    llm_temperature = (
+        temperature if temperature is not None else _byok_config.get("temperature", 0.1)
+    )
+
+    # Provider logic:
+    # - If user specifies a custom model, clear provider (Cerebras only works with default models)
+    # - Unless user also explicitly specifies a provider
+    if provider is not None:
+        # User explicitly set provider (could be empty string to clear it)
+        provider_preference = provider if provider else None
+    elif model is not None:
+        # User specified custom model but no provider → clear provider
+        # (Cerebras only works with default models)
+        provider_preference = None
+    else:
+        # No custom model or provider → use server defaults
+        provider_preference = _byok_config.get("provider_preference")
+
+    # Get model configuration: user override > server env var > default
+    default_annotation_model = os.getenv("ANNOTATION_MODEL", "openai/gpt-oss-120b")
+    default_evaluation_model = os.getenv("EVALUATION_MODEL", "qwen/qwen3-235b-a22b-2507")
+    default_assessment_model = os.getenv("ASSESSMENT_MODEL", "openai/gpt-oss-120b")
 
-    # Create LLMs with user's key
+    # If user provides a model, use it for all agents (default override)
+    annotation_model = get_model_name(model if model else default_annotation_model)
+    evaluation_model = get_model_name(model if model else default_evaluation_model)
+    assessment_model = get_model_name(model if model else default_assessment_model)
+
+    # Create LLMs with user's key and settings
     annotation_llm = create_openrouter_llm(
         model=annotation_model,
         api_key=openrouter_key,
@@ -102,23 +132,46 @@ def create_byok_workflow(openrouter_key: str) -> HedAnnotationWorkflow:
     )
 
 
-def create_byok_vision_agent(openrouter_key: str) -> VisionAgent:
+def create_byok_vision_agent(
+    openrouter_key: str,
+    vision_model: str | None = None,
+    provider: str | None = None,
+    temperature: float | None = None,
+) -> VisionAgent:
     """Create a vision agent instance using the user's OpenRouter key (BYOK mode).
 
     Args:
         openrouter_key: User's OpenRouter API key
+        vision_model: Override vision model (uses server default if None)
+        provider: Override provider preference (uses server default if None)
+        temperature: Override temperature (uses 0.3 default if None)
 
     Returns:
-        Configured VisionAgent using the user's key
+        Configured VisionAgent using the user's key and model settings
     """
-    vision_model = os.getenv("VISION_MODEL", "qwen/qwen3-vl-30b-a3b-instruct")
-    vision_provider = os.getenv("VISION_PROVIDER", "deepinfra/fp8")
+    # Use user-provided settings or fall back to server defaults
+    default_vision_model = os.getenv("VISION_MODEL", "qwen/qwen3-vl-30b-a3b-instruct")
+    default_vision_provider = os.getenv("VISION_PROVIDER", "deepinfra/fp8")
+
+    actual_model = vision_model if vision_model else default_vision_model
+    actual_temperature = temperature if temperature is not None else 0.3
+
+    # Provider logic:
+    # - If user specifies a custom vision model, clear provider
+    # - Unless user also explicitly specifies a provider
+    if provider is not None:
+        actual_provider = provider if provider else None
+    elif vision_model is not None:
+        # Custom vision model → clear provider
+        actual_provider = None
+    else:
+        actual_provider = default_vision_provider
 
     vision_llm = create_openrouter_llm(
-        model=vision_model,
+        model=actual_model,
         api_key=openrouter_key,
-        temperature=0.3,
-        provider=vision_provider,
+        temperature=actual_temperature,
+        provider=actual_provider,
     )
 
     return VisionAgent(llm=vision_llm)
@@ -365,6 +418,10 @@ def get_default_path(docker_path: str, local_path: str) -> str:
         "X-Requested-With",
         "X-API-Key",
         "X-OpenRouter-Key",  # BYOK mode
+        "X-OpenRouter-Model",  # BYOK model override
+        "X-OpenRouter-Vision-Model",  # BYOK vision model override
+        "X-OpenRouter-Provider",  # BYOK provider preference
+        "X-OpenRouter-Temperature",  # BYOK temperature override
     ],
     max_age=3600,  # Cache preflight requests for 1 hour
 )
@@ -447,12 +504,29 @@ async def annotate(
     """
     # Determine which workflow to use
     if api_key == "byok":
-        # BYOK mode: Create workflow with user's key
+        # BYOK mode: Create workflow with user's key and model settings
         openrouter_key = req.headers.get("x-openrouter-key")
         if not openrouter_key:
             raise HTTPException(status_code=401, detail="Missing X-OpenRouter-Key header")
+
+        # Get model config: request body > headers > server defaults
+        model = request.model or req.headers.get("x-openrouter-model")
+        provider = request.provider or req.headers.get("x-openrouter-provider")
+        temp_header = req.headers.get("x-openrouter-temperature")
+        temperature = request.temperature
+        if temperature is None and temp_header:
+            try:
+                temperature = float(temp_header)
+            except ValueError:
+                pass  # Invalid header value, use default
+
         try:
-            active_workflow = create_byok_workflow(openrouter_key)
+            active_workflow = create_byok_workflow(
+                openrouter_key,
+                model=model,
+                provider=provider,
+                temperature=temperature,
+            )
         except Exception as e:
             raise HTTPException(
                 status_code=500, detail=f"Failed to initialize BYOK workflow: {str(e)}"
@@ -530,13 +604,36 @@ async def annotate_from_image(
     """
     # Determine which workflow and vision agent to use
     if api_key == "byok":
-        # BYOK mode: Create workflow and vision agent with user's key
+        # BYOK mode: Create workflow and vision agent with user's key and model settings
         openrouter_key = req.headers.get("x-openrouter-key")
         if not openrouter_key:
             raise HTTPException(status_code=401, detail="Missing X-OpenRouter-Key header")
+
+        # Get model config: request body > headers > server defaults
+        model = request.model or req.headers.get("x-openrouter-model")
+        vision_model = request.vision_model or req.headers.get("x-openrouter-vision-model")
+        provider = request.provider or req.headers.get("x-openrouter-provider")
+        temp_header = req.headers.get("x-openrouter-temperature")
+        temperature = request.temperature
+        if temperature is None and temp_header:
+            try:
+                temperature = float(temp_header)
+            except ValueError:
+                pass  # Invalid header value, use default
+
         try:
-            active_workflow = create_byok_workflow(openrouter_key)
-            active_vision_agent = create_byok_vision_agent(openrouter_key)
+            active_workflow = create_byok_workflow(
+                openrouter_key,
+                model=model,
+                provider=provider,
+                temperature=temperature,
+            )
+            active_vision_agent = create_byok_vision_agent(
+                openrouter_key,
+                vision_model=vision_model,
+                provider=provider,
+                temperature=temperature,
+            )
         except Exception as e:
             raise HTTPException(
                 status_code=500, detail=f"Failed to initialize BYOK agents: {str(e)}"