Greptile Overview

Summary

This PR adds parameterization for the agent execution model in eval runners. Previously, agent evaluations ran without an execution model set. Now, non-CUA models get a default execution model (google/gemini-2.5-flash) that can be overridden via the AGENT_EVAL_EXECUTION_MODEL environment variable.

• Added environment variable AGENT_EVAL_EXECUTION_MODEL for execution model configuration
• Set default execution model to google/gemini-2.5-flash for non-CUA models
• CUA models (computer-use-preview and claude) continue to work without execution model

Confidence Score: 4/5

• This PR is safe to merge with minimal risk
• The change is straightforward and well-contained, adding proper fallback behavior for execution models. The logic correctly differentiates between CUA and non-CUA models, and the default value choice is reasonable
• No files require special attention

Important Files Changed

File Analysis

Sequence Diagram

sequenceDiagram
    participant Eval as Eval Runner
    participant Init as initStagehand
    participant Env as Environment
    participant Agent as Stagehand Agent

    Eval->>Init: Call initStagehand with modelName
    Init->>Init: Check if isCUAModel(modelName)
    
    alt CUA Model (computer-use-preview or claude)
        Init->>Agent: Create agentConfig with model & provider
        Note right of Init: executionModel not set for CUA models
    else Non-CUA Model
        Init->>Env: Check AGENT_EVAL_EXECUTION_MODEL
        Env-->>Init: Return env var or undefined
        Init->>Init: Set executionModel = env ?? "google/gemini-2.5-flash"
        Init->>Agent: Create agentConfig with model & executionModel
    end
    
    Init->>Agent: stagehand.agent(agentConfig)
    Agent-->>Eval: Return configured agent

_{1 file reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}