Guide agent scaffolding by alex-w-99 · Pull Request #86 · VectorlyApp/bluebox

alex-w-99 · 2026-01-15T19:34:15Z

Guide agent scaffolding

Adds a conversational AI agent that helps users define and create web automation routines through natural language interaction.

What's new

Guide agent (web_hacker/agents/guide_agent/)

Conversational assistant that gathers task requirements (what to automate, parameters, expected output, target website) and initiates routine discovery
Tool-calling support with user confirmation flow before execution
Streaming response support for real-time output

Unified LLM client (web_hacker/llms/)

Facade client supporting both OpenAI (GPT-5 series) and Anthropic (Claude 4.5 series) models
Vendor-specific implementations with async support
Tool/function registration via register_tool_from_function() with automatic schema generation from type hints and docstrings
Structured output support using Pydantic models

Chat data models (web_hacker/data_models/chat.py)

Chat and ChatThread models for conversation persistence
PendingToolInvocation for tracking tool calls awaiting user confirmation
LLMChatResponse and streaming response types

CLI and scripts

Interactive terminal CLI (web_hacker/scripts/run_guide_agent.py) for local testing
Root-level script (scripts/run_guide_agent.py) as alternate entry point

Other changes

Added ResourceBase for consistent ID and timestamp handling across models
Added UnknownToolError exception
Fixed deprecated datetime.utcnow() usage with timezone-aware alternative
Added unit tests for tool utility functions

Next steps

Pass the guide agent the CDP captures dir/path
Implement the "invoke discovery agent" tool, used by the guide agent

- Add StartRoutineDiscoveryJobCreationParams Pydantic model for tool schema - Add data_models/guide_agent/ with conversation state and message types - Add data_models/websockets/ with base WS types and guide-specific commands/responses - Update GuideAgent with callback pattern, tool confirmation flow, state management - Business logic stubs marked with NotImplementedError for subsequent PR Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move all WebSocket types (base, browser, guide) into one consolidated websockets.py file. Also move test_websockets.py from servers repo. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replace Pydantic model + constants with a simple function stub that colleague will implement. Guide agent now uses register_tool_from_function and calls the function directly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add tool_utils.py with extract_description_from_docstring and generate_parameters_schema for converting Python functions to LLM tool definitions using pydantic TypeAdapter - Add register_tool_from_function method to LLMClient that extracts name, description, and parameters schema from a function - Add unit tests for tool_utils Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Merge GuideWebSocketClientCommandType into WebSocketClientCommandType - Merge ParsedGuideWebSocketClientCommand into ParsedWebSocketClientCommand - Remove Guide- prefix from response types (WebSocketMessageResponse, etc.) - Consolidate response type enums (MESSAGE, STATE, TOOL_INVOCATION_RESULT) - Add tests for all previously untested models and commands - Increase test coverage from ~50% to 100% of websockets module Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…LLM API - Add Chat and ChatThread models extending ResourceBase with bidirectional linking - Rename duplicate ChatMessage to EmittedChatMessage for callback messages - Add LLMToolCall and LLMChatResponse models for tool calling support - Implement GuideAgent with conversation logic, persistence callbacks, and self-aware system prompt for web automation routine creation - Update all LLM client methods to accept messages array instead of single prompt (get_text_sync/async, get_structured_response_sync/async, chat_sync) - Add run_guide_agent.py terminal chat script Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replaces stub script with full terminal interface featuring ANSI colors, ASCII banner, tool invocation confirmation flow, and conversation commands. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update welcome message to describe CDP capture analysis workflow - Add links to Vectorly docs and console - Change banner color to purple - Fix OpenAI client to use max_completion_tokens for GPT-5 models Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

GPT-5 models only support temperature=1 (default), so we omit the parameter entirely to avoid API errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add chat_stream_sync method to abstract, OpenAI, and Anthropic clients - Add stream_chunk_callable parameter to GuideAgent - Update terminal CLI to print chunks as they arrive for typewriter effect Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add STREAM_CHUNK and STREAM_END to WebSocketStreamResponseType - Add WebSocketStreamChunkResponse for text deltas during streaming - Add WebSocketStreamEndResponse with full accumulated content - Update WebSocketServerResponse union to include new types Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update tests to use thread_id instead of guide_chat_id to match the WebSocketStateResponse model change. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…into llms/ subpackage - Remove Chat and ChatThread (ResourceBase-dependent) from this repo - Add ChatLite and ChatThreadLite as lightweight replacements - Move data models to web_hacker/data_models/llms/ subpackage: - vendors.py: LLM vendor enums and model types - interaction.py: chat/conversation types for agent communication - Update all imports across codebase to use new submodule paths - Delete chat.py (models moved to servers repo) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

dimavrem22 · 2026-01-16T18:08:49Z

web_hacker/data_models/llms/vendors.py

+    """OpenAI models."""
+    GPT_5_2 = "gpt-5.2"
+    GPT_5_MINI = "gpt-5-mini"
+    GPT_5_NANO = "gpt-5-nano"


lets add 5 and 5.1

Okay, updated in commit 27a3866.

alex-w-99 · 2026-01-16T21:54:11Z

Update:

ChatLite → Chat
ChatThreadLite → ChatThread

Edit: Updated in commit a23b99d.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Include message IDs in emitted chat responses so WebSocket clients can track and reference individual messages. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update callback signatures from Callable[[T], None] to Callable[[T], T] so the persistence layer can assign IDs and return them to GuideAgent. This allows servers to use ResourceBase-generated IDs while keeping web_hacker's models decoupled from ResourceBase. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

dimavrem22 · 2026-01-17T20:41:32Z

web_hacker/data_models/llms/interaction.py

+class ChatRole(StrEnum):
+    """
+    Role in a chat message.
+    """
+    USER = "user"
+    ASSISTANT = "assistant"  # AI
+    SYSTEM = "system"
+    TOOL = "tool"


can we just do Role?

alex-w-99 and others added 30 commits January 15, 2026 19:22

llms

e372981

agents/

35d78c8

ckpt

ff5d99e

ckpt

0deecc4

pydoc

6252a5c

Add UnknownToolError exception and use in guide agent

52e4146

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

ckpt

daf3603

Fix imports to use fully qualified web_hacker module paths

d0a71d8

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Consolidate websocket data models into single file

a4d3f95

Move all WebSocket types (base, browser, guide) into one consolidated websockets.py file. Also move test_websockets.py from servers repo. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix section divider formatting in websockets.py

2103d2f

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

ckpt

e701f36

Replace deprecated datetime.utcnow with timezone-aware alternative

6418b86

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add TODO for CDP assets path, use tz= keyword arg for datetime

e543fef

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Refactor guide_agent_tools to be a callable function

1b33141

Replace Pydantic model + constants with a simple function stub that colleague will implement. Guide agent now uses register_tool_from_function and calls the function directly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

shell script

a68438f

try to simplify

803250f

fix imports

cdf72a3

resource base

90a9c43

Add interactive terminal CLI for guide agent

a74bb29

Replaces stub script with full terminal interface featuring ANSI colors, ASCII banner, tool invocation confirmation flow, and conversation commands. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Remove temperature parameter from OpenAI client for GPT-5 compatibility

6c3db1e

GPT-5 models only support temperature=1 (default), so we omit the parameter entirely to avoid API errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

thread id

f045ddf

Fix websocket tests for thread_id rename

02b5c12

Update tests to use thread_id instead of guide_chat_id to match the WebSocketStateResponse model change. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

mv WS

ba0953b

alex-w-99 marked this pull request as ready for review January 16, 2026 06:08

alex-w-99 requested review from dimavrem22 and rayruizhiliao as code owners January 16, 2026 06:08

alex-w-99 and others added 3 commits January 16, 2026 16:48

mv in ResourceBase test

a4deca4

tweak

645f83c

dimavrem22 reviewed Jan 16, 2026

View reviewed changes

alex-w-99 changed the base branch from main to beta January 16, 2026 18:16

alex-w-99 added 2 commits January 16, 2026 18:42

Dima comment 1

27a3866

default factory UUIDv4

fb9c7c3

alex-w-99 marked this pull request as draft January 16, 2026 21:52

alex-w-99 and others added 5 commits January 16, 2026 22:05

Rename ChatLite to Chat and ChatThreadLite to ChatThread

a23b99d

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add chat_id and chat_thread_id to EmittedChatMessage

24f43a7

Include message IDs in emitted chat responses so WebSocket clients can track and reference individual messages. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

OpenAI for now

b4dd1b1

update conftest to mock OpenAI client

ccfd933

alex-w-99 requested a review from dimavrem22 January 17, 2026 20:20

alex-w-99 marked this pull request as ready for review January 17, 2026 20:21

dimavrem22 reviewed Jan 17, 2026

View reviewed changes

dimavrem22 approved these changes Jan 17, 2026

View reviewed changes

dimavrem22 merged commit 4e6586d into beta Jan 18, 2026
7 checks passed

alex-w-99 deleted the guide-agent-scaffolding branch January 18, 2026 18:09

alex-w-99 mentioned this pull request Jan 19, 2026

Beta v2026.01.22 #92

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Guide agent scaffolding#86

Guide agent scaffolding#86
dimavrem22 merged 40 commits intobetafrom
guide-agent-scaffolding

alex-w-99 commented Jan 15, 2026 •

edited

Loading

Uh oh!

dimavrem22 Jan 16, 2026

Uh oh!

alex-w-99 Jan 16, 2026

Uh oh!

alex-w-99 commented Jan 16, 2026 •

edited

Loading

Uh oh!

dimavrem22 Jan 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

alex-w-99 commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!