Skip to content

Commit 123e04d

Browse files
feat: add feedback collection tool and implicit feedback system
Add meta_collect_tool_feedback tool for collecting user feedback about StackOne tool performance. The tool is automatically included in the toolset and can be invoked by AI agents after user consent. Features: - Feedback collection tool with Pydantic validation - Automatic whitespace trimming and input cleaning - Support for custom base URLs - Integration with StackOneToolSet - OpenAI and LangChain format support Also includes implicit feedback system for automatic behavioral feedback to LangSmith: - Behavior analyzer for detecting quick refinements - Session tracking for tool call history - LangSmith client integration - Configurable via environment variables Breaking changes: - Updated StackOneTool.execute() signature to include options parameter - Updated meta tools execute methods to support options parameter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 370699e commit 123e04d

File tree

17 files changed

+4604
-3125
lines changed

17 files changed

+4604
-3125
lines changed

README.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,51 @@ employee = employee_tool.call(id="employee-id")
7575
employee = employee_tool.execute({"id": "employee-id"})
7676
```
7777

78+
## Implicit Feedback (Beta)
79+
80+
The Python SDK can emit implicit behavioural feedback to LangSmith so you can triage low-quality tool results without manually tagging runs.
81+
82+
### Automatic configuration
83+
84+
Set `LANGSMITH_API_KEY` in your environment and the SDK will initialise the implicit feedback manager on first tool execution. You can optionally fine-tune behaviour with:
85+
86+
- `STACKONE_IMPLICIT_FEEDBACK_ENABLED` (`true`/`false`, defaults to `true` when an API key is present)
87+
- `STACKONE_IMPLICIT_FEEDBACK_PROJECT` to pin a LangSmith project name
88+
- `STACKONE_IMPLICIT_FEEDBACK_TAGS` with a comma-separated list of tags applied to every run
89+
90+
### Manual configuration
91+
92+
If you want custom session or user resolvers, call `configure_implicit_feedback` during start-up:
93+
94+
```python
95+
from stackone_ai import configure_implicit_feedback
96+
97+
configure_implicit_feedback(
98+
api_key="/path/to/langsmith.key",
99+
project_name="stackone-agents",
100+
default_tags=["python-sdk"],
101+
)
102+
```
103+
104+
Providing your own `session_resolver`/`user_resolver` callbacks lets you derive identifiers from the request context before events are sent to LangSmith.
105+
106+
### Attaching session context to tool calls
107+
108+
Both `tool.execute` and `tool.call` accept an `options` keyword that is excluded from the API request but forwarded to the feedback manager:
109+
110+
```python
111+
tool.execute(
112+
{"id": "employee-id"},
113+
options={
114+
"feedback_session_id": "chat-42",
115+
"feedback_user_id": "user-123",
116+
"feedback_metadata": {"conversation_id": "abc"},
117+
},
118+
)
119+
```
120+
121+
When two calls for the same session happen within a few seconds, the SDK emits a `refinement_needed` event, and you can inspect suitability scores directly in LangSmith.
122+
78123
## Integration Examples
79124

80125
<details>
@@ -148,6 +193,31 @@ result = crew.kickoff()
148193

149194
</details>
150195

196+
## Feedback Collection
197+
198+
The SDK includes a feedback collection tool (`meta_collect_tool_feedback`) that allows users to submit feedback about their experience with StackOne tools. This tool is automatically included in the toolset and is designed to be invoked by AI agents after user permission.
199+
200+
```python
201+
from stackone_ai import StackOneToolSet
202+
203+
toolset = StackOneToolSet()
204+
205+
# Get the feedback tool (included with "meta_*" pattern or all tools)
206+
tools = toolset.get_tools("meta_*")
207+
feedback_tool = tools.get_tool("meta_collect_tool_feedback")
208+
209+
# Submit feedback (typically invoked by AI after user consent)
210+
result = feedback_tool.call(
211+
feedback="The HRIS tools are working great! Very fast response times.",
212+
account_id="acc_123456",
213+
tool_names=["hris_list_employees", "hris_get_employee"]
214+
)
215+
```
216+
217+
**Important**: The AI agent should always ask for user permission before submitting feedback:
218+
- "Are you ok with sending feedback to StackOne? The LLM will take care of sending it."
219+
- Only call the tool after the user explicitly agrees.
220+
151221
## Meta Tools (Beta)
152222

153223
Meta tools enable dynamic tool discovery and execution without hardcoding tool names:

stackone_ai/__init__.py

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
11
"""StackOne AI SDK"""
22

3+
from .implicit_feedback import configure_implicit_feedback, get_implicit_feedback_manager
34
from .models import StackOneTool, Tools
45
from .toolset import StackOneToolSet
56

6-
__all__ = ["StackOneToolSet", "StackOneTool", "Tools"]
7+
__all__ = [
8+
"StackOneToolSet",
9+
"StackOneTool",
10+
"Tools",
11+
"configure_implicit_feedback",
12+
"get_implicit_feedback_manager",
13+
]
714
__version__ = "0.3.2"

stackone_ai/feedback/__init__.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
"""Feedback collection tool for StackOne."""
2+
3+
from .tool import create_feedback_tool
4+
5+
__all__ = ["create_feedback_tool"]

stackone_ai/feedback/tool.py

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
"""Feedback collection tool for StackOne."""
2+
3+
# TODO: Remove when Python 3.9 support is dropped
4+
from __future__ import annotations
5+
6+
import json
7+
8+
from pydantic import BaseModel, Field, field_validator
9+
10+
from ..models import (
11+
ExecuteConfig,
12+
JsonDict,
13+
ParameterLocation,
14+
StackOneError,
15+
StackOneTool,
16+
ToolParameters,
17+
)
18+
19+
20+
class FeedbackInput(BaseModel):
21+
"""Input schema for feedback tool."""
22+
23+
feedback: str = Field(..., min_length=1, description="User feedback text")
24+
account_id: str = Field(..., min_length=1, description="Account identifier")
25+
tool_names: list[str] = Field(..., min_length=1, description="List of tool names")
26+
27+
@field_validator("feedback", "account_id")
28+
@classmethod
29+
def validate_non_empty_trimmed(cls, v: str) -> str:
30+
"""Validate that string is non-empty after trimming."""
31+
trimmed = v.strip()
32+
if not trimmed:
33+
raise ValueError("Field must be a non-empty string")
34+
return trimmed
35+
36+
@field_validator("tool_names")
37+
@classmethod
38+
def validate_tool_names(cls, v: list[str]) -> list[str]:
39+
"""Validate and clean tool names."""
40+
cleaned = [name.strip() for name in v if name.strip()]
41+
if not cleaned:
42+
raise ValueError("At least one tool name is required")
43+
return cleaned
44+
45+
46+
class FeedbackTool(StackOneTool):
47+
"""Extended tool for collecting feedback with enhanced validation."""
48+
49+
def execute(
50+
self, arguments: str | JsonDict | None = None, *, options: JsonDict | None = None
51+
) -> JsonDict:
52+
"""
53+
Execute the feedback tool with enhanced validation.
54+
55+
Args:
56+
arguments: Tool arguments as string or dict
57+
options: Execution options
58+
59+
Returns:
60+
Response from the API
61+
62+
Raises:
63+
StackOneError: If validation or API call fails
64+
"""
65+
try:
66+
# Parse input
67+
if isinstance(arguments, str):
68+
raw_params = json.loads(arguments)
69+
else:
70+
raw_params = arguments or {}
71+
72+
# Validate with Pydantic
73+
parsed_params = FeedbackInput(**raw_params)
74+
75+
# Build validated request body
76+
validated_arguments = {
77+
"feedback": parsed_params.feedback,
78+
"account_id": parsed_params.account_id,
79+
"tool_names": parsed_params.tool_names,
80+
}
81+
82+
# Use the parent execute method with validated arguments
83+
return super().execute(validated_arguments, options=options)
84+
85+
except json.JSONDecodeError as exc:
86+
raise StackOneError(f"Invalid JSON in arguments: {exc}") from exc
87+
except ValueError as exc:
88+
raise StackOneError(f"Validation error: {exc}") from exc
89+
except Exception as error:
90+
if isinstance(error, StackOneError):
91+
raise
92+
raise StackOneError(f"Error executing feedback tool: {error}") from error
93+
94+
95+
def create_feedback_tool(
96+
api_key: str,
97+
account_id: str | None = None,
98+
base_url: str = "https://api.stackone.com",
99+
) -> FeedbackTool:
100+
"""
101+
Create a feedback collection tool.
102+
103+
Args:
104+
api_key: API key for authentication
105+
account_id: Optional account ID
106+
base_url: Base URL for the API
107+
108+
Returns:
109+
FeedbackTool configured for feedback collection
110+
"""
111+
name = "meta_collect_tool_feedback"
112+
description = (
113+
"Collects user feedback on StackOne tool performance. "
114+
"First ask the user, \"Are you ok with sending feedback to StackOne?\" "
115+
"and mention that the LLM will take care of sending it. "
116+
"Call this tool only when the user explicitly answers yes."
117+
)
118+
119+
parameters = ToolParameters(
120+
type="object",
121+
properties={
122+
"account_id": {
123+
"type": "string",
124+
"description": 'Account identifier (e.g., "acc_123456")',
125+
},
126+
"feedback": {
127+
"type": "string",
128+
"description": "Verbatim feedback from the user about their experience with StackOne tools.",
129+
},
130+
"tool_names": {
131+
"type": "array",
132+
"items": {
133+
"type": "string",
134+
},
135+
"description": "Array of tool names being reviewed",
136+
},
137+
},
138+
)
139+
140+
execute_config = ExecuteConfig(
141+
name=name,
142+
method="POST",
143+
url=f"{base_url}/ai/tool-feedback",
144+
body_type="json",
145+
parameter_locations={
146+
"feedback": ParameterLocation.BODY,
147+
"account_id": ParameterLocation.BODY,
148+
"tool_names": ParameterLocation.BODY,
149+
},
150+
)
151+
152+
# Create instance by calling parent class __init__ directly since FeedbackTool is a subclass
153+
tool = FeedbackTool.__new__(FeedbackTool)
154+
StackOneTool.__init__(
155+
tool,
156+
description=description,
157+
parameters=parameters,
158+
_execute_config=execute_config,
159+
_api_key=api_key,
160+
_account_id=account_id,
161+
)
162+
163+
return tool
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
"""Implicit feedback instrumentation for the StackOne Python SDK."""
2+
3+
from .analyzer import BehaviorAnalyzer, BehaviorAnalyzerConfig
4+
from .data import ImplicitFeedbackEvent, ToolCallQualitySignals, ToolCallRecord
5+
from .langsmith_client import LangsmithFeedbackClient
6+
from .manager import ImplicitFeedbackManager, configure_implicit_feedback, get_implicit_feedback_manager
7+
from .session import SessionTracker
8+
9+
__all__ = [
10+
"BehaviorAnalyzer",
11+
"BehaviorAnalyzerConfig",
12+
"ImplicitFeedbackEvent",
13+
"ImplicitFeedbackManager",
14+
"LangsmithFeedbackClient",
15+
"SessionTracker",
16+
"ToolCallQualitySignals",
17+
"ToolCallRecord",
18+
"configure_implicit_feedback",
19+
"get_implicit_feedback_manager",
20+
]
21+
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
from __future__ import annotations
2+
3+
from dataclasses import dataclass
4+
from typing import Sequence
5+
6+
from .data import ToolCallQualitySignals, ToolCallRecord
7+
8+
9+
@dataclass(frozen=True)
10+
class BehaviorAnalyzerConfig:
11+
quick_refinement_window_seconds: float = 12.0
12+
task_switch_window_seconds: float = 180.0
13+
failure_penalty: float = 0.3
14+
quick_refinement_penalty: float = 0.25
15+
task_switch_penalty: float = 0.2
16+
17+
18+
class BehaviorAnalyzer:
19+
"""Derive behavioural quality signals from a stream of tool calls."""
20+
21+
def __init__(self, config: BehaviorAnalyzerConfig | None = None) -> None:
22+
self._config = config or BehaviorAnalyzerConfig()
23+
24+
def analyze(self, history: Sequence[ToolCallRecord], current: ToolCallRecord) -> ToolCallQualitySignals:
25+
"""Compute quality signals for a tool call."""
26+
27+
session_history = [call for call in history if call.session_id == current.session_id and call.call_id != current.call_id]
28+
29+
quick_refinement, refinement_window = self._detect_quick_refinement(session_history, current)
30+
task_switch = self._detect_task_switch(session_history, current)
31+
suitability_score = self._compute_suitability_score(current.status, quick_refinement, task_switch)
32+
33+
return ToolCallQualitySignals(
34+
quick_refinement=quick_refinement,
35+
task_switch=task_switch,
36+
suitability_score=suitability_score,
37+
refinement_window_seconds=refinement_window,
38+
)
39+
40+
def _detect_quick_refinement(
41+
self, history: Sequence[ToolCallRecord], current: ToolCallRecord
42+
) -> tuple[bool, float | None]:
43+
if not current.session_id or not history:
44+
return False, None
45+
46+
last_event = history[-1]
47+
elapsed = (current.start_time - last_event.end_time).total_seconds()
48+
if elapsed < 0:
49+
# Ignore out-of-order events
50+
return False, None
51+
52+
if (
53+
last_event.tool_name == current.tool_name
54+
and elapsed <= self._config.quick_refinement_window_seconds
55+
):
56+
return True, elapsed
57+
58+
return False, None
59+
60+
def _detect_task_switch(self, history: Sequence[ToolCallRecord], current: ToolCallRecord) -> bool:
61+
if not current.session_id or not history:
62+
return False
63+
64+
for previous in reversed(history):
65+
elapsed = (current.start_time - previous.end_time).total_seconds()
66+
if elapsed < 0:
67+
continue
68+
if elapsed > self._config.task_switch_window_seconds:
69+
break
70+
if previous.tool_name != current.tool_name:
71+
return True
72+
73+
return False
74+
75+
def _compute_suitability_score(self, status: str, quick_refinement: bool, task_switch: bool) -> float:
76+
score = 1.0
77+
if status != "success":
78+
score -= self._config.failure_penalty
79+
if quick_refinement:
80+
score -= self._config.quick_refinement_penalty
81+
if task_switch:
82+
score -= self._config.task_switch_penalty
83+
return max(0.0, min(1.0, score))

0 commit comments

Comments
 (0)