Skip to content

Commit 9179918

Browse files
NicolasBelissentclaude
NicolasBelissent
and
claude
authored
feat: feedback tool (#36)
* feat: add feedback collection tool and implicit feedback system Add meta_collect_tool_feedback tool for collecting user feedback about StackOne tool performance. The tool is automatically included in the toolset and can be invoked by AI agents after user consent. Features: - Feedback collection tool with Pydantic validation - Automatic whitespace trimming and input cleaning - Support for custom base URLs - Integration with StackOneToolSet - OpenAI and LangChain format support Also includes implicit feedback system for automatic behavioral feedback to LangSmith: - Behavior analyzer for detecting quick refinements - Session tracking for tool call history - LangSmith client integration - Configurable via environment variables Breaking changes: - Updated StackOneTool.execute() signature to include options parameter - Updated meta tools execute methods to support options parameter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * changing tests * Fix linting errors: line length, imports, and trailing whitespace * feat: add minimal feedback collection tool - Add feedback tool that sends data to API endpoint - No LangSmith integration - just API calls - Simple validation with Pydantic - Auto-included in StackOneToolSet - 6 focused tests covering validation, execution, and integration * cleanup: remove implicit feedback files - focus on feedback tool only * fix: remove implicit feedback imports and fix call method - Remove implicit feedback imports from __init__.py - Fix base class call method to not pass options parameter - All 6 feedback tests now passing * fix: remove unused typing.Any import * fix: resolve mypy errors - Remove implicit feedback imports and unreachable code - Fix execute method signature to match base class - All mypy checks now pass * fix: remove unused variables after implicit feedback cleanup - Remove unused start_time, call_params, end_time variables - All linting checks now pass (Ruff + Mypy) * feat: support multiple account IDs in feedback tool - Allow account_id to be string or list of strings - Send same feedback to each account individually - Return combined results with success/failure counts - Add comprehensive tests for validation and execution - All 9 tests passing, linting clean * refactor: format tool.execute calls for improved readability - Adjust formatting of tool.execute calls in test_feedback.py for consistency - Enhance code clarity by aligning dictionary entries - All tests remain passing * refactor: improve feedback tests - remove redundancy - Split large validation test into focused, single-purpose tests - Consolidate execution tests to avoid duplication - Combine multiple account success/error scenarios into one test - Remove redundant integration test class - All 10 tests still passing, better organization * improving test suite --------- Co-authored-by: Claude <[email protected]>
1 parent 983e2f7 commit 9179918

File tree

9 files changed

+4663
-3396
lines changed

9 files changed

+4663
-3396
lines changed

‎README.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,51 @@ employee = employee_tool.call(id="employee-id")
7575
employee = employee_tool.execute({"id": "employee-id"})
7676
```
7777

78+
## Implicit Feedback (Beta)
79+
80+
The Python SDK can emit implicit behavioural feedback to LangSmith so you can triage low-quality tool results without manually tagging runs.
81+
82+
### Automatic configuration
83+
84+
Set `LANGSMITH_API_KEY` in your environment and the SDK will initialise the implicit feedback manager on first tool execution. You can optionally fine-tune behaviour with:
85+
86+
- `STACKONE_IMPLICIT_FEEDBACK_ENABLED` (`true`/`false`, defaults to `true` when an API key is present)
87+
- `STACKONE_IMPLICIT_FEEDBACK_PROJECT` to pin a LangSmith project name
88+
- `STACKONE_IMPLICIT_FEEDBACK_TAGS` with a comma-separated list of tags applied to every run
89+
90+
### Manual configuration
91+
92+
If you want custom session or user resolvers, call `configure_implicit_feedback` during start-up:
93+
94+
```python
95+
from stackone_ai import configure_implicit_feedback
96+
97+
configure_implicit_feedback(
98+
api_key="/path/to/langsmith.key",
99+
project_name="stackone-agents",
100+
default_tags=["python-sdk"],
101+
)
102+
```
103+
104+
Providing your own `session_resolver`/`user_resolver` callbacks lets you derive identifiers from the request context before events are sent to LangSmith.
105+
106+
### Attaching session context to tool calls
107+
108+
Both `tool.execute` and `tool.call` accept an `options` keyword that is excluded from the API request but forwarded to the feedback manager:
109+
110+
```python
111+
tool.execute(
112+
{"id": "employee-id"},
113+
options={
114+
"feedback_session_id": "chat-42",
115+
"feedback_user_id": "user-123",
116+
"feedback_metadata": {"conversation_id": "abc"},
117+
},
118+
)
119+
```
120+
121+
When two calls for the same session happen within a few seconds, the SDK emits a `refinement_needed` event, and you can inspect suitability scores directly in LangSmith.
122+
78123
## Integration Examples
79124

80125
<details>
@@ -200,6 +245,31 @@ result = crew.kickoff()
200245

201246
</details>
202247

248+
## Feedback Collection
249+
250+
The SDK includes a feedback collection tool (`meta_collect_tool_feedback`) that allows users to submit feedback about their experience with StackOne tools. This tool is automatically included in the toolset and is designed to be invoked by AI agents after user permission.
251+
252+
```python
253+
from stackone_ai import StackOneToolSet
254+
255+
toolset = StackOneToolSet()
256+
257+
# Get the feedback tool (included with "meta_*" pattern or all tools)
258+
tools = toolset.get_tools("meta_*")
259+
feedback_tool = tools.get_tool("meta_collect_tool_feedback")
260+
261+
# Submit feedback (typically invoked by AI after user consent)
262+
result = feedback_tool.call(
263+
feedback="The HRIS tools are working great! Very fast response times.",
264+
account_id="acc_123456",
265+
tool_names=["hris_list_employees", "hris_get_employee"]
266+
)
267+
```
268+
269+
**Important**: The AI agent should always ask for user permission before submitting feedback:
270+
- "Are you ok with sending feedback to StackOne? The LLM will take care of sending it."
271+
- Only call the tool after the user explicitly agrees.
272+
203273
## Meta Tools (Beta)
204274

205275
Meta tools enable dynamic tool discovery and execution without hardcoding tool names:

‎stackone_ai/__init__.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,9 @@
33
from .models import StackOneTool, Tools
44
from .toolset import StackOneToolSet
55

6-
__all__ = ["StackOneToolSet", "StackOneTool", "Tools"]
6+
__all__ = [
7+
"StackOneToolSet",
8+
"StackOneTool",
9+
"Tools",
10+
]
711
__version__ = "0.3.2"

‎stackone_ai/feedback/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
"""Feedback collection tools for StackOne."""
2+
3+
from .tool import create_feedback_tool
4+
5+
__all__ = ["create_feedback_tool"]

‎stackone_ai/feedback/tool.py‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
"""Feedback collection tool for StackOne."""
2+
3+
# TODO: Remove when Python 3.9 support is dropped
4+
from __future__ import annotations
5+
6+
import json
7+
8+
from pydantic import BaseModel, Field, field_validator
9+
10+
from ..models import (
11+
ExecuteConfig,
12+
JsonDict,
13+
ParameterLocation,
14+
StackOneError,
15+
StackOneTool,
16+
ToolParameters,
17+
)
18+
19+
20+
class FeedbackInput(BaseModel):
21+
"""Input schema for feedback tool."""
22+
23+
feedback: str = Field(..., min_length=1, description="User feedback text")
24+
account_id: str | list[str] = Field(..., description="Account identifier(s) - single ID or list of IDs")
25+
tool_names: list[str] = Field(..., min_length=1, description="List of tool names")
26+
27+
@field_validator("feedback")
28+
@classmethod
29+
def validate_feedback(cls, v: str) -> str:
30+
"""Validate that feedback is non-empty after trimming."""
31+
trimmed = v.strip()
32+
if not trimmed:
33+
raise ValueError("Feedback must be a non-empty string")
34+
return trimmed
35+
36+
@field_validator("account_id")
37+
@classmethod
38+
def validate_account_id(cls, v: str | list[str]) -> list[str]:
39+
"""Validate and normalize account ID(s) to a list."""
40+
if isinstance(v, str):
41+
trimmed = v.strip()
42+
if not trimmed:
43+
raise ValueError("Account ID must be a non-empty string")
44+
return [trimmed]
45+
46+
if isinstance(v, list):
47+
if not v:
48+
raise ValueError("At least one account ID is required")
49+
cleaned = [str(item).strip() for item in v if str(item).strip()]
50+
if not cleaned:
51+
raise ValueError("At least one valid account ID is required")
52+
return cleaned
53+
54+
raise ValueError("Account ID must be a string or list of strings")
55+
56+
@field_validator("tool_names")
57+
@classmethod
58+
def validate_tool_names(cls, v: list[str]) -> list[str]:
59+
"""Validate and clean tool names."""
60+
cleaned = [name.strip() for name in v if name.strip()]
61+
if not cleaned:
62+
raise ValueError("At least one tool name is required")
63+
return cleaned
64+
65+
66+
class FeedbackTool(StackOneTool):
67+
"""Extended tool for collecting feedback with enhanced validation."""
68+
69+
def execute(
70+
self, arguments: str | JsonDict | None = None, *, options: JsonDict | None = None
71+
) -> JsonDict:
72+
"""
73+
Execute the feedback tool with enhanced validation.
74+
75+
If multiple account IDs are provided, sends the same feedback to each account individually.
76+
77+
Args:
78+
arguments: Tool arguments as string or dict
79+
options: Execution options
80+
81+
Returns:
82+
Combined response from all API calls
83+
84+
Raises:
85+
StackOneError: If validation or API call fails
86+
"""
87+
try:
88+
# Parse input
89+
if isinstance(arguments, str):
90+
raw_params = json.loads(arguments)
91+
else:
92+
raw_params = arguments or {}
93+
94+
# Validate with Pydantic
95+
parsed_params = FeedbackInput(**raw_params)
96+
97+
# Get list of account IDs (already normalized by validator)
98+
account_ids = parsed_params.account_id
99+
feedback = parsed_params.feedback
100+
tool_names = parsed_params.tool_names
101+
102+
# If only one account ID, use the parent execute method
103+
if len(account_ids) == 1:
104+
validated_arguments = {
105+
"feedback": feedback,
106+
"account_id": account_ids[0],
107+
"tool_names": tool_names,
108+
}
109+
return super().execute(validated_arguments, options=options)
110+
111+
# Multiple account IDs - send to each individually
112+
results = []
113+
errors = []
114+
115+
for account_id in account_ids:
116+
try:
117+
validated_arguments = {
118+
"feedback": feedback,
119+
"account_id": account_id,
120+
"tool_names": tool_names,
121+
}
122+
result = super().execute(validated_arguments, options=options)
123+
results.append({
124+
"account_id": account_id,
125+
"status": "success",
126+
"result": result
127+
})
128+
except Exception as exc:
129+
error_msg = str(exc)
130+
errors.append({
131+
"account_id": account_id,
132+
"status": "error",
133+
"error": error_msg
134+
})
135+
results.append({
136+
"account_id": account_id,
137+
"status": "error",
138+
"error": error_msg
139+
})
140+
141+
# Return combined results
142+
return {
143+
"message": f"Feedback sent to {len(account_ids)} account(s)",
144+
"total_accounts": len(account_ids),
145+
"successful": len([r for r in results if r["status"] == "success"]),
146+
"failed": len(errors),
147+
"results": results
148+
}
149+
150+
except json.JSONDecodeError as exc:
151+
raise StackOneError(f"Invalid JSON in arguments: {exc}") from exc
152+
except ValueError as exc:
153+
raise StackOneError(f"Validation error: {exc}") from exc
154+
except Exception as error:
155+
if isinstance(error, StackOneError):
156+
raise
157+
raise StackOneError(f"Error executing feedback tool: {error}") from error
158+
159+
160+
def create_feedback_tool(
161+
api_key: str,
162+
account_id: str | None = None,
163+
base_url: str = "https://api.stackone.com",
164+
) -> FeedbackTool:
165+
"""
166+
Create a feedback collection tool.
167+
168+
Args:
169+
api_key: API key for authentication
170+
account_id: Optional account ID
171+
base_url: Base URL for the API
172+
173+
Returns:
174+
FeedbackTool configured for feedback collection
175+
"""
176+
name = "meta_collect_tool_feedback"
177+
description = (
178+
"Collects user feedback on StackOne tool performance. "
179+
"First ask the user, \"Are you ok with sending feedback to StackOne?\" "
180+
"and mention that the LLM will take care of sending it. "
181+
"Call this tool only when the user explicitly answers yes."
182+
)
183+
184+
parameters = ToolParameters(
185+
type="object",
186+
properties={
187+
"account_id": {
188+
"oneOf": [
189+
{
190+
"type": "string",
191+
"description": 'Single account identifier (e.g., "acc_123456")',
192+
},
193+
{
194+
"type": "array",
195+
"items": {"type": "string"},
196+
"description": "List of account identifiers for multiple accounts",
197+
},
198+
],
199+
"description": "Account identifier(s) - single ID or list of IDs",
200+
},
201+
"feedback": {
202+
"type": "string",
203+
"description": "Verbatim feedback from the user about their experience with StackOne tools.",
204+
},
205+
"tool_names": {
206+
"type": "array",
207+
"items": {
208+
"type": "string",
209+
},
210+
"description": "Array of tool names being reviewed",
211+
},
212+
},
213+
)
214+
215+
execute_config = ExecuteConfig(
216+
name=name,
217+
method="POST",
218+
url=f"{base_url}/ai/tool-feedback",
219+
body_type="json",
220+
parameter_locations={
221+
"feedback": ParameterLocation.BODY,
222+
"account_id": ParameterLocation.BODY,
223+
"tool_names": ParameterLocation.BODY,
224+
},
225+
)
226+
227+
# Create instance by calling parent class __init__ directly since FeedbackTool is a subclass
228+
tool = FeedbackTool.__new__(FeedbackTool)
229+
StackOneTool.__init__(
230+
tool,
231+
description=description,
232+
parameters=parameters,
233+
_execute_config=execute_config,
234+
_api_key=api_key,
235+
_account_id=account_id,
236+
)
237+
238+
return tool

‎stackone_ai/meta_tools.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,9 @@ def __init__(self) -> None:
193193
_account_id=None,
194194
)
195195

196-
def execute(self, arguments: str | JsonDict | None = None) -> JsonDict:
196+
def execute(
197+
self, arguments: str | JsonDict | None = None, *, options: JsonDict | None = None
198+
) -> JsonDict:
197199
return execute_filter(arguments)
198200

199201
return MetaSearchTool()
@@ -272,7 +274,9 @@ def __init__(self) -> None:
272274
_account_id=None,
273275
)
274276

275-
def execute(self, arguments: str | JsonDict | None = None) -> JsonDict:
277+
def execute(
278+
self, arguments: str | JsonDict | None = None, *, options: JsonDict | None = None
279+
) -> JsonDict:
276280
return execute_tool(arguments)
277281

278282
return MetaExecuteTool()

0 commit comments

Comments
 (0)