Skip to content

Commit b7ec672

Browse files
lbeurerkellnerlbeurerkellner
committed
Merge branch 'main' of github.com:invariantlabs-ai/docs
2 parents a476f30 + 0fe35fd commit b7ec672

File tree

1 file changed

+263
-0
lines changed

1 file changed

+263
-0
lines changed

‎docs/testing/Examples/openai-python-agent.md‎

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
---
2+
title: OpenAI Python Agent
3+
---
4+
5+
# OpenAI Function Calling Agent
6+
7+
<div class="subtitle">
8+
Test an OpenAI function calling agent using <code>testing</code>.
9+
</div>
10+
11+
# Intro
12+
13+
OpenAI's function calling can be used to build agents that integrate with external tools and APIs, allowing the agent to call custom functions and deliver enhanced, context-aware responses. More details can be found here: [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling)
14+
15+
In this chapter we are testing simple OpenAI function calling agent as [implemented in this file](https://github.com/invariantlabs-ai/testing/blob/main/sample_tests/openai/test_python_agent.py).
16+
17+
## Agent Overview
18+
19+
The agent generates and executes Python code in response to user requests and returns the computed results. It operates under a strict prompt and utilizes the run_python tool to guarantee accurate code execution and adherence to its intended functionality.
20+
21+
A loop is implemented to run the client until the chat is completed without further tool calls. During this process, all chat interactions are stored in `messages`. A simplifed implementation is shown below:
22+
23+
```python
24+
while True:
25+
26+
# call the client to get response
27+
response = self.client.chat.completions.create(
28+
messages=messages,
29+
model="gpt-4o",
30+
tools=tools,
31+
)
32+
33+
# check if the response calling tools, if not means the chat is completed
34+
tool_calls = response.choices[0].message.tool_calls
35+
if tool_calls:
36+
37+
# append current response message t o messages
38+
messages.append(response_message.to_dict())
39+
40+
# In this demo there's only one tool call in the response
41+
tool_call = tool_calls[0]
42+
if tool_call.function.name == "run_python":
43+
44+
# get the arguments generated by agent for the function
45+
function_args = json.loads(tool_call.function.arguments)
46+
47+
# run the function with the argument with "code"
48+
function_response = run_python(function_args["code"])
49+
50+
# append the response of the function to messages for next round chat
51+
messages.append(
52+
{
53+
"tool_call_id": tool_call.id,
54+
"role": "tool",
55+
"name": "run_python",
56+
"content": str(function_response),
57+
}
58+
)
59+
else:
60+
break
61+
```
62+
63+
## Run the Example
64+
65+
You can run the example by running the following command in the root of the repository:
66+
67+
```bash
68+
poetry run invariant test sample_tests/openai/test_python_agent.py --push --dataset_name test_python_agent
69+
```
70+
71+
>**Note** If you want to run the example without sending the results to the Explorer UI, you can always run without the `--push` flag. You will still see the parts of the trace that fail
72+
as higihlighted in the terminal.
73+
74+
75+
## Unit Tests
76+
77+
Here, we design three unit tests to cover different scenarios.
78+
79+
In these tests, we set varied `input` to reflect different situations. Within each test, we create an instance of the agent named `python_agent`, and retrieve its response by calling `python_agent.get_response(input)`.
80+
81+
The agent's response is subsequently transformed into a Trace object using` TraceFactory.from_openai(response)` for further validation.
82+
83+
### Test 1: Valid Python Code Execution:
84+
85+
<div class='tiles'>
86+
<a href="https://explorer.invariantlabs.ai/u/zishan-wei/openai_python_agent-1733417505/t/1" class='tile'>
87+
<span class='tile-title'>Open in Explorer →</span>
88+
<span class='tile-description'>See this example in the Invariant Explorer</span>
89+
</a>
90+
</div>
91+
92+
In the first test, we ask the agent to calculate the Fibonacci series for the first 10 elements using Python.
93+
94+
The implementation of the first test is shown below:
95+
```python
96+
def test_python_question():
97+
input = "Calculate fibonacci series for the first 10 elements in python"
98+
99+
# run the agent
100+
python_agent = PythonAgent()
101+
response = python_agent.get_response(input)
102+
103+
# convert trace
104+
trace = TraceFactory.from_openai(response)
105+
106+
# test the agent behavior
107+
with trace.as_context():
108+
run_python_tool_call = trace.tool_calls(name="run_python")
109+
110+
# assert the agent calls "run_python" exactly once
111+
assert_true(F.len(run_python_tool_call) == 1)
112+
113+
# assert the argument passed to the tool_call is valid Python code.
114+
assert_true(
115+
run_python_tool_call[0]["function"]["arguments"]["code"].is_valid_code(
116+
"python"
117+
)
118+
)
119+
120+
# assert if 34 is included in the agent's final response.
121+
assert_true("34" in trace.messages(-1)["content"])
122+
```
123+
124+
Our primary objective is to verify that the agent correctly calls the `run_python` tool and provides valid Python code as its parameter. To achieve this, we first filter the tool_calls where `name = "run_python"`. Then, we assert that exactly one `tool_call` meets this condition. Next, we confirm that the argument passed to the `tool_call` is valid Python code.
125+
126+
Then we validate that the Python code executes correctly. To confirm this, we check if one of the calculated result, "34," is included in the agent's final response.
127+
128+
### Test 2: Invalid Response:
129+
130+
<div class='tiles'>
131+
<a href="https://explorer.invariantlabs.ai/u/zishan-wei/openai_python_agent-1733417505/t/2" class='tile'>
132+
<span class='tile-title'>Open in Explorer →</span>
133+
<span class='tile-description'>See this example in the Invariant Explorer</span>
134+
</a>
135+
</div>
136+
137+
In this test, we use `unittest.mock.MagicMock` to simulate a scenario where the agent incorrectly responds with Java code instead of Python, ensuring such behavior is detected. The actual response from `python_agent.get_response(input)` is replaced with our custom content stored in `mock_invalid_response`
138+
139+
The implementation of the second test is shown below:
140+
```python
141+
142+
def test_python_question_invalid():
143+
input = "Calculate fibonacci series for the first 10 elements in python"
144+
python_agent = PythonAgent()
145+
146+
# set custom response that contains Java code instead of Python code
147+
mock_invalid_response = [
148+
{
149+
"role": "system",
150+
"content": '\n You are an assistant that strictly responds with Python code only. \n The code should print the result.\n You always use tool run_python to execute the code that you write to present the results.\n If the user specifies other programming language in the question, you should respond with "I can only help with Python code."\n ',
151+
},
152+
{"role": "user", "content": "Calculate fibonacci series for 10"},
153+
{
154+
"content": "None",
155+
"refusal": "None",
156+
"role": "assistant",
157+
"tool_calls": [
158+
{
159+
"id": "call_GMx1WYM7sN0BGY1ISCk05zez",
160+
"function": {
161+
"arguments": '{"code":"public class Fibonacci { public static void main(String[] args) { for (int n = 10, a = 0, b = 1, i = 0; i < n; i++, b = a + (a = b)) System.out.print(a + '
162+
'); } }"}',
163+
"name": "run_python",
164+
},
165+
"type": "function",
166+
}
167+
],
168+
},
169+
]
170+
171+
# the response will be replaced by our mock_invalid_response
172+
python_agent.get_response = MagicMock(return_value=mock_invalid_response)
173+
response = python_agent.get_response(input)
174+
175+
# convert trace
176+
trace = TraceFactory.from_openai(response)
177+
178+
# test the agent behavior
179+
with trace.as_context():
180+
run_python_tool_call = trace.tool_calls(name="run_python")
181+
182+
assert_true(F.len(run_python_tool_call) == 1)
183+
assert_true(
184+
not run_python_tool_call[0]["function"]["arguments"]["code"].is_valid_code(
185+
"python"
186+
)
187+
)
188+
189+
```
190+
191+
In this test we still verify that the agent correctly calls the run_python tool once, but it provids invalid Python code as its parameter. So we assert that the parameter passed to this call is not valid Python code.
192+
193+
### Test 3: Non-Python Language Request:
194+
195+
<div class='tiles'>
196+
<a href="https://explorer.invariantlabs.ai/u/zishan-wei/openai_python_agent-1733417505/t/3" class='tile'>
197+
<span class='tile-title'>Open in Explorer →</span>
198+
<span class='tile-description'>See this example in the Invariant Explorer</span>
199+
</a>
200+
</div>
201+
202+
This test's request included another programming langguage Java and the agent should be able to handle it nicely as clarifyed in the prompt.
203+
204+
This test evaluates the agent's ability to handle requests involving a programming language other than Python, specifically Java. The agent is expected to respond appropriately by clarifying its limitation to Python code as outlined in the prompt.
205+
206+
The implementation of the third test is shown below:
207+
```python
208+
209+
def test_java_question():
210+
input = "How to calculate fibonacci series in Java?"
211+
# run the agent
212+
python_agent = PythonAgent()
213+
response = python_agent.get_response(input)
214+
215+
# convert trace
216+
trace = TraceFactory.from_openai(response)
217+
218+
# set expected response as clarified in prompt
219+
expected_response = "I can only help with Python code."
220+
221+
# test the agent behavior
222+
with trace.as_context():
223+
224+
# assert that the agent does not call the `run_python` tool
225+
run_python_tool_call = trace.tool_calls(name="run_python")
226+
assert_true(F.len(run_python_tool_call) == 0)
227+
228+
# assert that the real repsonse is close enough with expected response
229+
expect_equals(
230+
"I can only help with Python code.", trace.messages(-1)["content"]
231+
)
232+
assert_true(trace.messages(-1)["content"].levenshtein(expected_response) < 5)
233+
234+
```
235+
236+
The first validation confirms that the agent does not call the `run_python` tool.
237+
238+
The agent’s response should align closely with `expected_response = "I can only help with Python code."`.
239+
We use the `expect_equals` assertion, which is less strict than `assert_equal`, to validate similarity.
240+
241+
To further confirm similarity, weo use our `levenshtein()` function which calculate Levenshtein distance. So we assert that the Levenshtein distance between the response and the expected response is smaller than 5.
242+
243+
To further confirm similarity, we compute the Levenshtein distance between the agent's response and the expected output, ensuring it is less than 5 using our `levenshtein()` function.
244+
245+
## Conclusion
246+
247+
We have seen how to build an OpenAI Function Calling Agent and how to write unit tests to ensure the agent functions correctly by using `testing`.
248+
249+
To learn more, please select a topic from the tiles below.
250+
251+
<div class='tiles'>
252+
253+
<a href="https://en.wikipedia.org/wiki/Levenshtein_distance" class='tile primary'>
254+
<span class='tile-title'>Levenshtein Distance →</span>
255+
<span class='tile-description'>Wikipedia's introduction of Levenshtein Distance</span>
256+
</a>
257+
258+
<a href="https://docs.python.org/3/library/unittest.mock.html" class='tile primary'>
259+
<span class='tile-title'>Intro of unittest.mock →</span>
260+
<span class='tile-description'>Docs of unittest.mock — mock object library</span>
261+
</a>
262+
263+
</div>

0 commit comments

Comments
 (0)