Skip to content

Commit cebe84d

Browse files
RoxyFarhadRoxyFarhad
committed
fixing tests
1 parent 258045d commit cebe84d

File tree

2 files changed

+164
-36
lines changed

2 files changed

+164
-36
lines changed

‎.github/workflows/agentex-tutorials-test.yml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,132 @@ jobs:
101101
working-directory: ./examples/tutorials
102102
env:
103103
OPENAI_API_KEY: ${{ secrets.TUTORIAL_OPENAI_API_KEY }}
104+
HEALTH_CHECK_PORT: 8080 # Use non-privileged port for temporal worker health checks
104105
run: |
105106
echo "Testing tutorial: ${{ matrix.tutorial }}"
106107
AGENTEX_API_BASE_URL="http://localhost:5003" \
107108
./run_agent_test.sh --build-cli "${{ matrix.tutorial }}"
109+
110+
- name: Upload Test Results
111+
if: always()
112+
uses: actions/upload-artifact@v4
113+
with:
114+
name: test-results-${{ replace(matrix.tutorial, '/', '-') }}
115+
path: |
116+
/tmp/agentex-*.log
117+
retention-days: 1
118+
119+
test-summary:
120+
if: always()
121+
needs: [find-tutorials, test-tutorial]
122+
runs-on: ubuntu-latest
123+
name: Test Summary
124+
steps:
125+
- name: Download All Test Results
126+
uses: actions/download-artifact@v4
127+
with:
128+
path: test-results
129+
pattern: test-results-*
130+
131+
- name: Generate Test Summary
132+
run: |
133+
echo "# 🧪 Tutorial Tests Summary" >> $GITHUB_STEP_SUMMARY
134+
echo "" >> $GITHUB_STEP_SUMMARY
135+
136+
# Get tutorial list from needs context
137+
tutorials='${{ needs.find-tutorials.outputs.tutorials }}'
138+
139+
# Initialize counters
140+
total_tutorials=0
141+
passed_tutorials=0
142+
failed_tutorials=0
143+
144+
# Arrays to track results
145+
passed_tests=()
146+
failed_tests=()
147+
148+
echo "## 📊 Overall Results" >> $GITHUB_STEP_SUMMARY
149+
echo "" >> $GITHUB_STEP_SUMMARY
150+
151+
# Process each tutorial result
152+
for tutorial_dir in test-results/test-results-*/; do
153+
if [ -d "$tutorial_dir" ]; then
154+
# Extract sanitized name and convert back to original tutorial path
155+
sanitized_name=$(basename "$tutorial_dir" | sed 's/test-results-//')
156+
tutorial_name=$(echo "$sanitized_name" | sed 's/-/\//g')
157+
total_tutorials=$((total_tutorials + 1))
158+
159+
# Determine success/failure based on presence of error logs or patterns
160+
if find "$tutorial_dir" -name "*.log" -exec grep -l "FAILED\|ERROR\|Traceback" {} \; | head -1 >/dev/null; then
161+
failed_tutorials=$((failed_tutorials + 1))
162+
failed_tests+=("$tutorial_name")
163+
else
164+
passed_tutorials=$((passed_tutorials + 1))
165+
passed_tests+=("$tutorial_name")
166+
fi
167+
fi
168+
done
169+
170+
# Show summary stats
171+
echo "| Status | Count |" >> $GITHUB_STEP_SUMMARY
172+
echo "|--------|-------|" >> $GITHUB_STEP_SUMMARY
173+
echo "| ✅ **Passed** | **$passed_tutorials** |" >> $GITHUB_STEP_SUMMARY
174+
echo "| ❌ **Failed** | **$failed_tutorials** |" >> $GITHUB_STEP_SUMMARY
175+
echo "| 📊 **Total** | **$total_tutorials** |" >> $GITHUB_STEP_SUMMARY
176+
echo "" >> $GITHUB_STEP_SUMMARY
177+
178+
# Show passed tests
179+
if [ $passed_tutorials -gt 0 ]; then
180+
echo "## ✅ Passed Tutorials ($passed_tutorials)" >> $GITHUB_STEP_SUMMARY
181+
echo "" >> $GITHUB_STEP_SUMMARY
182+
for test in "${passed_tests[@]}"; do
183+
echo "- ✅ \`$test\`" >> $GITHUB_STEP_SUMMARY
184+
done
185+
echo "" >> $GITHUB_STEP_SUMMARY
186+
fi
187+
188+
# Show failed tests with details
189+
if [ $failed_tutorials -gt 0 ]; then
190+
echo "## ❌ Failed Tutorials ($failed_tutorials)" >> $GITHUB_STEP_SUMMARY
191+
echo "" >> $GITHUB_STEP_SUMMARY
192+
193+
for test in "${failed_tests[@]}"; do
194+
echo "### 🔍 \`$test\`" >> $GITHUB_STEP_SUMMARY
195+
echo "" >> $GITHUB_STEP_SUMMARY
196+
197+
# Find the log file for this test (convert back to sanitized name)
198+
sanitized_test_name=$(echo "$test" | sed 's/\//-/g')
199+
log_file=$(find "test-results/test-results-$sanitized_test_name" -name "*.log" | head -1)
200+
if [ -f "$log_file" ]; then
201+
# Extract pytest failures
202+
if grep -q "FAILED\|ERROR" "$log_file"; then
203+
echo "**Failed Tests:**" >> $GITHUB_STEP_SUMMARY
204+
echo '```' >> $GITHUB_STEP_SUMMARY
205+
grep -A 5 -B 1 "FAILED\|ERROR" "$log_file" | head -20 >> $GITHUB_STEP_SUMMARY
206+
echo '```' >> $GITHUB_STEP_SUMMARY
207+
echo "" >> $GITHUB_STEP_SUMMARY
208+
fi
209+
210+
# Show any Python tracebacks
211+
if grep -q "Traceback" "$log_file"; then
212+
echo "**Error Details:**" >> $GITHUB_STEP_SUMMARY
213+
echo '```' >> $GITHUB_STEP_SUMMARY
214+
# Get the last traceback in the file
215+
awk '/Traceback \(most recent call last\)/{p=1} p{print} /^[^ ]/ && p && !/Traceback/{p=0}' "$log_file" | tail -20 >> $GITHUB_STEP_SUMMARY
216+
echo '```' >> $GITHUB_STEP_SUMMARY
217+
echo "" >> $GITHUB_STEP_SUMMARY
218+
fi
219+
else
220+
echo "_No log file found for detailed error analysis_" >> $GITHUB_STEP_SUMMARY
221+
echo "" >> $GITHUB_STEP_SUMMARY
222+
fi
223+
done
224+
fi
225+
226+
# Set exit code based on results
227+
if [ $failed_tutorials -gt 0 ]; then
228+
echo "❌ Some tutorials failed. Check the details above." >> $GITHUB_STEP_SUMMARY
229+
exit 1
230+
else
231+
echo "🎉 All tutorials passed successfully!" >> $GITHUB_STEP_SUMMARY
232+
fi

‎examples/tutorials/10_agentic/10_temporal/070_open_ai_agents_sdk_tools/project/workflow.py‎

Lines changed: 39 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
66
PATTERN 1: Simple External Tools as Activities (activity_as_tool)
77
- Convert individual Temporal activities directly into agent tools
8-
- 1:1 mapping between tool calls and activities
8+
- 1:1 mapping between tool calls and activities
99
- Best for: single non-deterministic operations (API calls, DB queries)
1010
- Example: get_weather activity → weather tool
1111
@@ -19,30 +19,30 @@
1919
2020
WHY THIS APPROACH IS GAME-CHANGING:
2121
===================================
22-
There's a crucial meta-point that should be coming through here: **why is this different?**
23-
This approach is truly transactional because of how the `await` works in Temporal workflows.
24-
Consider a "move money" example - if the operation fails between the withdraw and deposit,
25-
Temporal will resume exactly where it left off - the agent gets real-world flexibility even
22+
There's a crucial meta-point that should be coming through here: **why is this different?**
23+
This approach is truly transactional because of how the `await` works in Temporal workflows.
24+
Consider a "move money" example - if the operation fails between the withdraw and deposit,
25+
Temporal will resume exactly where it left off - the agent gets real-world flexibility even
2626
if systems die.
2727
28-
**Why even use Temporal? Why are we adding complexity?** The gain is enormous when you
28+
**Why even use Temporal? Why are we adding complexity?** The gain is enormous when you
2929
consider what happens without it:
3030
31-
In a traditional approach without Temporal, if you withdraw money but then the system crashes
32-
before depositing, you're stuck in a broken state. The money has been withdrawn, but never
33-
deposited. In a banking scenario, you can't just "withdraw again" - the money is already gone
31+
In a traditional approach without Temporal, if you withdraw money but then the system crashes
32+
before depositing, you're stuck in a broken state. The money has been withdrawn, but never
33+
deposited. In a banking scenario, you can't just "withdraw again" - the money is already gone
3434
from the source account, and your agent has no way to recover or know what state it was in.
3535
36-
This is why you can't build very complicated agents without this confidence in transactional
36+
This is why you can't build very complicated agents without this confidence in transactional
3737
behavior. Temporal gives us:
3838
3939
- **Guaranteed execution**: If the workflow starts, it will complete, even through failures
4040
- **Exact resumption**: Pick up exactly where we left off, not start over
41-
- **Transactional integrity**: Either both operations complete, or the workflow can be designed
41+
- **Transactional integrity**: Either both operations complete, or the workflow can be designed
4242
to handle partial completion
4343
- **Production reliability**: Build agents that can handle real-world complexity and failures
4444
45-
Without this foundation, agents remain fragile toys. With Temporal, they become production-ready
45+
Without this foundation, agents remain fragile toys. With Temporal, they become production-ready
4646
systems that can handle the complexities of the real world.
4747
"""
4848

@@ -72,11 +72,13 @@
7272

7373
logger = make_logger(__name__)
7474

75+
7576
@workflow.defn(name=environment_variables.WORKFLOW_NAME)
7677
class ExampleTutorialWorkflow(BaseWorkflow):
7778
"""
7879
Minimal async workflow template for AgentEx Temporal agents.
7980
"""
81+
8082
def __init__(self):
8183
super().__init__(display_name=environment_variables.AGENT_NAME)
8284
self._complete_task = False
@@ -85,35 +87,35 @@ def __init__(self):
8587
@workflow.signal(name=SignalName.RECEIVE_EVENT)
8688
async def on_task_event_send(self, params: SendEventParams) -> None:
8789
logger.info(f"Received task message instruction: {params}")
88-
89-
# Echo back the client's message to show it in the UI. This is not done by default
90+
91+
# Echo back the client's message to show it in the UI. This is not done by default
9092
# so the agent developer has full control over what is shown to the user.
9193
await adk.messages.create(task_id=params.task.id, content=params.event.content)
9294

9395
# ============================================================================
9496
# OpenAI Agents SDK + Temporal Integration: Two Patterns for Tool Creation
9597
# ============================================================================
96-
98+
9799
# #### When to Use Activities for Tools
98100
#
99101
# You'll want to use the activity pattern for tools in the following scenarios:
100102
#
101-
# - **API calls within the tool**: Whenever your tool makes an API call (external
102-
# service, database, etc.), you must wrap it as an activity since these are
103+
# - **API calls within the tool**: Whenever your tool makes an API call (external
104+
# service, database, etc.), you must wrap it as an activity since these are
103105
# non-deterministic operations that could fail or return different results
104-
# - **Idempotent single operations**: When the tool performs an already idempotent
105-
# single call that you want to ensure gets executed reliably with Temporal's retry
106+
# - **Idempotent single operations**: When the tool performs an already idempotent
107+
# single call that you want to ensure gets executed reliably with Temporal's retry
106108
# guarantees
107109
#
108-
# Let's start with the case where it is non-deterministic. If this is the case, we
109-
# want this tool to be an activity to guarantee that it will be executed. The way to
110-
# do this is to add some syntax to make the tool call an activity. Let's create a tool
111-
# that gives us the weather and create a weather agent. For this example, we will just
112-
# return a hard-coded string but we can easily imagine this being an API call to a
113-
# weather service which would make it non-deterministic. First we will create a new
114-
# file called `activities.py`. Here we will create a function to get the weather and
110+
# Let's start with the case where it is non-deterministic. If this is the case, we
111+
# want this tool to be an activity to guarantee that it will be executed. The way to
112+
# do this is to add some syntax to make the tool call an activity. Let's create a tool
113+
# that gives us the weather and create a weather agent. For this example, we will just
114+
# return a hard-coded string but we can easily imagine this being an API call to a
115+
# weather service which would make it non-deterministic. First we will create a new
116+
# file called `activities.py`. Here we will create a function to get the weather and
115117
# simply add an activity annotation on top.
116-
118+
117119
# There are TWO key patterns for integrating tools with the OpenAI Agents SDK in Temporal:
118120
#
119121
# PATTERN 1: Simple External Tools as Activities
@@ -147,7 +149,7 @@ async def on_task_event_send(self, params: SendEventParams) -> None:
147149
# The get_weather activity will be executed with durability guarantees
148150
activity_as_tool(
149151
get_weather, # This is defined in activities.py as @activity.defn
150-
start_to_close_timeout=timedelta(seconds=10)
152+
start_to_close_timeout=timedelta(seconds=10),
151153
),
152154
],
153155
)
@@ -156,7 +158,7 @@ async def on_task_event_send(self, params: SendEventParams) -> None:
156158
result = await Runner.run(weather_agent, params.event.content.content)
157159

158160
# ============================================================================
159-
# PATTERN 2: Multiple Activities Within Tools
161+
# PATTERN 2: Multiple Activities Within Tools
160162
# ============================================================================
161163
# Use this pattern when:
162164
# - You need multiple sequential non-deterministic operations within one tool
@@ -171,7 +173,7 @@ async def on_task_event_send(self, params: SendEventParams) -> None:
171173
#
172174
# BENEFITS:
173175
# - Guaranteed execution order (withdraw THEN deposit)
174-
# - Each step is durable and retryable individually
176+
# - Each step is durable and retryable individually
175177
# - Atomic operations from the agent's perspective
176178
# - Better than having LLM make multiple separate tool calls
177179

@@ -186,7 +188,7 @@ async def on_task_event_send(self, params: SendEventParams) -> None:
186188
# move_money,
187189
# ],
188190
# )
189-
191+
190192
# # Run the agent - when it calls move_money tool, it will create TWO activities:
191193
# # 1. withdraw_money activity
192194
# # 2. deposit_money activity (only after withdraw succeeds)
@@ -195,17 +197,17 @@ async def on_task_event_send(self, params: SendEventParams) -> None:
195197
# ============================================================================
196198
# PATTERN COMPARISON SUMMARY:
197199
# ============================================================================
198-
#
200+
#
199201
# Pattern 1 (activity_as_tool): | Pattern 2 (function_tool with activities):
200202
# - Single activity per tool call | - Multiple activities per tool call
201-
# - 1:1 tool to activity mapping | - 1:many tool to activity mapping
203+
# - 1:1 tool to activity mapping | - 1:many tool to activity mapping
202204
# - Simple non-deterministic ops | - Complex multi-step operations
203205
# - Let LLM sequence multiple tools | - Code controls activity sequencing
204206
# - Example: get_weather, db_lookup | - Example: money_transfer, multi_step_workflow
205207
#
206208
# BOTH patterns provide:
207209
# - Automatic retries and failure recovery
208-
# - Full observability in Temporal UI
210+
# - Full observability in Temporal UI
209211
# - Durable execution guarantees
210212
# - Seamless integration with OpenAI Agents SDK
211213
# ============================================================================
@@ -234,11 +236,12 @@ async def on_task_create(self, params: CreateTaskParams) -> str:
234236

235237
await workflow.wait_condition(
236238
lambda: self._complete_task,
237-
timeout=None, # Set a timeout if you want to prevent the task from running indefinitely. Generally this is not needed. Temporal can run hundreds of millions of workflows in parallel and more. Only do this if you have a specific reason to do so.
239+
timeout=None, # Set a timeout if you want to prevent the task from running indefinitely. Generally this is not needed. Temporal can run hundreds of millions of workflows in parallel and more. Only do this if you have a specific reason to do so.
238240
)
239241
return "Task completed"
240242

241243
@workflow.signal
242244
async def fulfill_order_signal(self, success: bool) -> None:
243245
if success == True:
244-
await self._pending_confirmation.put(True)
246+
await self._pending_confirmation.put(True)
247+

0 commit comments

Comments
 (0)