Skip to content

Commit f7d102b

Browse files
authored
Merge branch 'main' into dependabot/npm_and_yarn/website/multi-e981fcb12d
2 parents 73c57b8 + b11d405 commit f7d102b

File tree

8 files changed

+107
-38
lines changed

8 files changed

+107
-38
lines changed

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ Unlike many agent frameworks that only track the chat history with LLMs in text,
2323

2424

2525
## 🆕 News
26+
- 📅2024-12-23: TaskWeaver has been integrated with the [AgentOps](https://microsoft.github.io/TaskWeaver/docs/observability) for better observability and monitoring.🔍
2627
- 📅2024-09-13: We introduce the shared memory to store information that is shared between the roles in TaskWeaver. Please check the [memory](https://microsoft.github.io/TaskWeaver/docs/memory) for more details.🧠
2728
- 📅2024-09-13: We have enhanced the experience feature by allowing static and dynamic experience selection. Please check the [experience](https://microsoft.github.io/TaskWeaver/blog/experience) for more details.📚
2829
- 📅2024-07-02: We have optimized TaskWeaver to support not-that-large language models served locally. Please check this [post](https://microsoft.github.io/TaskWeaver/blog/local_llm) for more details.🔗
@@ -31,7 +32,7 @@ Unlike many agent frameworks that only track the chat history with LLMs in text,
3132
- 📅2024-03-27: TaskWeaver now switches to `container` mode by default for code execution. Please check the [code execution](https://microsoft.github.io/TaskWeaver/docs/code_execution) for more details.🐳
3233
- 📅2024-03-07: TaskWeaver now supports configuration of different LLMs for various components, such as the Planner and CodeInterpreter. Please check the [multi-llm](https://microsoft.github.io/TaskWeaver/docs/llms/multi-llm) for more details.🔗
3334
- 📅2024-03-04: TaskWeaver now supports a [container](https://microsoft.github.io/TaskWeaver/docs/code_execution) mode, which provides a more secure environment for code execution.🐳
34-
- 📅2024-02-28: TaskWeaver now offers a [CLI-only](https://microsoft.github.io/TaskWeaver/docs/advanced/cli_only) mode, enabling users to interact seamlessly with the Command Line Interface (CLI) using natural language.📟
35+
<!-- - 📅2024-02-28: TaskWeaver now offers a [CLI-only](https://microsoft.github.io/TaskWeaver/docs/advanced/cli_only) mode, enabling users to interact seamlessly with the Command Line Interface (CLI) using natural language.📟 -->
3536
<!-- - 📅2024-02-01: TaskWeaver now has a plugin [document_retriever](https://github.com/microsoft/TaskWeaver/blob/main/project/plugins/README.md#document_retriever) for RAG based on a knowledge base.📚 -->
3637
<!-- - 📅2024-01-30: TaskWeaver introduces a new plugin-only mode that securely generates calls to specified plugins without producing extraneous code.🪡 -->
3738
<!-- - 📅2024-01-23: TaskWeaver can now be personalized by transforming your chat histories into enduring [experiences](https://microsoft.github.io/TaskWeaver/docs/customization/experience) 🎉 -->

project/examples/planner_examples/example-planner-2.yaml

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,14 @@ rounds:
1111
send_from: Planner
1212
send_to: User
1313
attachment_list:
14-
- type: init_plan
15-
content: |-
16-
1. Respond to the user's greeting
17-
- type: plan
18-
content: |-
19-
1. Respond to the user's greeting
20-
- type: current_plan_step
21-
content: 1. Respond to the user's greeting
14+
- type: reasoning
15+
content: |-
16+
The user greets the Planner
17+
- type: init_plan
18+
content: |-
19+
1. Respond to the user's greeting
20+
- type: plan
21+
content: |-
22+
1. Respond to the user's greeting
23+
- type: current_plan_step
24+
content: 1. Respond to the user's greeting

project/examples/planner_examples/example-planner-echo.yaml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ rounds:
1111
send_from: Planner
1212
send_to: Echo
1313
attachment_list:
14+
- type: reasoning
15+
content: |-
16+
The user wants to echo the input 'Hello World'
1417
- type: init_plan
1518
content: |-
1619
1. Ask Echo to echo the user's input, 'Hello World'
@@ -27,6 +30,9 @@ rounds:
2730
send_from: Planner
2831
send_to: User
2932
attachment_list:
33+
- type: reasoning
34+
content: |-
35+
The user query is successfully answered
3036
- type: init_plan
3137
content: |-
3238
1. Ask Echo to echo the user's input, 'Hello World'
@@ -35,7 +41,4 @@ rounds:
3541
1. Ask Echo to echo the user's input, 'Hello World'
3642
- type: current_plan_step
3743
content: 1. Ask Echo to echo the user's input, 'Hello World'
38-
- type: review
39-
content: |-
40-
The user query is successfully answered
4144

project/examples/planner_examples/example-planner.yaml

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -11,17 +11,20 @@ rounds:
1111
send_from: Planner
1212
send_to: CodeInterpreter
1313
attachment_list:
14-
- type: init_plan
15-
content: |-
16-
1. Load the data file
17-
2. Count the rows of the loaded data <sequentially depends on 1>
18-
3. Check the execution result and report the result to the user <interactively depends on 2>
19-
- type: plan
20-
content: |-
21-
1. Instruct CodeInterpreter to load the data file and count the rows of the loaded data
22-
2. Check the execution result and report the result to the user if it is correct
23-
- type: current_plan_step
24-
content: 1. Instruct CodeInterpreter to load the data file and count the rows of the loaded data
14+
- type: reasoning
15+
content: |-
16+
The user wants to count the rows of the data file /home/data.csv. The first step is to load the data file and count the rows of the loaded data.
17+
- type: init_plan
18+
content: |-
19+
1. Load the data file
20+
2. Count the rows of the loaded data <sequentially depends on 1>
21+
3. Check the execution result and report the result to the user <interactively depends on 2>
22+
- type: plan
23+
content: |-
24+
1. Instruct CodeInterpreter to load the data file and count the rows of the loaded data
25+
2. Check the execution result and report the result to the user if it is correct
26+
- type: current_plan_step
27+
content: 1. Instruct CodeInterpreter to load the data file and count the rows of the loaded data
2528
- message: Load the data file /home/data.csv successfully and there are 100 rows in the data file
2629
send_from: CodeInterpreter
2730
send_to: Planner
@@ -30,6 +33,11 @@ rounds:
3033
send_from: Planner
3134
send_to: User
3235
attachment_list:
36+
- type: reasoning
37+
content: |-
38+
The data file /home/data.csv is loaded and there are 100 rows in the data file
39+
The execution result is correct
40+
The user query is successfully answered
3341
- type: init_plan
3442
content: |-
3543
1. Load the data file
@@ -41,8 +49,3 @@ rounds:
4149
2. Check the execution result and report the result to the user if it is correct
4250
- type: current_plan_step
4351
content: 2. report the result to the user
44-
- type: review
45-
content: |-
46-
The data file /home/data.csv is loaded and there are 100 rows in the data file
47-
The execution result is correct
48-
The user query is successfully answered

taskweaver/memory/attachment.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ class AttachmentType(Enum):
1212
init_plan = "init_plan"
1313
plan = "plan"
1414
current_plan_step = "current_plan_step"
15-
review = "review"
15+
reasoning = "reasoning"
1616

1717
# CodeInterpreter - generate code
1818
thought = "thought"

taskweaver/planner/planner_prompt.yaml

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -21,15 +21,14 @@ instruction_template: |-
2121
2222
## Planner Character
2323
- Planner's main job is to make planning and to instruct Workers to resolve the request from the User.
24-
- Planner can conduct basic analysis (e.g., comprehension, extraction, etc.) to solve simple problems after reading the messages from the User and the Workers.
25-
- Planner should first try to solve the task by itself before reaching out to the Workers for their special expertise.
24+
- Planner should independently handle basic tasks such as information extraction from text files, using its reasoning and comprehension skills before considering the involvement of Workers.
2625
- Planner can assign different subtasks to different Workers, and each subtask should be assigned to only one Worker.
2726
- Planner must reject the User's request if it contains potential security risks or illegal activities.
2827
- Planner should ask the User to provide additional information critical for problem solving, but only after trying the best.
2928
- Planner can talk to the User and Workers by specifying the `send_to` field in the response, but MUST NOT talk to the Planner itself.
3029
- Planner should refine the plan according to its observations from the replies of the Workers or the new requests of User.
3130
- Planner needs to inform Workers on the User's request, the current step, and necessary information to complete the task.
32-
- Planner must check the Worker's response and provide feedback to the Worker if the response is incorrect or incomplete.
31+
- Planner must thoroughly review Worker's response and provide feedback to the Worker if the response is incorrect or incomplete.
3332
- Planner can ignore the permission or file access issues since Workers are powerful and can handle them.
3433
3534
## Planner's planning process
@@ -120,6 +119,10 @@ response_json_schema: |-
120119
"response": {
121120
"type": "object",
122121
"properties": {
122+
"reasoning": {
123+
"type": "string",
124+
"description": "The reasoning of the Planner's decision. It should include the analysis of the User's request, the Workers' responses, and the current environment context."
125+
},
123126
"init_plan": {
124127
"type": "string",
125128
"description": "The initial plan to decompose the User's task into subtasks and list them as the detailed subtask steps. The initial plan must contain dependency annotations for sequential and interactive dependencies."
@@ -132,10 +135,6 @@ response_json_schema: |-
132135
"type": "string",
133136
"description": "The current step Planner is executing."
134137
},
135-
"review": {
136-
"type": "string",
137-
"description": "The review of the current step. If the Worker's response is incorrect or incomplete, Planner should provide feedback to the Worker."
138-
},
139138
"send_to": {
140139
"type": "string",
141140
"description": "The name of character (User or name of the Worker) that Planner wants to speak to."
@@ -146,12 +145,12 @@ response_json_schema: |-
146145
}
147146
},
148147
"required": [
148+
"reasoning",
149149
"init_plan",
150150
"plan",
151151
"current_plan_step",
152152
"send_to",
153-
"message",
154-
"review"
153+
"message"
155154
],
156155
"additionalProperties": false
157156
}
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
id: observability
3+
description: Observability
4+
slug: /observability
5+
---
6+
7+
# Observability
8+
9+
[AgentOps](https://www.agentops.ai/) is a platform that helps you to monitor and analyze the behavior and performance of your agents.
10+
TaskWeaver is integrated with AgentOps, so you can use TaskWeaver to generate code and use AgentOps to observe all the events from the planning to the code execution.
11+
12+
## Get Started
13+
14+
TaskWeaver integration with AgentOps is documented [here](https://www.agentops.ai/docs/taskweaver).
15+
16+
To get started, you need an API key from AgentOps by [signing up](https://www.agentops.ai/signup) for a free account. Then you need to create a new project and generate the API key from the project settings.
17+
18+
Once you have the API keys, the following steps will help you to get started.
19+
20+
1. Install the `agentops`` package in your project.
21+
```bash
22+
pip install agentops
23+
```
24+
2. Import the `agentops` package in your project.
25+
```python
26+
import agentops
27+
```
28+
3. Initialize the `agentops` client with your API key.
29+
```python
30+
agentops.init(api_key="your_api_key")
31+
```
32+
33+
:::note
34+
You can also set the tags here to track your sessions in the AgentOps dashboard. By default, the "taskweaver" tag will be added to all the AgentOps sessions for TaskWeaver.
35+
:::
36+
37+
4. Import the TaskWeaver handler `TaskWeaverEventHandler` and set it as the event handler for your TaskWeaver project.
38+
```python
39+
from agentops.providers.taskweaver import TaskWeaverEventHandler
40+
handler = TaskWeaverEventHandler()
41+
```
42+
43+
:::note
44+
There are two ways to set the handler:
45+
- Set the handler using the `session.event_handler.register()` method.
46+
- Set the handler in every `session.send_message` function call in the `event_handler` parameter.
47+
:::
48+
49+
:::warning
50+
If you encounter "stuttering" in the recorded messages from the events, it is because the handler has been set with more than one instance.
51+
:::
52+
53+
5. Now you can run your TaskWeaver project and observe the events in the AgentOps dashboard. When the client is initialized, it will automatically start to track the events and provide a link to the session in the AgentOps dashboard.
54+
55+
:::info
56+
It is important to note that the LLM calls and the other events are tracked together in different modules of AgentOps. By default, all the LLM calls are tracked when the AgentOps client is initialized. However, the other events are only tracked when the `TaskWeaverEventHandler` handler is used in conjunction within the code.
57+
58+
Therefore, event observability is limited to LLM calls when TaskWeaver is used in [Terminal mode](../usage/cmd.md).
59+
:::

website/sidebars.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ const sidebars = {
149149
'advanced/code_verification',
150150
'advanced/cli_only',
151151
'advanced/telemetry',
152+
'advanced/observability',
152153
'advanced/plugin_only'
153154
],
154155
}

0 commit comments

Comments
 (0)