Skip to content

Commit 586324a

Browse files
committed
add doc
1 parent 50f3232 commit 586324a

File tree

10 files changed

+100
-6
lines changed

10 files changed

+100
-6
lines changed

website/blog/authors.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
liqli:
2+
name: Liqun Li
3+
url: https://liqul.github.io
4+
title: Principal Researcher
5+
image_url: https://liqul.github.io/assets/logo_small_bw.png
6+
7+
xu:
8+
name: Xu Zhang
9+
url: https://scholar.google.com/citations?user=bqXdMMMAAAAJ&hl=zh-CN
10+
title: Senior Researcher
11+
image_url: https://scholar.googleusercontent.com/citations?view_op=view_photo&user=bqXdMMMAAAAJ&citpid=3

website/blog/evaluation.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# How to evaluate a LLM agent?
1+
---
2+
title: How to evaluate a LLM agent?
3+
authors: [liqli, xu]
4+
date: 2024-05-07
5+
---
26

37
## The challenges
48
It is nontrivial to evaluate the performance of a LLM agent.

website/blog/experience.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Experience selection
1+
---
2+
title: Experience Selection in TaskWeaver
3+
authors: liqli
4+
date: 2024-09-14
5+
---
26

37
We have introduced the motivation of the `experience` module in [Experience](/docs/customization/experience)
48
and how to create a handcrafted experience in [Handcrafted Experience](/docs/customization/experience/handcrafted_experience).

website/blog/local_llm.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Run TaskWeaver with Locally Deployed Not-that-Large Language Models
1+
---
2+
title: Run TaskWeaver with Locally Deployed Not-that-Large Language Models
3+
authors: liqli
4+
date: 2024-07-08
5+
---
26

37
:::info
48
The feature introduced in this blog post can cause incompatibility issue with the previous version of TaskWeaver

website/blog/plugin.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Plugins In-Depth
1+
---
2+
title: Plugins In-Depth
3+
authors: liqli
4+
date: 2024-05-23
5+
---
26

37
_**Pre-requisites**: Please refer to the [Introduction](/docs/plugin/plugin_intro) and the [Plugin Development](/docs/plugin/how_to_develop_a_new_plugin)
48
pages for a better understanding of the plugin concept and its development process._

website/blog/reasoning.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# What makes a good agent reasoning framework?
1+
---
2+
title: What makes a good agent reasoning framework?
3+
authors: liqli
4+
date: 2025-01-20
5+
---
26

37
An agent can listen to the user's request, understand the context, make plans, take actions, observe the results, and respond to the user. Its behavior is driven by the reasoning process, which is the core of the agent's intelligence.
48

website/blog/role.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Roles in TaskWeaver
1+
---
2+
title: Roles in TaskWeaver
3+
authors: [liqli, xu]
4+
date: 2024-05-07
5+
---
26

37

48
We frame TaskWeaver as a **code-first** agent framework. The term "code-first" means that the agent is designed to

website/blog/vision.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: Support vision input for the Planner
3+
authors: liqli
4+
date: 2025-03-13
5+
---
6+
7+
## Introduction
8+
9+
We have supported vision input for the Planner role in TaskWeaver.
10+
The Planner role is responsible for generating the high-level plan for the task.
11+
The vision input is a new type of input that contains images.
12+
This feature is useful when the task requires visual understanding.
13+
<!-- truncate -->
14+
15+
## How vision input is supported in TaskWeaver
16+
17+
In TaskWeaver, we added a new role called `ImageReader` to read images and provide the image url (for remote images) or
18+
the image encoded in base64 (for local images) to the Planner role.
19+
To have this new role, you need to include it in the project configure file as follows:
20+
21+
```json
22+
{
23+
"session.roles": [
24+
"planner",
25+
"code_interpreter",
26+
"image_reader"
27+
]
28+
}
29+
```
30+
31+
The ImageReader role takes the path or the url of the image as input and prepares a response Post for the Planner role. As described [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest) for Azure OpenAI API, if the image is local, ImageReader need to encode the image in base64 and pass it to the API. If the image is remote, ImageReader need to provide the url of the image.
32+
The Planner role can then use the image information for various tasks.
33+
34+
## Example
35+
36+
Let's ask the agent to describe any uploaded image.
37+
38+
39+
![image_reader](../static/img/image_reader.png)
40+
41+
The flow of the conversation is as follows:
42+
```mermaid
43+
graph TD
44+
User --image path--> Planner
45+
Planner --image path--> ImageReader
46+
ImageReader --image encoded in Base64--> Planner
47+
Planner --response--> User
48+
```
49+
50+
In the example above, the User talks to the agent in Web UI and uploads an image.
51+
TaskWeaver also support providing the image path in console mode, either using the `/load` command or just include
52+
the image path in the input message.
53+
54+
55+
56+
57+

website/docusaurus.config.js

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,8 @@ const config = {
5858
// Remove this to remove the "edit this page" links.
5959
editUrl:
6060
'https://github.com/microsoft/TaskWeaver/tree/main/website',
61+
blogSidebarTitle: 'All posts',
62+
blogSidebarCount: 'ALL',
6163
},
6264
theme: {
6365
customCss: './src/css/custom.css',
1.7 MB
Loading

0 commit comments

Comments
 (0)