tool: add web fetch #722

liuzengh · 2025-11-21T08:09:10Z

No description provided.

liuzengh · 2025-11-21T09:31:22Z

tool/webfetch/httpfetch/fetch.go

+	maxURLs        = 20
+)
+
+// Option configures the WebFetch tool.


这边再加一个 domain filter 功能，应该就差不多了 https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool#how-to-use-web-fetch

已增加url filter 功能，该功能是 domain filter 的超集

codecov · 2025-11-24T02:51:29Z

Codecov Report

❌ Patch coverage is 95.20548% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.33912%. Comparing base (42a1bba) to head (d4f29da).

Files with missing lines	Patch %	Lines
tool/webfetch/httpfetch/fetch.go	95.65217%	4 Missing and 3 partials ⚠️
tool/webfetch/geminifetch/fetch.go	93.50649%	3 Missing and 2 partials ⚠️
tool/webfetch/internal/urlfilter/filter.go	96.29630%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##                main        #722         +/-   ##
===================================================
+ Coverage   87.28073%   87.33912%   +0.05838%     
===================================================
  Files            276         279          +3     
  Lines          35914       36206        +292     
===================================================
+ Hits           31346       31622        +276     
- Misses          2987        2996          +9     
- Partials        1581        1588          +7

Flag	Coverage Δ
unittests	`87.33912% <95.20548%> (+0.05838%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sandyskies · 2025-11-24T07:37:53Z

tool/webfetch/urlfilter.go

+	"strings"
+)
+
+// URLFilter is a function that determines if a URL should be allowed.


文件名改为url_filter.go

重构下目前的文档结构：后面可以增加 LLM Server-Side 的 WebFetch

webfetch ├── README.md ├── claudefetch │ └── go.mod ├── geminifetch │ └── go.mod ├── httpfetch │ ├── fetch.go │ ├── fetch_real_test.go │ ├── fetch_test.go │ ├── go.mod │ └── go.sum └── internal └── urlfilter ├── filter.go └── filter_test.go

sandyskies · 2025-11-24T07:38:46Z

tool/webfetch/httpfetch/fetch.go

@@ -0,0 +1,362 @@
+//
+// Tencent is pleased to support the open source community by making trpc-agent-go available.


文档，example看看

hyprh · 2025-11-25T03:03:45Z

tool/webfetch/httpfetch/fetch.go

+// WithHTTPClient sets the HTTP client.
+func WithHTTPClient(c *http.Client) Option {
+	return func(cfg *config) {
+		cfg.httpClient = c


我们一些工具的http client，有必要接入内网的http不

可以考虑接入，当作一个 feture，再另外的 PR 里面来修改 http client

hyprh · 2025-11-25T06:24:34Z

tool/webfetch/geminifetch/fetch.go

+// This tool uses Gemini's URL Context feature to fetch and process web content.
+// modelName: the Gemini model-id to use.
+func NewTool(modelName string, opts ...Option) (tool.CallableTool, error) {
+	if modelName == "" {


genminifetch 和 claudefetch 需要抽象一个接口吗，方便后续其他组件，比如knowledge 复用

knowledge 那边需要什么样的接口？目前设计 genminifetch , claudefetch, httpfetch 为独立的三个工具，因为genminifetch 和 claudefetch 的模型 API 并不相同，上层统一的表示的话，要么会丢失一些各个模型特有的东西，要么会导致统一的表示过于臃肿。

OK，可以先这样，不暴露fetch的接口

WineChord · 2025-11-27T09:28:11Z

examples/tool/webfetch/geminifetch/go.mod

@@ -0,0 +1,59 @@
+module trpc.group/trpc-go/trpc-agent-go/examples/tool/webfetch/geminifetch


docs 下的中英文档也可以更新

liuzengh added 4 commits November 21, 2025 16:07

WIP(tool): add web fetch

29dea48

add html to markdown support

90586a3

change to ulrs input parameters

16542fa

add conent limit option

00e0732

liuzengh commented Nov 24, 2025

View reviewed changes

refactor fetchone

b48e364

add url filter

9c58d79

liuzengh changed the title ~~WIP(tool): add web fetch~~ tool: add web fetch Nov 24, 2025

liuzengh added 4 commits November 24, 2025 12:06

update comment

d787b37

Merge remote-tracking branch 'origin/main' into web-fetch

556e884

fix typo and add http client test

0e29993

Merge remote-tracking branch 'origin' into web-fetch

d114731

sandyskies reviewed Nov 24, 2025

View reviewed changes

liuzengh added 9 commits November 24, 2025 15:49

improve code coverage

8d8aed9

add todo for server-side websearh

1ae7f69

Merge remote-tracking branch 'origin/main' into web-fetch

c326483

add gemini webfetch

d26e5f7

Merge remote-tracking branch 'origin/main' into web-fetch

3ae2a9e

add webfetch/httpfetch example

d50cabe

update test

e2e2bcc

add gemini example

9d0af65

add pacakge comment

3f8dc49

liuzengh requested a review from hyprh November 24, 2025 13:37

liuzengh added 2 commits November 25, 2025 10:08

Merge remote-tracking branch 'origin/main' into web-fetch

23ae92e

Merge remote-tracking branch 'origin/main' into web-fetch

42ff59e

hyprh reviewed Nov 25, 2025

View reviewed changes

liuzengh added 2 commits November 25, 2025 11:03

add package comments for urlfilter

6897aa9

add URL filtering tests to fetch_test.go

3b23dfe

hyprh reviewed Nov 25, 2025

View reviewed changes

liuzengh added 2 commits November 25, 2025 14:53

update geminifetch testcase

43c0a20

Merge remote-tracking branch 'origin/main' into web-fetch

d4f29da

liuzengh mentioned this pull request Nov 25, 2025

我们一些工具的http client，有必要接入内网的http不 #746

Open

WineChord reviewed Nov 27, 2025

View reviewed changes

		@@ -0,0 +1,362 @@
		//
		// Tencent is pleased to support the open source community by making trpc-agent-go available.

		@@ -0,0 +1,59 @@
		module trpc.group/trpc-go/trpc-agent-go/examples/tool/webfetch/geminifetch

tool: add web fetch #722

Are you sure you want to change the base?

tool: add web fetch #722

Uh oh!

Conversation

liuzengh commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuzengh Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 24, 2025 •

edited

Loading

liuzengh Nov 25, 2025 •

edited

Loading