Skip to content

Commit 2bd852f

Browse files
authored
feat(tools): add WebScrapeExecutor with scrape-core integration (#99)
* feat(tools): add WebScrapeExecutor with scrape-core integration (#57) Implement safe HTML scraping via scrape-core with CSS selectors: - WebScrapeExecutor: fetch HTTPS URLs, parse HTML with Soup, extract data using CSS selectors (text, html, attr modes) in spawn_blocking - CompositeExecutor<A, B>: generic first-match-wins executor chaining that preserves Agent's single ToolExecutor generic parameter - SSRF protection: HTTPS-only, IPv4/IPv6 private IP blocking including link-local, unique local, IPv4-mapped, broadcast, unspecified - Bundled web-scrape skill with CSS selector examples - Shared extract_fenced_blocks() utility (DRY refactor from shell.rs) - Config: [tools.scrape] with timeout (15s) and max_body_bytes (1MB) Closes #57 Phase 2 * chore: bump version to 0.7.1, update CHANGELOG and README for web scrape
1 parent a247425 commit 2bd852f

File tree

16 files changed

+1203
-50
lines changed

16 files changed

+1203
-50
lines changed

CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
66

77
## [Unreleased]
88

9+
## [0.7.1] - 2026-02-09
10+
11+
### Added
12+
- `WebScrapeExecutor`: safe HTML scraping via scrape-core with CSS selectors, SSRF protection, and HTTPS-only enforcement (Issue #57)
13+
- `CompositeExecutor<A, B>`: generic executor chaining with first-match-wins dispatch
14+
- Bundled `web-scrape` skill with CSS selector examples for structured data extraction
15+
- `extract_fenced_blocks()` shared utility for fenced code block parsing (DRY refactor)
16+
- `[tools.scrape]` config section with timeout and max body size settings
17+
18+
### Changed
19+
- Agent tool output label from `[shell output]` to `[tool output]`
20+
- `ShellExecutor` block extraction now uses shared `extract_fenced_blocks()`
21+
922
## [0.7.0] - 2026-02-08
1023

1124
### Added

0 commit comments

Comments
 (0)