Commit 4322b3e
committed
feat(fetch): add distill parameter for token optimization
Add distill parameter to aggressively clean HTML before processing:
- Remove scripts, styles, navigation, headers, footers
- Remove ads, sidebars, popups, cookie banners
- Remove social widgets and non-content elements
- Normalize whitespace
Typical token reduction: 60-85%
This is an opt-in feature (distill=false by default) to maintain
backward compatibility.
Removes security-related code that belongs in a separate PR.1 parent 3f4ae21 commit 4322b3e
File tree
6 files changed
+128
-1093
lines changed- src/fetch
- src/mcp_server_fetch
- tests
6 files changed
+128
-1093
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | 19 | | |
21 | 20 | | |
22 | 21 | | |
| |||
34 | 33 | | |
35 | 34 | | |
36 | 35 | | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
| 36 | + | |
0 commit comments