feat: Add BrightData Unlocker tool component #5075
Conversation
Adds BrightData Unlocker component for web scraping and search capabilities. Features: - Dual input support: URLs or search queries - Automatic URL vs query detection - Utilizes BrightData's unlocking capabilities
|
@jimjimovich @matthias @appleboy @Getty I created a completely new component for web scraping after you had problems while trying the previous component package (I also created 1 node instead of 3) Thanks, |
| @@ -1,40 +1,40 @@ | |||
| ### Responsible Disclosure Policy | |||
| ### Responsible Disclosure Policy | |||
There was a problem hiding this comment.
can we keep unrelated changes out of this PR?
There was a problem hiding this comment.
Do you just want the commit not to contain it?
| | `S3_FORCE_PATH_STYLE` | 设置为 true 以强制请求使用路径样式寻址 | 布尔值 | false | | ||
| | `SHOW_COMMUNITY_NODES` | 显示由社区创建的节点 | 布尔值 | | | ||
| | `DISABLED_NODES` | 从界面中隐藏节点（以逗号分隔的节点名称列表） | 字符串 | | | ||
| | 变量名 | 描述 | 类型 | 默认值 | |
There was a problem hiding this comment.
can we keep unrelated changes out of this PR?
| this.category = 'Tools' | ||
| this.description = 'Search the web and unlock any website.' | ||
| this.inputs = [ | ||
| { |
There was a problem hiding this comment.
you can delete this, because the goal is to have LLM automatically generate the query, refer to BraveSearch and others.
There was a problem hiding this comment.
What do you mean by that? is that the same concerned that the agent will not be able to generate URLs?
If so, I checked it by connecting to the agent - if you give it a system prompt / ask in the regular prompt to use the serp function and then insert the URLs only into the scraping tool it works. I wanted the check to be easier for you with less code so I made it 1 component.
| description: 'Your BrightData API key (Bearer token)' | ||
| }, | ||
| { | ||
| label: 'Zone', |
There was a problem hiding this comment.
how to find this? please add docs link to the description. Refer to other credential how this is done
There was a problem hiding this comment.
will add references to the website
| } | ||
| } | ||
|
|
||
| private constructGoogleSearchUrl(query: string): string { |
There was a problem hiding this comment.
seems like everytime its just going to execute google search, because LLM will surely not generate a URL as input in line 40
There was a problem hiding this comment.
I checked it by connecting to the agent - if you give it a system prompt / ask in the regular prompt to use the serp function and then insert the URLs only into the scraping tool it works. I wanted the check to be easier for you with less code so I made it 1 component.
|
why not just create the MCP integration? https://brightdata.com/ai/mcp-server #5002 is a good reference |
|
Hey @HenryHengZJ , thanks for checking our integration. Regarding the last comment - we will later create an MCP integration also, but we first want to create a tool integration. I will address the code seperatly. |
Follow-up on PR #4700
Add 1 BrightData Unlocker component for web scraping and web search.
Credentials: User needs to paste his API, and one of his zone names (the zones listed per user in the bright data website).
Component Usage:
input is parsed, if query:
Added to google search base URL, sent to Bright Data API and returns SERP content.
if parsed as URL:
Sent to Bright Data API and return scraped content
Screenshots from usage:
