Visit specific url tool

[eliott07]

Situation
When users enter a specific url, they expect the Assistant to visit the URL and use its content to generate the answer

Current behavior
When a url is entered, the web search tool is activated but breaks the URL down into a web search like this:
site:https://docs.numerique.gouv.fr doc_id

Problem
When web pages are public but not referenced, this yields no results.

Desired behavior
When a url is detected in the user prompt, visit the website and gets its content to generate answer.

Success indicators

• Detection of URL in user prompt
• Clean acquisition of website content
• Clear messages when main content is not accessible (e.g. Docs require authentication)

[qbey]

@eliott07 : explain the expected behavior (tool..)

[eliott07]

Expected behavior :

• Detect URL in user prompt with regex
• Pass URL through filter : not localhost://, no http://, maybe no foreign websites (.ru ...)
• add a website_visit system prompt if filters pass : "if the user intent is to fetch content from this specific website, this tool can fetch the content from the website if it is in the correct format"

Once the tool runs :

• Get content headers to detect type and choose course of action ( pass through if pdf, rag with document backend if text document or html page)
• execute rag or pass pdf through
• gracefully exit if wrong content type : "You have provided the link to a video, but i cannot analyse that yet. Let the team know if this is in your priorities"

[eliott07]

#195

[eliott07]

Intention analysis : https://docs.numerique.gouv.fr/docs/f1299cfd-6402-46cd-95ab-1e586c20da12/