Skip to content

Conversation

@MuriloFP
Copy link
Contributor

@MuriloFP MuriloFP commented Jun 26, 2025

Related GitHub Issue

Closes: #1430

Description

Problem

URLs provided through @ mentions were failing to load with "Navigation timeout of 10000 ms exceeded" errors, preventing users from fetching web content for AI assistance.

Solution

  • Increased timeout limits: Extended UrlContentFetcher timeout from 10s to 30s and BrowserSession timeout from 7s to 15s for better reliability with slow-loading websites
  • Added fallback retry logic: If networkidle2 fails, automatically retry with domcontentloaded only (20s timeout) to handle JavaScript-heavy sites
  • Improved error handling: Enhanced error messages with specific guidance for timeout, DNS, network, and HTTP status errors
  • Enhanced browser compatibility: Added Chrome launch arguments (--no-sandbox, --disable-web-security, etc.) and set proper viewport/headers to reduce blocking by websites

Files Changed

  • src/services/browser/UrlContentFetcher.ts: Increased timeout, added retry logic, enhanced browser setup
  • src/services/browser/BrowserSession.ts: Increased navigation timeouts for consistency
  • src/core/mentions/index.ts: Improved error handling with user-friendly messages

Impact

Users can now successfully fetch content from slower websites and receive clearer error messages when issues occur, significantly improving the @ mention URL feature reliability.

Test Procedure

Mention an URL and ask for a summary, for example. I tested with "Give me a summary of @#1430".

Before my changes, I got the same result as the issue, with the timeout error.

After the changes, it loaded successfully and the model was able to provide an answer. I tested with models that aren't able to access URL content too, to make sure they were getting the info from our fetch URL tool.

Pre-Submission Checklist

  • Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
  • Scope: My changes are focused on the linked issue (one major feature/fix per PR).
  • Self-Review: I have performed a thorough self-review of my code.
  • Testing: New and/or updated tests have been added to cover my changes (if applicable).
  • Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
  • Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Screenshots / Videos

Documentation Updates

Additional Notes

Also added better error handling for common types of failed fetch URL errors.

Get in Touch

@MuriloFP


Important

Fix URL loading timeout issues by increasing timeouts, adding retry logic, and improving error handling in UrlContentFetcher, BrowserSession, and index.ts.

  • Behavior:
    • Increased UrlContentFetcher timeout from 10s to 30s and BrowserSession timeout from 7s to 15s.
    • Added retry logic in UrlContentFetcher to fallback to domcontentloaded if networkidle2 fails.
    • Improved error handling in parseMentions() in index.ts with specific messages for timeout, DNS, network, and HTTP errors.
  • Browser Configuration:
    • Added Chrome launch arguments in UrlContentFetcher to improve compatibility.
    • Set viewport and headers in UrlContentFetcher to reduce website blocking.
  • Files Changed:
    • UrlContentFetcher.ts: Increased timeout, added retry logic, enhanced browser setup.
    • BrowserSession.ts: Increased navigation timeouts for consistency.
    • index.ts: Improved error handling with user-friendly messages.

This description was created by Ellipsis for 40156b6. You can customize this summary. It will automatically update as commits are pushed.

## Problem
URLs provided through @ mentions were failing to load with "Navigation timeout of 10000 ms exceeded" errors, preventing users from fetching web content for AI assistance.

## Solution
- **Increased timeout limits**: Extended UrlContentFetcher timeout from 10s to 30s and BrowserSession timeout from 7s to 15s for better reliability with slow-loading websites
- **Added fallback retry logic**: If networkidle2 fails, automatically retry with domcontentloaded only (20s timeout) to handle JavaScript-heavy sites
- **Improved error handling**: Enhanced error messages with specific guidance for timeout, DNS, network, and HTTP status errors
- **Enhanced browser compatibility**: Added Chrome launch arguments (--no-sandbox, --disable-web-security, etc.) and set proper viewport/headers to reduce blocking by websites

## Files Changed
- `src/services/browser/UrlContentFetcher.ts`: Increased timeout, added retry logic, enhanced browser setup
- `src/services/browser/BrowserSession.ts`: Increased navigation timeouts for consistency
- `src/core/mentions/index.ts`: Improved error handling with user-friendly messages

## Impact
Users can now successfully fetch content from slower websites and receive clearer error messages when issues occur, significantly improving the @ mention URL feature reliability.

Fixes RooCodeInc#1430
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Jun 26, 2025
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jun 26, 2025
@daniel-lxs daniel-lxs moved this from Triage to PR [Needs Prelim Review] in Roo Code Roadmap Jun 27, 2025
@hannesrudolph hannesrudolph added PR - Needs Preliminary Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Jun 27, 2025
MuriloFP and others added 3 commits June 27, 2025 12:19
Fixes inconsistent error handling where errorMessage was safely extracted
but error.message was still accessed directly in conditionals, which
could cause runtime errors if error.message was undefined.
- Move timeout values to named constants for better maintainability
- Increase timeouts: URL fetch (30s), fallback (20s), browser navigation (15s)
- Improve error handling with cleaner code structure
- Add internationalized error messages for all supported languages
- Fix unsafe error.message access issue noted in PR review
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jun 27, 2025
- Removed --disable-web-security flag that bypasses CORS and CSP
- Removed --no-sandbox flag that disables Chrome's security sandbox
- Removed --disable-setuid-sandbox flag that weakens process isolation

These flags were not necessary for resolving the URL loading timeout issues
and introduced significant security risks. The timeout issues are better
addressed through the increased timeouts and retry logic already in the PR.
- Fix error handling to check if error is an Error instance before accessing .message
- Improve retry logic to only retry for timeout/network errors
- Add comprehensive test coverage for URL fetching and error handling
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jun 27, 2025
- Fixed cheerio mock to return a callable function with html method
- Fixed turndown mock to use a proper class constructor
- Fixed error handling in parseMentions for non-Error objects
- Added retry logic for unknown error types in UrlContentFetcher
- Use path.join() in test expectation to match the implementation
- Fixes test failure on Windows due to backslash vs forward slash differences
…Fetcher

- Replace manual error type checking with serialize-error package
- Simplifies error handling logic and ensures all error types are handled consistently
- Remove unnecessary console.warn as serialize-error handles edge cases
- Update tests to match new behavior
@daniel-lxs daniel-lxs moved this from PR [Needs Prelim Review] to PR [Needs Review] in Roo Code Roadmap Jun 30, 2025
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 30, 2025
@mrubens mrubens merged commit 6427916 into RooCodeInc:main Jun 30, 2025
17 checks passed
@github-project-automation github-project-automation bot moved this from PR [Needs Review] to Done in Roo Code Roadmap Jun 30, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Jun 30, 2025
hannesrudolph pushed a commit that referenced this pull request Jul 3, 2025
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Daniel Riccio <[email protected]>
utarn pushed a commit to modelharbor/ModelHarbor-Agent that referenced this pull request Jul 4, 2025
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Daniel Riccio <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer PR - Needs Review size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Unable to load URLs

4 participants