Search Feature: URLs usually end with “:”

[XHyperDEVX]

What happened?

When I use the new search feature, in my case “serper” searches for domains and librechat then tries to retrieve them with “firecrawl”. Unfortunately, the URLs are unfortunately provided with a “:” suffix, which makes the URL invalid.

Therefore firecrawl gets a 404 error:

Instead of the correct page:

Version Information

ghcr.io/danny-avila/librechat-dev latest 7ee5765fe941 10 hours ago 1.04GB

Steps to Reproduce

1. Set Up Search Feature with serper and firecrawl-simple (selfhosted)
2. Use it

What browsers are you seeing the problem on?

Chrome

Relevant log output

2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/: Firecrawl API request failed: Invalid URL
2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/: 
401 error, refreshing token
2025-05-25 10:11:40 error: Error scraping https://status.librechat.ai/: Firecrawl API request failed: Invalid URL
2025-05-25 10:11:40 error: Error scraping https://status.librechat.ai/: 
401 error, refreshing token
2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/docs/local/docker: Firecrawl API request failed: Invalid URL
2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/docs/local/docker: 
401 error, refreshing token
2025-05-25 10:11:40 error: Error scraping https://github.com/danny-avila/LibreChat: Firecrawl API request failed: Invalid URL
2025-05-25 10:11:40 error: Error scraping https://github.com/danny-avila/LibreChat: 
401 error, refreshing token
2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/docs: Firecrawl API request failed: Invalid URL
2025-05-25 10:11:40 error: Error scraping https://www.librechat.ai/docs:

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

The URL is not ending with the colon :, this is part of the error message, i.e.: Error scraping ${result.url}: ${result.content}

You are getting a 401 error, which is an authorization issue, usually.

Make sure the Firecrawl API key is correct

[XHyperDEVX]

I think I was able to solve the problem, but now I have another one. I have now set up the normal Firecrawl image, LibreChat also gets a connection to it, starts the jobs, but they time out. The Firecrawl container issues the following error:
error [:]: Error in scrapeController: Error: Job wait {“jobId”: “643c4f56-6135-4601-83fe-e1154f0dfabe”, “scrapeId”: “643c4f56-6135-4601-83fe-e1154f0dfabe”, “startTime”:1748183841070}
But you probably can't help me here, right?

[danny-avila]

No, a local firecrawl installation is not tested yet, though I did setup an API URL config in anticipation of it. The one that firecrawl officially references is very bloated and full of problems.

This is the one I'll be testing, but haven't gotten around to: https://github.com/devflowinc/firecrawl-simple