Use of local path

[MentalGear]

Is it possible to use a local FS path for crawlee, like:

await crawler.run(["src/test-data.html"])

[B4nan]

Definitely not with a relative path, but you could try an absolute file URL (e.g. file:///Users/adamek/htdocs/test.html - just open the file you have locally in your browser and copy the URL that gets resolved), if that won't work, you could use a local web server.

[B4nan]

Try something like this:

https://www.npmjs.com/package/http-server

[MentalGear]

Thx, I used http-server to spin up a local server in another shell. It works fine and I can access the page, but trying to reach the server from crawle does not work.

await crawler.run(["http://localhost:8080/index.html"])

Have you ever tried inputting a local(host) url into it?

[B4nan]

Are you on mac? There are some restrictions on how local proxies are handled (on system level), maybe it's about that. What crawler type are you using?

[MentalGear]

I'm indeed on mac 10.15. using the CheerioCrawler.

[B4nan]

Try a browser maybe, maybe it will work even with the file URLs (as browsers do support them).