Prevent crawling of the parent page or page not in domain

[rushi]

I am using Apify.utils.enqueueLinks and I'd like to prevent URLs that are not in the root domain so I don't crawl anything other than the website itself.

For example if I rushi.com with several pages, and one of those pages links to blog.github.com I don't want to recurse into that URL because github.com != rushi.com

What I'm looking for is Apify's equivalent of wget's --no-parent option. I also want to ensure relative URLs are picked up so baseUrl: request.loadedUrl needs to be present

This is the code I have so far direct from the examples

Apify.main(async () => {
        const requestQueue = await Apify.openRequestQueue();
        await requestQueue.addRequest({ url: "rushi.com" }); // My chosen example domain
        const handlePageFunction = async ({ request, $ }) => {
            console.log(chalk.green(request.url));
                        
            try {
                await Apify.utils.enqueueLinks({
                    $,
                    requestQueue,
                    baseUrl: request.loadedUrl,
                });
            } catch (err) {
                console.log(chalk.red(err.message));
            }
        };
        
        try {
            const crawler = new Apify.CheerioCrawler({
                requestQueue,
                handlePageFunction,
                maxRequestsPerCrawl: 500,
            });
            
            await crawler.run();
        } catch (err) {
            console.log(chalk.yellow(err.message));
        }
    });

[metalwarrior665]

Hello,

One way to do this is to use the pseudoUrls property of Apify.utils.enqueueLinks as you can see here: https://sdk.apify.com/docs/api/utils#utilsenqueuelinksoptions

This is very simple example, the pseudo URL can be made more precise.

// You can dynamically change this
const parentDomain = rushi.com;
await Apify.utils.enqueueLinks({
                    $,
                    requestQueue,
                    baseUrl: request.loadedUrl,
                    pseudoUrls: [new Apify.PseudoUrl(`http[.+]${parentDomain}[.*]`)]
                });

For more lively discussion, I recommend using our Discord community server.

[kenlyle2]

Should http[.+]${parentDomain}[.* work in the Console then?