Yahoo Slurp crawler

[lewark]

I'm posting this under Discussions because, while being unsure whether it merits a full issue ticket or not, I wanted to hear people's thoughts. Since Anubis is used by a growing number of websites (especially within the FOSS community), the crawlers it decides to allow can potentially have a sizable impact on what is accessible from search engines.

In addition to indexes provided by Bing, Yahoo Search also seems to use a separate crawler called Slurp, which is currently not on Anubis' list of allowed bots. The xeiaso.net webpage still appears on Yahoo Search, and I suspect this may be due to Yahoo's use of Bing indexes. However, I still wonder if it might be a good idea to explicitly add Yahoo's crawler to the list, or if there was a reason it hasn't been included.

On the subject, Yahoo writes that the crawler is used for Yahoo Mobile search, as well as the company's news, sports, and finance sites. Like Google, Yahoo Mobile does advertise its own set of AI features, and it wasn't clear to me whether this is accomplished through the Slurp crawler or not. (Dark Visitors does not currently label Slurp as being AI-related.) I imagine whether maintainers wish to allow Slurp may depend on (a) what purpose the bot is actually used for, (b) the ease of keeping associated IP ranges up to date, and (c) how nicely the bot plays with websites.

[Xe]

I emailed Yahoo asking for a list of egress IP addresses. Once I get that, I can allowlist Yahoo slurp.

[lewark]

Thanks for letting me know. Sorry if you were already aware of the issue.

[Xe]

I didn't know Slurp existed until you let me know!

[Xe]

I haven't gotten an email back. Do you know anyone at Yahoo so I can get in contact with an actual human?