Other IP-Lists #8
Description
Greetings.
Thank you for gathering this information 👍
We also have a project where we list some of the IP-List API's: https://github.com/O-X-L/open-bot-list
You could copy some to your workflow.
It might make sense to separate the AI-training-crawlers from the other ones as they might not be seen as 'good-ones' but the AI-user-crawlers might be important for user-search (?).
Note - these could be simplified:
- DuckDuckGo has some good API-endpoints: Duckduckbot, Duckduckassist (AI) - instead of parsing the MD-file
- Bunny-CDN seems to also have a plain-IPv6 endpoint => https://bunnycdn.com/api/system/edgeserverlist/IPv6/plain