Does anyone have a database of bots/spiders that attempt to read RSS feeds #432
-
|
I am trying to create an analytics plugin for our podcast system however we are noticing a high number of spiders/bots. While we could use the data we have collected and create a bot / spider database (which we will if there is not one already). I was wondering if there is a known community lead podcast spider/bots database. We are noticing companies like redCircle have a spider which craws our podcasts (we don't use them) - We have already marked their user-agent as a bot. Other bot user-agents (based on testing):
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Yes, there's an open one here managed by @johnspurlock Also, you should take a look at https://op3.dev which will save you from reinventing the wheel. |
Beta Was this translation helpful? Give feedback.
Yes, there's an open one here managed by @johnspurlock
Also, you should take a look at https://op3.dev which will save you from reinventing the wheel.