This Discourse plugin adds an integration with Known Agents, previously called: Dark Visitors. Via Known Agents you will get some insights into which bots or scrapers visit your forum.
It provides the following features:
- augmenting robots.txt
- server analytics
- client analytics
In order to use this plugin you need to sign up with Known Agents.
For more information and discussion see this thread on the Discourse Meta forum.
The above status is based on the plugin's executed tests against the specified Discourse branch. It is no definite guarantee that there are no issues.
With this enabled the robots.txt file created by Discourse will be augmented with agents from the configured categories. Once a day the latest list of agents is retrieved and the robots.txt is updated accordingly. Only agents which are not already registered in the robots.txt are added.
This can be used to instruct AI scrapers and other bots to exclude your forum.
This feature only works if you have not manually overridden robots.txt.
Requests to the server are reported to Known Agents.
This feature can be enabled for everybody, or only unauthenticated users (recommended).
A javascript based tracker is added to the forum which, under certain conditions, will report back to Known Agents.
At the moment of writing under the following conditions trigger a callback to Known Agents:
- User is referred to the forum from an AI service
- The browser might be a scraper
This feature can be enabled for everybody, or only unauthenticated users.
