Dark Visitors rebranded to Known Agents

https://knownagents.com/posts/dark-visitors-is-now-known-agents

The Discourse plugin is still named Dark Visitors.

Author: elmuerte

.gitignore

@@ -0,0 +1 @@
+bin/rspec

Gemfile.lock

@@ -35,7 +35,7 @@ GEM
     prettier_print (1.2.1)
     prism (1.4.0)
     racc (1.8.1)
-    rack (3.2.3)
+    rack (2.2.17)
     rainbow (3.1.1)
     regexp_parser (2.11.2)
     rubocop (1.79.2)

README.md

@@ -1,15 +1,15 @@
-# Discourse Dark Visitors Plugin
+# Dark Visitors: Discourse Known Agents Plugin
 
-This [Discourse](https://discourse.com) plugin adds an integration with [Dark Visitors](https://darkvisitors.com). 
-Via Dark Visitors you will get some insights into which bots or scrapers visit your forum.
+This [Discourse](https://discourse.com) plugin adds an integration with [Known Agents](https://knownagents.com), previously called: _Dark Visitors_. 
+Via Known Agents you will get some insights into which bots or scrapers visit your forum.
 
 It provides the following features:
 
 - augmenting robots.txt 
 - server analytics
 - client analytics
 
-In order to use this plugin you need to sign up with [Dark Visitors](https://darkvisitors.com).
+In order to use this plugin you need to sign up with [Known Agents](https://knownagents.com).
 
 For more information and discussion see [this thread](https://meta.discourse.org/t/dark-visitors/365158) on the Discourse Meta forum.
 
@@ -24,7 +24,7 @@ It is no definite guarantee that there are no issues.
 
 ## Augmenting robots.txt
 
-With this enabled the robots.txt file created by Discourse will be augmented with [agents](https://darkvisitors.com/agents) from the configured categories.
+With this enabled the robots.txt file created by Discourse will be augmented with [agents](https://knownagents.com/agents) from the configured categories.
 Once a day the latest list of agents is retrieved and the robots.txt is updated accordingly.
 Only agents which are not already registered in the robots.txt are added.
 
@@ -34,15 +34,15 @@ This feature only works if you have not manually overridden robots.txt.
 
 ## Server Analytics
 
-Requests to the server are reported to Dark Visitors.
+Requests to the server are reported to Known Agents.
 
 This feature can be enabled for everybody, or only unauthenticated users (recommended).
 
 ## Client Analytics
 
-A javascript based tracker is added to the forum which, under certain conditions, will report back to Dark Visitors.
+A javascript based tracker is added to the forum which, under certain conditions, will report back to Known Agents.
 
-At the moment of writing under the following conditions trigger a callback to Dark Visitors:
+At the moment of writing under the following conditions trigger a callback to Known Agents:
 - User is referred to the forum from an AI service
 - The browser might be a scraper
 

config/locales/server.en.yml

@@ -1,11 +1,11 @@
 en:
   site_settings:
-    darkvisitors_access_token: "Go to the project settings at Dark Visitors and copy the Access Token."
-    darkvisitors_robots_txt_enabled: "Only when enabled will the robots.txt be updated with agents retrieved from Dark Visitors."
-    darkvisitors_robots_txt_agents: "Agent types to include in the robots.txt. See https://darkvisitors.com/agents for an overview of all agents."
+    darkvisitors_access_token: "Go to the project settings at Known Agents and copy the Access Token."
+    darkvisitors_robots_txt_enabled: "Only when enabled will the robots.txt be updated with agents retrieved from Known Agents."
+    darkvisitors_robots_txt_agents: "Agent types to include in the robots.txt. See the <a href=\"https://knownagents.com/agents\">agent list</a> for an overview of all agents and what they are used for."
     darkvisitors_robots_txt_path: "Path not allowed to be visited by the agents."
-    darkvisitors_server_analytics: "Send request data to Dark Visitors for analysis. With 'anonymous_only' only requests form unauthenticated visitors are reported."
-    darkvisitors_server_analytics_include: "Besides standard tracked requests, what other requests should be reported to Dark Visitors."
+    darkvisitors_server_analytics: "Send request data to Known Agents for analysis. With 'anonymous_only' only requests form unauthenticated visitors are reported."
+    darkvisitors_server_analytics_include: "Besides standard tracked requests, what other requests should be reported to Known Agents."
     darkvisitors_server_analytics_ignore: "User agent substrings (case sensitive) to ignore. Use it to exclude uptime monitoring bots."
     darkvisitors_client_analytics: "Add client-side analytics. Redirects from AI services can be traced this way. It also helps to identify possible hidden AI agents."
-    darkvisitors_client_analytics_project_key: "Go to the project settings at Dark Visitors and copy the project_key value from the \"JavaScript Tag\"."
+    darkvisitors_client_analytics_project_key: "Go to the project settings at Known Agents and copy the project_key value from the \"JavaScript Tag\"."

config/settings.yml

@@ -16,21 +16,21 @@ discourse_darkvisitors:
     allow_any: true
     list_type: compact
     choices:
-      - AI Agent
-      - AI Assistant
-      - AI Data Scraper
-      - AI Search Crawler
-      - Archiver
-      - Developer Helper
-      - Fetcher
-      - Headless Agent
-      - Intelligence Gatherer
-      - Scraper
-      - SEO Crawler
-      - Search Engine Crawler
-      - Security Scanner
-      - Undocumented AI Agent
-      - Uncategorized
+    - AI Agent
+    - AI Assistant
+    - AI Data Scraper
+    - AI Search Crawler
+    - Archiver
+    - Developer Helper
+    - Fetcher
+    - Automated Agent
+    - Intelligence Gatherer
+    - Scraper
+    - SEO Crawler
+    - Search Engine Crawler
+    - Security Scanner
+    - Undocumented AI Agent
+    - Uncategorized
   darkvisitors_robots_txt_path:
     default: '/'
     client: false
@@ -74,14 +74,14 @@ discourse_darkvisitors:
     default: false
     hidden: true
   darkvisitors_robots_txt_api:
-    default: "https://api.darkvisitors.com/robots-txts"
+    default: "https://api.knownagents.com/robots-txts"
     client: false
     hidden: true
   darkvisitors_server_analytics_api:
-    default: "https://api.darkvisitors.com/visits"
+    default: "https://api.knownagents.com/visits"
     client: false
     hidden: true
   darkvisitors_client_analytics_script:
-    default: "https://darkvisitors.com/tracker.js"
+    default: "https://knownagents.com/tracker.js"
     client: true
     hidden: true

db/migrate/20260204194031_headless_agent_to_automated_agent.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+class HeadlessAgentToAutomatedAgent < ActiveRecord::Migration[7.2]
+  def up
+    execute "UPDATE site_settings SET value = REPLACE(value, 'Headless Agent', 'Automated Agent') WHERE name = 'darkvisitors_robots_txt_agents' AND value LIKE '%Headless Agent%'"
+  end
+
+  def down
+    execute "UPDATE site_settings SET value = REPLACE(value, 'Automated Agent', 'Headless Agent') WHERE name = 'darkvisitors_robots_txt_agents' AND value LIKE '%Automated Agent%'"
+  end
+end

lib/robots_txt.rb

@@ -20,16 +20,16 @@ def self.on_robots_info(robots_info)
         end
       end
       robots_info[:header] = robots_info[:header] +
-        "\n# Augmented by Dark Visitors on #{config[:last_update]} with #{config[:agents].count} agents"
+        "\n# Augmented by Known Agents on #{config[:last_update]} with #{config[:agents].count} agents"
     end
 
     def self.update_robots_txt
       return unless SiteSetting.darkvisitors_robots_txt_enabled
       if SiteSetting.darkvisitors_access_token == ""
-        Rails.logger.warn "Cannot update robots.txt from Dark Visitors. No access_token configured."
+        Rails.logger.warn "Cannot update robots.txt from Known Agents. No access_token configured."
         return
       end
-      Rails.logger.info "Updating Dark Visitors robots.txt"
+      Rails.logger.info "Updating Known Agents robots.txt"
 
       uri =
         URI(
@@ -48,7 +48,7 @@ def self.update_robots_txt
       }
       response = Net::HTTP.post(uri, request, headers)
       unless response.code == "200"
-        Rails.logger.error "Dark Visitors robots-txt API failure: #{response.code}"
+        Rails.logger.error "Known Agents robots-txt API failure: #{response.code}"
         return
       end
 
@@ -76,7 +76,7 @@ def self.update_robots_txt
         ROBOTS_TXT,
         { last_update: DateTime.now.to_s, agents: agents }
       )
-      Rails.logger.info "Received #{agents.count} agents to deny from Dark Visitors"
+      Rails.logger.info "Received #{agents.count} agents to deny from Known Agents"
     end
   end
 end

plugin.rb

@@ -1,9 +1,9 @@
 # frozen_string_literal: true
 
 # name: discourse-darkvisitors
-# about: Connects to Dark Visitors to keep the robots.txt up to date and monitor crawlers and scrapers visiting your forum.
+# about: Connects to Known Agents (formerly known as Dark Visitors) to keep the robots.txt up to date and monitor crawlers and scrapers visiting your community.
 # meta_topic_id: 365158
-# version: 1.2
+# version: 1.3
 # authors: elmuerte
 # url: https://github.com/magicball-network/discourse-darkvisitors
 # required_version: 3.4.0