Poison AI with anubis, nepenthes and nginx auth-subrequest module

[Sommerwiesel]

I just recently found out how, instead of simply blocking Ai scarpers and bad actors with anubis, you can instead poison them with garbage text, powered by nepenthes and markov babbler.

Important: You'll have to set up nepenthes (or a similar) tarpit first.

You'll also only gonna need one anubis instance for all of your services thanks to nginx auth-subrequest module. Don't forget whitelisting your domains in the anubis env var REDIRECT_DOMAINS.

/etc/nginx/nginx.conf

upstream anubis {
    server 127.0.0.1:8080 fail_timeout=3s;
}

# nepenthes or similar ai poisoning tool
upstream bot_honeypot {
    server 127.0.0.1:8893 fail_timeout=30s;
}

# We need these two maps so that nepenthes always creates valid maze links
map $uri $uri_trimmed {
    ~^(.+)/$ $1;
    default  $uri;
}

map $uri $x_prefix {
    ""      "links";
    "/"     "links";
    default $uri_trimmed;
}

/etc/nginx/snippets/ai_poison.conf

location @botHoneypot {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Important for dynamic content sites such as redlib
    proxy_set_header X-Prefix $x_prefix;

    # It's important to not buffer the response so that the AI scapers don't do a refresh after not getting any data after a few secs
    proxy_buffering off;

    proxy_pass http://bot_honeypot;
    proxy_pass_request_body off;
    proxy_set_header content-length "";
    auth_request off;
}

vhost config (partially)

location /.within.website/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass http://anubis;
    proxy_pass_request_body off;
    proxy_set_header content-length "";
    auth_request off;
}

location @redirectToAnubis {
    return 307 /.within.website/?redir=$request_uri;
    auth_request off;
}
    
include snippets/ai_poison.conf;
    
location / {
    auth_request /.within.website/x/cmd/anubis/api/check;
    error_page 401 = @redirectToAnubis;
    error_page 403 = @botHoneypot;
    [...]

anubis docker compose.yaml

services:
  anubis:
    container_name: "anubis"
    cpus: 2
    mem_limit: "256m"
    restart: unless-stopped
    image: ghcr.io/techarohq/anubis:latest
    logging:
      driver: json-file
      options:
        max-size: 128m
        max-file: 3
    environment:
      BIND: ":8080"
      DIFFICULTY: "4"
      METRICS_BIND: ":9090"
      SERVE_ROBOTS_TXT: "true"
      TARGET: " "
      POLICY_FNAME: "/data/cfg/botPolicy.yaml"
      REDIRECT_DOMAINS: "invidious.nerdvpn.de,wiki.nerdvpn.de,fandom.nerdvpn.de,reddit.nerdvpn.de,imgur.nerdvpn.de,tumblr.nerdvpn.de,quora.nerdvpn.de,imdb.nerdvpn.de,soflow.nerdvpn.de"
      COOKIE_DOMAIN: "nerdvpn.de"
    volumes:
      - "./botPolicy.yaml:/data/cfg/botPolicy.yaml:ro"

anubis botPolicy.yaml

# optional: create a ua to test anubis with
- name: test-anubis
    user_agent_regex: >-
      AnubisTest
    action: DENY

# required: change status codes
status_codes:
  CHALLENGE: 200
  DENY: 403

# insert your anubis policy or the default policy (https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.yaml) after here 
[...]

And done.

Short explaination:

1. Request first gets checked by anubis (using nginx auth subrequest)
2. On ALLOW or CHALLENGE, it's as usual with anubis
3. On DENY, anubis returns 403, which nginx catches and passes to nepenthes
4. Nepenthes always returns 200, thus the AI thinks it's got a valid response and scrapes the markov babbler garbage

Tested on my redlib instance with user agent changed to "AnubisTest":

Works on dynamic redlib paths too thanks to X-Prefix header:

Have fun poisoning the well!

[Sommerwiesel]

One day later, works like a charm:

cat nepenthes/data/statsfile.json | jq

"bd7b2e8197d3c72c297560d7ba2e350d4a4dd6a4": {
    "first_seen": 1751357869,
    "agstring": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",
    "hits": 121,
    "last_seen": 1751359308
},
"4dcf72f5cf82927a9bac1117e65d613229f86843": {
    "first_seen": 1751297773,
    "last_seen": 1751348517,
    "agstring": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)",
    "hits": 84
},

small excerpt, the file is much bigger with much more ai crawlers getting what they fucking deserve