Share your custom botPolicy.json !

[Albirew]

Let's happily share our custom botPolicy.json for each of our cases!

Rules:

1. USE TAGS! Example below
2. name your additions use. It's better to know the scope of your changes
3. add only your additions. No need to paste what's already in main botPolicies.json
4. use comments to post additions, and answers to discuss and/or improve a specific addition

Exemple to copy-paste:

Here's mine fine addition, I use it for {app_name}:
```json
    {
      "name": "great-addition",
      "path_regex": "^/such regex, wow!$",
      "action": "ALLOW"
    },
```

[Albirew]

And without ado, here's my Forgejo / Gitea addition:

    {
      "name": "forgejo-feeds",
      "path_regex": "^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*[.](rss|atom)$",
      "action": "ALLOW"
    },

last edited: 2025-06-09

[aukthewolf]

I fixed this by removing the backslash, so it instead says:

    {
      "name": "forgejo-feeds",
      "path_regex": ".(rss|atom)$",
      "action": "ALLOW"
    },

I'm pretty sure it works for me now (I'm using this to place the challenge in front of everything except my rss feed on my blog). Tysm for giving me a config to start with lol

[Albirew]

without escaping, the dot means "any character" in regex, so it will accept /whatever.rss as well as /whateverss or /blog/?article=use-anubis-to-protect-your-rss

[guest20]

If you're doing regexes inside json it's usually easier to use a character class [] to escape characters that have special meaning:

{
"path_regex": "[.]json$"    #<- a dot, then a j, an s, an o and an n at the end of a line 
}

Since \ is special to both in json and regexes you need to \ the \ to get a \ to take the special meaning from .
[] is only special to regexes, so it doesn't need a \ to get the power to take the magic away from ..

You can't take away magic from \ with [], since \ takes the magic away from ] first. Also, ^ and - in [] have special meaning, so look out for that... besides that, regex escaping with [] can save you typing a bunch of \s

[Albirew]

Many thanks!
i knew both had escaping characters, but haven't associated both had to be applied one after another...
i updated my 1st post with working regex ^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*[.](rss|atom)$

[aukthewolf]

thank y'all so much btw! I honestly kinda just did a thing and it worked fine for me (I'm fairly new to regex and I barely understand how it works lmao)

I did update my regex to the working one so it'd actually only allow the rss/atom feeds (not that I currently have any pages ending in *rss or *atom)

[Xe]

I love this. I'm pinning this and this will directly feed into the docs. Thank you so much for making this thread @Albirew!

[Pofilo]

My robots.txt is quite explicit:

User-agent: *
Disallow: /

So I don't want any crawler/bot to reach my Gitea instance (but I'm a bit too aggressive with wget, curl, etc..)

{
  "bots": [
    {
      "name": "robots-txt",
      "path_regex": "^/robots.txt$",
      "action": "ALLOW"
    },
    {
      "name": "internal-traffic",
      "remote_addresses": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
      "action": "ALLOW"
    },
    {
      "name": "generic-challenging",
      "user_agent_regex": "(?i)(bot|spider|crawl|fetch|scrapy|wget|curl|python-requests|libwww|Java|Go-http-client)",
      "action": "CHALLENGE",
      "challenge": {
        "difficulty": 16,
        "report_as": 4,
        "algorithm": "slow"
      }
    },
    {
      "name": "ai-bots",
      "user_agent_regex": "(?i)(AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo Bot|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|PanguBot|Perplexity-User|PerplexityBot|PetalBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YouBot)",
      "action": "CHALLENGE",
      "challenge": {
        "difficulty": 16,
        "report_as": 4,
        "algorithm": "slow"
      }
    },
    {
      "name": "default",
      "user_agent_regex": ".*",
      "action": "CHALLENGE"
    }
  ]
}

The list of user agents for IA crawlers comes from #249.

[Albirew]

thanks for your contribution!
i believe botPolicy works on "stops at first validated rule" so i'd put robot-txt before generic-challenging (and ia-bots right after internal-traffic) so it can be served before challenge =)

[Leseratte10]

If this is intended as a copy-paste example for others, I'd extend the "internal-traffic" rule with fc00::/7, both to whitelist IPv6 ULA addresses and to remind the user to add their own (public) IPv6 range in there, too if they have one, instead of just IPv4.

[Dryusdan]

I update "generic-browser" to include Opera (like #220)

    {
      "name": "generic-browser",
      "user_agent_regex": "Mozilla|Opera",
      "action": "CHALLENGE"
    }

[lotharsm]

I added a new ACCEPT policy for the NetSurf browser which identifies itself with both Mozilla/5.0 and Netsurf. Since it is not compatible with the POW challenge, I need to add it before the generic-browser block.

    {
      "name": "netsurf-browser",
      "user_agent_regex": "NetSurf",
      "action": "ALLOW"
    },

[jvoisin]

Something like this to allow feed readers to work:

    {
      "name": "feeds",
      "user_agent_regex": "feedbin|feedly|inoreader|miniflux|newsblur|rss|thunderbird",
      "path_regex": "([./]rss|[./]atom|/feed/?)$",
      "action": "ALLOW"
    },

It would be nice to have it upstreamed, so that feed-reader won't have to have some brittle logic to bypass anubis, as suggested in miniflux/v2#3382 for example.

[Albirew]

nice for blog protection while allowing rss feed readers!
btw, are both conditions user_agent_regex and path_regex required in this configuration, or is it either one of them? (AND or OR?)

[jvoisin]

It's an and, just to be on the safe side.

[guest20]

It's not clear to me why you keep using the word "bypass" when that issue is about actually submitting the proof of work.

That's compliance not a bypass.