v1.20.0-pre1

[Xe]

v1.20.0 prerelease 1: Thancred Waters

The big ticket items are as follows:

• Implement a no-JS challenge method: metarefresh (#95)
• Implement request "weight", allowing administrators to customize the behaviour of Anubis based on specific criteria
• Implement GeoIP and ASN based checks via Thoth (#206)
• Add custom weight thresholds via CEL (#688)
• Move Open Graph configuration to the policy file
• Enable support for Open Graph metadata to be returned by default instead of doing lookups against the target
• Add robots2policy CLI utility to convert robots.txt files to Anubis challenge policies using CEL expressions (#409)
• Refactor challenge presentation logic to use a challenge registry
• Allow challenge implementations to register HTTP routes

A lot of performance improvements have been made:

• Replace internal SHA256 hashing with xxhash for 4-6x performance improvement in policy evaluation and cache operations
• Optimized the OGTags subsystem with reduced allocations and runtime per request by up to 66%
• Replace cidranger with bart for IP range checking, improving IP matching performance by 3-20x with zero heap
  allocations

And some cleanups/refactors were added:

• Remove the unused /test-error endpoint and update the testing endpoint /make-challenge to only be enabled in
  development
• Add --xff-strip-private flag/envvar to toggle skipping X-Forwarded-For private addresses or not
• Bump AI-robots.txt to version 1.37
• Make progress bar styling more compatible (UXP, etc)
• Add --strip-base-prefix flag/envvar to strip the base prefix from request paths when forwarding to target servers

Request weight is one of the biggest ticket features in Anubis. This enables Anubis to be much closer to a Web Application Firewall and when combined with custom thresholds allows administrators to have Anubis take advanced reactions. For more information about request weight, see the request weight section of the policy file documentation.

TL;DR when you have one or more WEIGHT rules like this:

bots:
  - name: gitea-session-token
    action: WEIGH
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("i_love_gitea=")
    # Remove 5 weight points
    weight:
      adjust: -5

You can configure custom thresholds like this:

thresholds:
  - name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather
    expression: weight < 0 # a feather weighs zero units
    action: ALLOW # Allow the traffic through

  # For clients that had some weight reduced through custom rules, give them a
  # lightweight challenge.
  - name: mild-suspicion
    expression:
      all:
        - weight >= 0
        - weight < 10
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
      algorithm: metarefresh
      difficulty: 1
      report_as: 1

  # For clients that are browser-like but have either gained points from custom
  # rules or report as a standard browser.
  - name: moderate-suspicion
    expression:
      all:
        - weight >= 10
        - weight < 20
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
      algorithm: fast
      difficulty: 2 # two leading zeros, very fast for most clients
      report_as: 2

  # For clients that are browser like and have gained many points from custom
  # rules
  - name: extreme-suspicion
    expression: weight >= 20
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
      algorithm: fast
      difficulty: 4
      report_as: 4

These thresholds apply when no other ALLOW, DENY, or CHALLENGE rule matches the request. WEIGHT rules add and remove request weight as needed:

bots:
  - name: gitea-session-token
    action: WEIGH
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("i_love_gitea=")
    # Remove 5 weight points
    weight:
      adjust: -5

  - name: bot-like-user-agent
    action: WEIGH
    expression: '"Bot" in userAgent'
    # Add 5 weight points
    weight:
      adjust: 5

Of note: the default "generic browser" rule assigns 10 weight points:

# Generic catchall rule
- name: generic-browser
  user_agent_regex: >-
    Mozilla|Opera
  action: WEIGH
  weight:
    adjust: 10

Adjust this as you see fit.

What's Changed

• build(deps-dev): bump esbuild from 0.25.4 to 0.25.5 in the npm group by @dependabot in build(deps-dev): bump esbuild from 0.25.4 to 0.25.5 in the npm group #600
• build(deps): bump docker/build-push-action from 6.17.0 to 6.18.0 in the github-actions group by @dependabot in build(deps): bump docker/build-push-action from 6.17.0 to 6.18.0 in the github-actions group #602
• build(deps): bump github.com/a-h/templ from 0.3.865 to 0.3.887 in the gomod group by @dependabot in build(deps): bump github.com/a-h/templ from 0.3.865 to 0.3.887 in the gomod group #601
• docs(faq): anubis does not mine bitcoin by @Xe in docs(faq): anubis does not mine bitcoin #609
• feat: implement challenge registry by @Xe in feat: implement challenge registry #607
• docs(known-instances): add Alliance of Hessian Libraries by @CryptoCopter in docs(known-instances): add Alliance of Hessian Libraries #611
• docs(subrequest-auth): document required policy changes by @foosinn in docs(subrequest-auth): document required policy changes #613
• docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth" by @jieter in docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth" #615
• fix(lib/challenge): allow challenges to register HTTP routes by @Xe in fix(lib/challenge): allow challenges to register HTTP routes #620
• docs(known-instances): add wiki.dolphin-emu.org to known instances by @lotharsm in docs(known-instances): add wiki.dolphin-emu.org to known instances #626
• feat(lib/challenge): HTTP meta refresh challenge method by @Xe in feat(lib/challenge): HTTP meta refresh challenge method #623
• style: Some minor fixes by @JasonLovesDoggo in style: Some minor fixes #548
• Bump ai.robots.txt to v1.34 by @Dryusdan in Bump ai.robots.txt to v1.34 #632
• build(deps): bump the gomod group with 2 updates by @dependabot in build(deps): bump the gomod group with 2 updates #634
• docs(admin/environments): Prefer IPv6 over IPv4 for apache2 listener directive by @lotharsm in docs(admin/environments): Prefer IPv6 over IPv4 for apache2 listener directive #628
• build(deps): bump github/codeql-action from 3.28.18 to 3.28.19 in the github-actions group by @dependabot in build(deps): bump github/codeql-action from 3.28.18 to 3.28.19 in the github-actions group #635
• Adds ability to toggle off stripping of private addrs from XFF by @dchandekstark in Adds ability to toggle off stripping of private addrs from XFF #619
• Make progress bar styling more compatible (UXP, etc) by @Fierelier in Make progress bar styling more compatible (UXP, etc) #636
• feat(lib): implement request weight by @Xe in feat(lib): implement request weight #621
• Update known-instances.md to include SquirrelJME by @XerTheSquirrel in Update known-instances.md to include SquirrelJME #643
• fix(anubis): nil check policy loading by @JasonLovesDoggo in fix(anubis): nil check policy loading #645
• test: introduce SSH based CI for non-native test hosts by @Xe in test: introduce SSH based CI for non-native test hosts #644
• build(deps): bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 by @dependabot in build(deps): bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 #650
• test(ssh-ci): re-enable GOARCH=ppc64le by @Xe in test(ssh-ci): re-enable GOARCH=ppc64le #651
• fix(gitattributes): update pattern for generated files by @JasonLovesDoggo in fix(gitattributes): update pattern for generated files #652
• fix(ci): conditionally run SSH jobs for TecharoHQ/anubis by @JasonLovesDoggo in fix(ci): conditionally run SSH jobs for TecharoHQ/anubis #654
• feat: add a strip-base-prefix option by @JasonLovesDoggo in feat: add a strip-base-prefix option #655
• refactor(ogtags): optimize URL construction and memory allocations by @JasonLovesDoggo in refactor(ogtags): optimize URL construction and memory allocations #647
• docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org by @lotharsm in docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org #661
• feat: add robots2policy CLI to convert robots.txt to Anubis CEL by @JasonLovesDoggo in feat: add robots2policy CLI to convert robots.txt to Anubis CEL #657
• Add ReactOS to known-instances.md by @ColinFinck in Add ReactOS to known-instances.md #664
• build(deps): bump the github-actions group with 3 updates by @dependabot in build(deps): bump the github-actions group with 3 updates #666
• Add the blog section back by @Xe in Add the blog section back #670
• feat: implement a client for Thoth, the IP reputation database for Anubis by @Xe in feat: implement a client for Thoth, the IP reputation database for Anubis #637
• chore(sponsors): update canine.tools logo by @hyperdefined in chore(sponsors): update canine.tools logo #672
• perf: Replace internal SHA256 hashing with xxhash for 4-6x performance improvement by @JasonLovesDoggo in perf: Replace internal SHA256 hashing with xxhash for 4-6x performance improvement #676
• perf: replace cidranger with bart for significant performance improvements by @JasonLovesDoggo in perf: replace cidranger with bart for significant performance improvements #675
• docs(known-instances): add wiki.koha-community.org by @prettysunflower in docs(known-instances): add wiki.koha-community.org #683
• chore: remove duplicate CHANGELOG entry by @JasonLovesDoggo in chore: remove duplicate CHANGELOG entry #684
• fix(geo): correct typo "counties" to "countries" by @hydrargyrum in fix(geo): correct typo "counties" to "countries" #678
• docs(known-instances): add extensions.typo3.org by @lotharsm in docs(known-instances): add extensions.typo3.org #691
• Bump ai.robots.txt to v1.37 by @Dryusdan in Bump ai.robots.txt to v1.37 #689
• feat(config): custom weight thresholds via CEL by @Xe in feat(config): custom weight thresholds via CEL #688
• fix(config): actually load threshold config by @Xe in fix(config): actually load threshold config #696
• feat: add default OpenGraph tags to configuration file by @Xe in feat: add default OpenGraph tags to configuration file #694

New Contributors

• @CryptoCopter made their first contribution in docs(known-instances): add Alliance of Hessian Libraries #611
• @foosinn made their first contribution in docs(subrequest-auth): document required policy changes #613
• @jieter made their first contribution in docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth" #615
• @Fierelier made their first contribution in Make progress bar styling more compatible (UXP, etc) #636
• @XerTheSquirrel made their first contribution in Update known-instances.md to include SquirrelJME #643
• @ColinFinck made their first contribution in Add ReactOS to known-instances.md #664
• @hydrargyrum made their first contribution in fix(geo): correct typo "counties" to "countries" #678

Full Changelog: v1.19.1...v1.20.0-pre1

---

This discussion was created from the release v1.20.0-pre1.

[Xe]

Known issue: #600, Anubis can sometimes false-block the Git client. To fix this, import the Git client allow rule from the standard library:

- import: (data)/clients/git.yaml

Be sure to do this before your default CHALLENGE rule if you have one.

Alternatively, you can use this policy file that is about to be released as -pre2:

bots:
  - import: (data)/bots/ai-robots-txt.yaml
  - import: (data)/bots/cloudflare-workers.yaml
  - import: (data)/bots/headless-browsers.yaml
  - import: (data)/bots/us-ai-scraper.yaml

  - import: (data)/crawlers/googlebot.yaml
  - import: (data)/crawlers/bingbot.yaml
  - import: (data)/crawlers/duckduckbot.yaml
  - import: (data)/crawlers/qwantbot.yaml
  - import: (data)/crawlers/internet-archive.yaml
  - import: (data)/crawlers/kagibot.yaml
  - import: (data)/crawlers/marginalia.yaml
  - import: (data)/crawlers/mojeekbot.yaml

  - import: (data)/clients/git.yaml

  - import: (data)/common/keep-internet-working.yaml

  - name: generic-browser
    user_agent_regex: >-
      Mozilla|Opera
    action: CHALLENGE

dnsbl: false

thresholds:
  - name: minimal-suspicion
    expression: weight <= 0
    action: ALLOW

  - name: mild-suspicion
    expression:
      all:
        - weight > 0
        - weight < 10
    action: CHALLENGE
    challenge:
      algorithm: metarefresh
      difficulty: 1
      report_as: 1

  - name: moderate-suspicion
    expression:
      all:
        - weight >= 10
        - weight < 20
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 2
      report_as: 2

  - name: extreme-suspicion
    expression: weight >= 20
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 4
      report_as: 4