CloudFlare Turnstile

Why are these changes being introduced:

* Bots are using more resources than desired

Relevant ticket(s):

* https://mitlibraries.atlassian.net/browse/TIMX-480

How does this address that need:

* Adds rack-attack and bot_challenge_page gems
* Adds redis to heroku app.json (will need to manually enable in
  staging and production environments before merge)

Author: JPrevost

Gemfile

@@ -4,13 +4,15 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 ruby '3.2.7'
 
 gem 'bootsnap', require: false
+gem 'bot_challenge_page'
 gem 'graphql'
 gem 'graphql-client'
 gem 'http'
 gem 'importmap-rails'
 gem 'jbuilder'
 gem 'mitlibraries-theme', git: 'https://github.com/mitlibraries/mitlibraries-theme', tag: 'v1.4'
 gem 'puma'
+gem 'rack-attack'
 gem 'rails', '~> 7.1.0'
 gem 'sentry-rails'
 gem 'sentry-ruby'

Gemfile.lock

@@ -106,6 +106,10 @@ GEM
       debug_inspector (>= 1.2.0)
     bootsnap (1.18.4)
       msgpack (~> 1.2)
+    bot_challenge_page (0.3.0)
+      http (~> 5.2)
+      rack-attack (~> 6.7)
+      rails (>= 7.1, < 8.1)
     builder (3.3.0)
     capybara (3.40.0)
       addressable
@@ -239,6 +243,8 @@ GEM
       nio4r (~> 2.0)
     racc (1.8.1)
     rack (3.1.12)
+    rack-attack (6.7.0)
+      rack (>= 1.0, < 4)
     rack-session (2.1.0)
       base64 (>= 0.1.0)
       rack (>= 3.0.0)
@@ -395,6 +401,7 @@ DEPENDENCIES
   better_errors
   binding_of_caller
   bootsnap
+  bot_challenge_page
   capybara
   climate_control
   debug
@@ -410,6 +417,7 @@ DEPENDENCIES
   mocha
   pg
   puma
+  rack-attack
   rails (~> 7.1.0)
   rubocop
   rubocop-rails

README.md

@@ -67,6 +67,22 @@ If the `flipflop` gem has been updated, check that the `:gdt` feature is working
 UI elements specific to GDT (e.g., geospatial search fields or the 'Ask GIS' link) appear with the
 feature flag enabled, and do not when it is disabled.
 
+### CloudFlare Turnstile
+
+This application uses [CloudFlare Turnstile](https://www.cloudflare.com/application-services/products/turnstile/) via
+the [Bot Challenge Page](https://github.com/samvera-labs/bot_challenge_page) gem.
+
+In development, you can enable/disable this by running `rails dev:cache`. When `dev:cache` is not enabled, the cache is
+set to `null` so no enforcement is in place. As we do not register `localhost` with CloudFlare, if you have `dev:cache`
+enabled locally, you won't actually see the Turnstile challenge and instead will see a message saying you have been
+blocked. This is what users would also see if a deployed app is not registered with CloudFlare so we need to ensure all
+apps we intend to protect are registered with the site key we have enabled.
+
+`Bot Challenge Page` uses [rack-attack](https://github.com/rack/rack-attack). On Heroku deployed apps, we'll be using
+Redis to track requests.
+
+See `Optional Environment Variables` for more information.
+
 ### Required Environment Variables
 
 - `TIMDEX_GRAPHQL`: Set this to the URL of the GraphQL endpoint. There is no default value in the application.
@@ -79,6 +95,10 @@ feature flag enabled, and do not when it is disabled.
 - `BOOLEAN_OPTIONS`: comma separated list of values to present to testers on instances where `BOOLEAN_PICKER` feature is enabled.
 - `BOOLEAN_PICKER`: feature to allow users to select their preferred boolean type. If set, feature is enabled. This feature is only intended for internal team
   testing and should never be enabled in production (mostly because the UI is a mess more than it would cause harm).
+- `CLOUDFLARE_SITE_KEY`: obtained through our cloudflare account (see lastpass for account info)
+- `CLOUDFLARE_SECRET_KEY`: obtained through our cloudflare account (see lastpass for account info)
+- `CLOUDFLARE_REQUEST_PERIOD_IN_HOURS`: integer in hours we use for grouping requests. Combined with `CLOUDFLARE_REQUESTS_PER_PERIOD` this makes up the "requests allowed per time period". Defaults to 12.
+- `CLOUDFLARE_REQUESTS_PER_PERIOD`: integer representing number of results and records pages allowed in the period defined in `CLOUDFLARE_REQUEST_PERIOD_IN_HOURS`. Defaults to 10.
 - `FACT_PANELS_ENABLED`: Comma separated list of enabled fact panels. See `/views/results.html.erb` for implemented panels/valid options. Leave unset to disable all.
 - `FILTER_ACCESS_TO_FILES`: The name to use instead of "Access to files" for that filter / aggregation.
 - `FILTER_CONTENT_TYPE`: The name to use instead of "Content type" for that filter / aggregation.

app.json

@@ -1,4 +1,12 @@
 {
   "name": "timdex-ui",
-  "stack": "heroku-22"
+  "stack": "heroku-22",
+  "addons": [
+    "heroku-redis"
+  ],
+  "buildpacks": [
+    {
+      "url": "heroku/ruby"
+    }
+  ]
 }

app/controllers/application_controller.rb

@@ -1,3 +1,9 @@
 class ApplicationController < ActionController::Base
+  # This will only protect CONFIGURED routes, but also could be put on just certain
+  # controllers, it does not need to be in ApplicationController
+  before_action do |controller|
+    BotChallengePage::BotChallengePageController.bot_challenge_enforce_filter(controller)
+  end
+
   helper Mitlibraries::Theme::Engine.helpers
 end

config/application.rb

@@ -1,6 +1,8 @@
 require_relative "boot"
 
 require "rails/all"
+
+require 'rack/attack'
 require 'sprockets/railtie'
 
 # Require the gems listed in Gemfile, including any gems

config/initializers/bot_challenge_page.rb

@@ -0,0 +1,51 @@
+Rails.application.config.to_prepare do
+
+  BotChallengePage::BotChallengePageController.bot_challenge_config.enabled = true
+
+  # Get from CloudFlare Turnstile: https://www.cloudflare.com/application-services/products/turnstile/
+  # Some testing keys are also available: https://developers.cloudflare.com/turnstile/troubleshooting/testing/
+  #
+  # Always pass testing sitekey: "1x00000000000000000000AA"
+  BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_sitekey = ENV.fetch('CLOUDFLARE_SITE_KEY', "NOT SET")
+  # Always pass testing secret_key: "1x0000000000000000000000000000000AA"
+  BotChallengePage::BotChallengePageController.bot_challenge_config.cf_turnstile_secret_key = ENV.fetch('CLOUDFLARE_SECRET_KEY', "NOT SET")
+
+  BotChallengePage::BotChallengePageController.bot_challenge_config.redirect_for_challenge = false
+
+  # What paths do you want to protect?
+  #
+  # You can use path prefixes: "/catalog" or even "/"
+  #
+  # Or hashes with controller and/or action:
+  #
+  #   { controller: "catalog" }
+  #   { controller: "catalog", action: "index" }
+  #
+  # Note that we can only protect GET paths, and also think about making sure you DON'T protect
+  # any path your front-end needs JS `fetch` access to, as this would block it (at least
+  # without custom front-end code we haven't really explored)
+
+  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limited_locations = ['/results', '/record']
+
+  # allow rate_limit_count requests in rate_limit_period, before issuing challenge
+  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_period = ENV.fetch('CLOUDFLARE_REQUEST_PERIOD_IN_HOURS', 12).to_i.hour
+  BotChallengePage::BotChallengePageController.bot_challenge_config.rate_limit_count = ENV.fetch('CLOUDFLARE_REQUESTS_PER_PERIOD', 10).to_i
+
+  # How long will a challenge success exempt a session from further challenges?
+  # BotChallengePage::BotChallengePageController.bot_challenge_config.session_passed_good_for = 36.hours
+
+  # Exempt some requests from bot challenge protection
+  # BotChallengePage::BotChallengePageController.bot_challenge_config.allow_exempt = ->(controller) {
+  #   # controller.params
+  #   # controller.request
+  #   # controller.session
+
+  #   # Here's a way to identify browser `fetch` API requests; note
+  #   # it can be faked by an "attacker"
+  #   controller.request.headers["sec-fetch-dest"] == "empty"
+  # }
+
+  # More configuration is available
+
+  BotChallengePage::BotChallengePageController.rack_attack_init
+end

config/routes.rb

@@ -1,4 +1,5 @@
 Rails.application.routes.draw do
+  post "/challenge", to: "bot_challenge_page/bot_challenge_page#verify_challenge", as: :bot_detect_challenge
   mount Flipflop::Engine => "/flipflop"
   root "basic_search#index"