Look into Cloudflare Turnstile for searches

[jechols]

To avoid bot traffic hitting our search, we should consider Cloudflare Turnstile. While Solr is usually fast and reliable, something is causing it to OOM lately, and it's likely odd bot traffic. Since we have a sitemap exposing every critical page that we want in search engines, only blocking bots from search results pages should be fine. I think. May need more research there, too....

The Turnstile dev docs look promising, but I think it'll require overriding one or more django templates, and possibly hacking up some core ONI code. This would mean an ONI change or a new plugin for ONI. Though if a plugin could do the template side and the core-hack side, that could be pretty slick to put up for others to use.

[jechols]

One issue that I'm not sure what to do about: you put the turnstile client-side bits on the page you want to protect, which typically means the page before heavy computation occurs. But our search box is on every page, and we absolutely do not want to block bots from all pages. Some users need to scrape large subsets of our data, and we don't want to have to make them jump through hoops.

So... the "easy" option might be to have the search form direct to an intermediate page that does the turnstile check and then loads the real page, but this might be a big change that requires more ONI hacking than we want to manage.

[jechols]

A better approach, if we're having to hack things up anyway, is to asynchronously fetch the results. So the search hits a handler which renders the results page, but doesn't actually do the search. That page has the turnstile on it and whatever search details were sent in the user's request, and we'd use a JS callback for the turnstile widget. Our backend turnstile validation would check the turnstile token, and perform the search, returning the rendered search results template. Or on failure, returning a rendered failure message of some kind.

To make it accessible we'd need the initial page load to state that results are loading. We'd need to handle short success / error messages in some kind of a live region, and populate the search results at the same time that live region is updated so users know the page has new information.

This is probably not a small hack, but it seems like the only approach that would be worth the time we'd have to spend building and maintaining a hack that only saves us from a once-a-quarter Solr outage....

[jechols]

I've opened a related issue in open oni: open-oni/open-oni#631