[Discussion] Provide DSpace tools or better documentation regarding managing aggressive bots / harvesters

[tdonohue]

Is your feature request related to a problem? Please describe.

This ticket comes out of discussion with the Google Scholar team in July 2025. Google Scholar is noticing more and more repositories are accidentally blocking Google Scholar bots while attempting to alleviate the aggressive behaviors of other bots (especially AI bots). The Google Scholar team understands that sites have to find ways to address aggressive bots, but asked if we could brainstorm whether there are tools or documentation we could add to help DSpace sites make better decisions (and hopefully alleviate some of the common bot-related issues).

A few brainstorms include:

1. Could DSpace improve our documentation around Apache (or similar) to document recommended tools/services/settings that can be used to block or rate-limit aggressive bots? This documentation might be added to our Performance Tuning DSpace guide, but it'd require one or more sites being willing to share their knowledge for how they've successively dealt with aggressive bots.
2. Could DSpace have a basic rate limiter that can be better configured to block aggressive IPs, "/24" ranges, or even via geolocation? This wouldn't solve the entire issue, but might help sites to avoid some basic aggressive bot behaviors.
   • This might be an improved "rateLimiter", which already exists in our UI configuration. Our current setup is very basic and uses express-rate-limit. Are there ways to enhance that (whether using the same library or a different one)?

Describe the solution you'd like

It doesn't matter whether the solution is built into DSpace or is just better documentation/recommendations. The main goal is to provide individual DSpace sites with better advice / hints / tips, so that they aren't all "rebuilding the wheel" (and sometimes accidentally blocking Google Scholar or other "good bots" while doing so).

We can start small, and build upon that into better solutions. For example, what is something basic that DSpace sites can do which has even a small benefit for alleviating the effect of aggressive bots (or blocking some of them)?

Additional information

Related discussions on dealing with aggressive bots include:

• Aggressive AI Harvesting of Digital Resources
• https://wiki.code4lib.org/Blocking_Bots and the #bots channel on the code4lib Slack team.

[mwoodiupui]

I suggest that we also highlight "what not to do". That's what Google Scholar wants: don't do things that block Googlebot, because it is a good netizen and supports their service which we appreciate.

Example: connections =/= traffic. A good bot may have a score of connections open concurrently, but each one sips slowly and that organization's overall impact on your service is negligible. Don't just firewall an entire netblock because you see lots of connections from there.

[tdonohue]

This topic was discussed in today's Developers Meeting. Therefore, I'm copying the brainstorms from that discussion into this ticket.

Brainstorms from our meeting:

• Could we ask Google Scholar to limit their bot to just one per site? Some sites see several at once.
• Could we address this by continuing to address performance in DSpace? If DSpace has improved performance it helps our users and also decreases the likelihood that bot attacks will take down a site.
  • We all agree performance is something we have to continue to improve on. Tim asks if everyone can start to create more tickets around areas of the application that need performance improvements. These improvements are likely to be incremental, so it will take likely lots of developers making small improvements to provide big impacts.
  • Will this actually address the problem of bad bots? The bad bots seem to be harvesting as fast as possible...if DSpace gets faster, the bots will just harvest faster.
  • Maybe it needs to be combined with a rate limiter type of solution
• Improving rate-limiter
  • We have a built in rate-limiter (see "rateLimiter" settings), but it's unclear if it's beneficial. It's configuration may need looking at. Or we could look at whether other rate limiters are more beneficial
  • Could we consider adding a Captcha to it? It could let us lower the limit significantly, and if a human hits the limit, they can get past it via a Captcha.
  • (Idea from Slack) Could we have a rate limiter that is lower for anonymous access, but higher if you provide "information" that lets us trust you? (e.g. CrossRef, Unpaywall and OpenAlex have lower rate limits in the default request pool, and higher rate limits in the "polite" pool if you specify an email address)
• Improvements to robots.txt from 4Science
  • 4Science has made some improvements to robots.txt which seem to benefit Google Scholar bots. They can share this back to DSpace.
  • This won't fix behavior of bad bots, but it still would make good bots more efficient. So, it'd still be great to add to DSpace.
• Sharing what others are doing that has shown a good benefit:
  • Paulo's team is using "several caching levels. Apache Cache+Varnish and DSpace" to provide minimum service
  • 4Science notes that they've seen great performance benefits from enabling the "ssrBaseUrl" settings in the "rest" settings of the User Interface. This setting was introduced in 7.6.3, 8.1 and 9.0.

Please add your own brainstorms, recommended configurations or findings to this discussion ticket as comments.

[paulo-graca]

DSpace 7.6.1 – Our Experience

Our experience with DS 7.6.1 has had its ups and downs. We have now managed to stabilize our services, but at a cost.

By way of introduction, we currently manage 27 DSpace installations for national institutions such as universities and hospitals, and we also run a single multi‑tenant repository that serves more than 100 institutions. We use a slightly extended version of DSpace, including the Additions module and some custom features that integrate with national services (e.g. thesis deposit and curriculum management).

We began migrating our repositories from DS 5.2 to DS 7.6.1 last May, and only two remain before the migration is complete.

As I noted in my comment:
#3110 (comment)
we hit several bumps along the way. Our workaround was to throw extra resources at the problem, cache as aggressively as possible, and block certain kinds of access.

---

Resource Requirements

• DSpace 5: a single instance could run on 4 GB RAM and 4 CPU cores (repository‑size‑dependent).
• DSpace 7.6.1: the practical minimum is 8 GB RAM and 6 CPU cores.

---

Caching Strategy

1. Angular Layer

We enable the built-in cache in dspace-angular.

2. Varnish

We added an extra caching layer in front of Angular:

backend default {
    .host = "dspace-angular";
    .port = "4000";
    .max_connections = 500;
    .probe = {
        .request =
            "HEAD / HTTP/1.1"
            "Host: localhost"
            "Connection: close"
            "User-Agent: Varnish Health Probe";
        .interval  = 10s;
        .timeout   = 35s;
        .window    = 5;
        .threshold = 3;
    }
    .connect_timeout        = 10s;
    .first_byte_timeout     = 120s;
    .between_bytes_timeout  = 5s;
}

3. Apache mod_cache

We use Apache's mod_cache for static assets:

<IfModule mod_cache.c>
    CacheQuickHandler off
    CacheLock on
    CacheIgnoreNoLastMod On
    CacheIgnoreCacheControl Off
    CacheIgnoreHeaders Set-Cookie Cookie Expires Cache-Control Pragma If-None-Match If-Modified-Since

    CacheHeader on
    SetOutputFilter CACHE

    # Enable compression for text assets
    AddOutputFilterByType DEFLATE text/html text/plain text/css application/javascript application/rss+xml text/xml image/svg+xml

    <IfModule mod_cache_disk.c>
        CacheRoot /var/cache/apache2/mod_cache_disk
        <LocationMatch ".*\.(css|js|png|jpg|svg|ico|html|woff2)$">
            CacheEnable disk
        </LocationMatch>
        CacheDirLevels 2
        CacheDirLength 1
        CacheMaxFileSize 10000000      # 10 MB
        CacheMinFileSize 1
        CacheDefaultExpire 3600        # 1 hour
        CacheMaxExpire 86400           # 1 day
    </IfModule>

    <IfModule mod_expires.c>
        ExpiresActive On
        ExpiresByType text/html              "access plus 1 day"
        ExpiresByType text/css               "access plus 1 day"
        ExpiresByType application/javascript "access plus 1 day"
        ExpiresByType image/png              "access plus 1 day"
        ExpiresByType image/jpeg             "access plus 1 day"
        ExpiresByType image/x-icon           "access plus 1 day"
        ExpiresByType image/svg+xml          "access plus 1 day"
        ExpiresByType font/woff2             "access plus 1 day"
    </IfModule>
</IfModule>

---

Controlling Crawlers

Since SSR control only occurred in earlier DSpace versions, to avoid robots crawling our search pages we use Apache mod_qos:

<Location /search>
    # Block all access to /search?configuration=default
    QS_DenyPath +search deny "^.*$"
</Location>

We also use iptables with a curated block list for the most aggressive crawlers.

[alanorth]

I've just made a pull request to upgrade express-rate-limit to the latest version, which brings some new features that might be interesting for us.

[tdonohue]

Interesting article on battling aggressive bots from U of North Carolina: https://journal.code4lib.org/articles/18489 This article talks through all the approaches they tried and which seemed to work better than others.

Overall, they found Cloudflare Turnstile (a "smart CAPTCHA alternative") to work the best for them. There is a free plan for Cloudflare Turnstile available, but it has limits. This tool can be used without requiring other Cloudflare services.

Cloudflare Turnstile does have a (community-maintained, open source) library available for Angular: https://developers.cloudflare.com/turnstile/community-resources/ Direct link: https://www.npmjs.com/package/ngx-turnstile

[mwoodiupui]

It's sad that the bot builders are pouring so much talent into being more effectively obnoxious users, when they could hire a network engineer for an afternoon and learn how to write bots that limit their own demands and thus don't attract our attention. I don't want to block bots; I want to slow them down to reasonable rates. But they go out of their way to make it difficult to be generous. It becomes harder and harder not to regard this activity as deliberately hostile acts.