robots.txt: Allow only minimal access to REST API endpoints for crawlers

[bram-atmire]

Is your feature request related to a problem? Please describe.

DSpace 7+ sites can get a lot of unwanted crawling requests against rest api endpoints that are irrelevant for search engines to index, as they essentially only should be getting the item pages and bitstreams.

An example of an affected repository, getting millions of crawling requests, and TB of bandwidth consumption over a 90 day period, and where you see that the requests for (rest api) JSON are +50% of all requests

Updated proposal 2025-09-11 Describe the solution you'd like

# Crawlers should get minimal accesss to the rest api, allowing to crawl item pages with javascript and download bitstreams
Allow: /server/api
Allow: /server/api/core/bitstreams/
Allow: /server/api/authn/status
Allow: /server/api/security/csrf
Disallow: /server/api/
Disallow: /server/opensearch/
Disallow: /server/oai/

Improvements

• More permissive to allow robots to crawl/render javascript
• dropping the use of *, as this is not an official operator. The trailing slash is what is important to allow sub-paths (very important for core/bitstreams/)

ORIGINAL PROPOSAL FLAWED 2025-06-17 Describe the solution you'd like

The following could be added to robots.txt

Allow: /server/api/core/bitstreams
Disallow: /server/api/*
Disallow: /server/api
Disallow: /server/opensearch/*
Disallow: /server/oai/*

Additional context

For reference, see these docs on the use of the Allow directive, confirming that it can be used to effectively override (broader) disallow directives:
https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

[tdonohue]

@bram-atmire : I think any development must somehow be configurable. Obviously, there's a major assumption in this ticket that the REST API is available at /server under the same domain as the User Interface. While that may often be the case, it's not guaranteed.

I'm also wondering how this would impact bots which can understand Javascript. Supposedly, both Google and Bing bots can process Javascript. If that's the case, blocking them from the REST API would essentially block their ability to process the Javascript, and might have a negative impact on SEO.

That said, many other bots (e.g. Google Scholar's bots) do not understand Javascript. So, as you note, this may be beneficial to them.

Overall, though, I'm a little worried this could be more complex than it may seem. I'm not sure whether blocking bots from the REST API will ultimately improve or worsen SEO.

I do understand though how some of this REST traffic may be unwanted/undesired. I'm just not sure whether blocking good bots will be beneficial or not.

[bram-atmire]

@tdonohue you are absolutely right that some crawlers CAN render javascript and that we should give enough access to facilitate crawlers rendering javascript and the associated REST endpoints needed. The original proposal I made is too restrictive. Here is an alternative proposal but might still need tweaking/further evaluation in PROD

# Crawlers should get minimal accesss to the rest api, allowing to crawl item pages with javascript and download bitstreams
Allow: /server/api
Allow: /server/api/core/bitstreams/
Allow: /server/api/authn/status
Allow: /server/api/security/csrf
Disallow: /server/api/
Disallow: /server/opensearch/
Disallow: /server/oai/

I'm also updating the title of this ticket to better reflect the goal here.