Robots.txt: disallowing bots from comm and coll search facets

[bram-atmire]

Describe the bug

DSpace 8 (re-)introduced search functionality on community and collection pages. The search queries, especially with many facets active, are rather heavy queries.

Bots should be disallowed from going there.

Recommended directive

Disallow: /collections/*?f
Disallow: /communities/*?spc.page

To Reproduce

Steps to reproduce the behavior:

1. Check robots.txt
2. Look for the directives mentioned above

Expected behavior

The directives should be present.

[tdonohue]

Agreed that this might be nice to add to the robots.txt. However, I will also note that, as of version 8.1, all search/browse components no longer undergo Server Side Rendering (SSR). This means that most bots will be unable to access these facets anyways, unless SSR is enabled for the search component (see enableSearchComponent at https://wiki.lyrasis.org/display/DSDOC8x/User+Interface+Configuration#UserInterfaceConfiguration-ServerSideRendering(SSR)Settings)

So, this robots.txt update may be nice to have for consistency, but most bots won't be able to access these facets as long as ssr.enableSearchComponent is set to false (which is the default in 8.1+)