Merge branch 'develop' into 11562-api-get-templates

Author: stevenwinship

doc/release-notes/10606-windows-dev.md

@@ -0,0 +1 @@
+Development of Dataverse on Windows has been confirmed to work as long as you use WSL rather than cmd.exe. See [the guides](https://dataverse-guide--11583.org.readthedocs.build/en/11583/developers/dev-environment.html#quickstart), #10606, and #11583.

doc/release-notes/11281-ExternalSearch.md

@@ -0,0 +1,29 @@
+### Configurable Search Services
+
+Dataverse now has an experimental capability to dynamically add and configure new search engines.
+The current Dataverse user interface can be configured to use a specified search engine instead of the built-in solr search.
+The search API now supports an optional `searchService` query parameter that allows using any configured search engine.
+An additional /api/search/services endpoint allows discovery of the services installed.
+
+In addition to two trivial example services designed for testing, Dataverse ships with two search engine classes that support calling an externally-hosted search service (via HTTP GET or POST).
+These classes rely on the internal solr search to perform access-control and to format the final results, simplifying development of such an external engine.
+
+Details about the new functionality are described in https://dataverse-guide--11281.org.readthedocs.build/en/11281/developers/search-services.html
+
+See also #11281.
+
+## Settings
+
+### Database Settings:
+
+***New:***
+
+- :GetExternalSearchUrl
+- :GetExternalSearchName
+- :PostExternalSearchUrl
+- :PostExternalSearchName
+
+### New Configuration Options
+
+- `dataverse.search.services.directory`
+- `dataverse.search.default-service`

doc/sphinx-guides/source/api/native-api.rst

@@ -2751,9 +2751,9 @@ The limit can be set via JVM setting :ref:`dataverse.files.default-dataset-file-
 
 For Installation wide limit, the limit can be set via JVM. ./asadmin $ASADMIN_OPTS create-jvm-options "-Ddataverse.files.default-dataset-file-count-limit=<limit>"
 
-For Collections, the attribute can be controlled by calling the Create or Update Dataverse API and adding ``datasetFileCountLimit=500`` to the Json body.
+For Collections, the attribute can be controlled by calling the Create or Update Dataverse API and adding ``datasetFileCountLimit=500`` to the Json body (Must be a superuser to change this value).
 
-For Datasets, the attribute can be set using the `Update Dataset Files Limit <#setting-the-files-count-limit-on-a-dataset>`_ API and passing the qp `fileCountLimit=500`.
+For Datasets, the attribute can be set using the `Update Dataset Files Limit <#setting-the-files-count-limit-on-a-dataset>`_ API and passing the qp `fileCountLimit=500` (Must be a superuser to change this value).
 
 Setting a value of -1 will clear the limit for that level. If no limit is found on the Dataset, the hierarchy of parent nodes will be checked until finally the JVM setting is checked.
 
@@ -2769,7 +2769,7 @@ Setting the files count limit on a Dataset
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 In order to update the number of files allowed for a Dataset, without causing a Draft version of the Dataset being created, the following API can be used
 
-.. note:: To clear the limit simply set the limit to -1 or call the DELETE API.
+.. note:: To clear the limit simply set the limit to -1 or call the DELETE API (Must be a superuser to change this value).
 
 .. code-block:: bash
 

doc/sphinx-guides/source/api/search.rst

@@ -29,16 +29,19 @@ type             string   Can be either "dataverse", "dataset", or "file". Multi
 subtree          string   The identifier of the Dataverse collection to which the search should be narrowed. The subtree of this Dataverse collection and all its children will be searched.  Multiple "subtree" parameters can be used to include multiple Dataverse collections. For example, https://demo.dataverse.org/api/search?q=data&subtree=birds&subtree=cats .
 sort             string   The sort field. Supported values include "name" and "date". See example under "order".
 order            string   The order in which to sort. Can either be "asc" or "desc".  For example, https://demo.dataverse.org/api/search?q=data&sort=name&order=asc
-per_page         int      The number of results to return per request. The default is 10. The max is 1000. See :ref:`iteration example <iteration-example>`.
+per_page         int      The number of results to return per request. The default is 10. The max is 1000. See :ref:`iteration example <iteration-example>`.
 start            int      A cursor for paging through search results. See :ref:`iteration example <iteration-example>`.
 show_relevance   boolean  Whether or not to show details of which fields were matched by the query. False by default. See :ref:`advanced search example <advancedsearch-example>`.
 show_facets      boolean  Whether or not to show facets that can be operated on by the "fq" parameter. False by default. See :ref:`advanced search example <advancedsearch-example>`.
 fq               string   A filter query on the search term. Multiple "fq" parameters can be used. See :ref:`advanced search example <advancedsearch-example>`.
 show_entity_ids  boolean  Whether or not to show the database IDs of the search results (for developer use).
-geo_point        string	  Latitude and longitude in the form ``geo_point=42.3,-71.1``. You must supply ``geo_radius`` as well. See also :ref:`geospatial-search`.
-geo_radius       string	  Radial distance in kilometers from ``geo_point`` (which must be supplied as well) such as ``geo_radius=1.5``.
-metadata_fields  string	  Includes the requested fields for each dataset in the response. Multiple "metadata_fields" parameters can be used to include several fields. The value must be in the form "{metadata_block_name}:{field_name}" to include a specific field from a metadata block (see :ref:`example <dynamic-citation-some>`) or "{metadata_field_set_name}:\*" to include all the fields for a metadata block (see :ref:`example <dynamic-citation-all>`). "{field_name}" cannot be a subfield of a compound field. If "{field_name}" is a compound field, all subfields are included.
+show_api_urls    boolean  Whether or not to show API URLs for the search results
+query_entities   boolean  Whether to query entities for extra metadata (slower). Default is true.
+metadata_fields  string   Includes the requested fields for each dataset in the response. Multiple "metadata_fields" parameters can be used to include several fields. The value must be in the form "{metadata_block_name}:{field_name}" to include a specific field from a metadata block (see :ref:`example <dynamic-citation-some>`) or "{metadata_field_set_name}:\*" to include all the fields for a metadata block (see :ref:`example <dynamic-citation-all>`). "{field_name}" cannot be a subfield of a compound field. If "{field_name}" is a compound field, all subfields are included.
+geo_point        string   Latitude and longitude in the form ``geo_point=42.3,-71.1``. You must supply ``geo_radius`` as well. See also :ref:`geospatial-search`.
+geo_radius       string   Radial distance in kilometers from ``geo_point`` (which must be supplied as well) such as ``geo_radius=1.5``.
 show_type_counts boolean  Whether or not to include total_count_per_object_type for types: Dataverse, Dataset, and Files.
+search_service   string   The name of the search service to use for this query. If omitted, the default search service will be used. For available search services, see :ref:`discovering-available-search-services`.
 ================ =======  ===========
 
 Basic Search Example
@@ -775,3 +778,58 @@ Output from iteration example
       <span class="label label-success pull-right">
         CORS
       </span>
+
+.. _search-services:
+
+Search Services
+---------------
+
+.. _discovering-available-search-services:
+
+Discovering Available Search Services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To discover available search services and their capabilities, you can use the Search Services API endpoint.
+Note: Configurable Search Services are an optional, experimental feature than may evolve faster than other parts of Dataverse.
+
+Example API endpoint: https://demo.dataverse.org/api/search/services
+
+This endpoint returns a list of available search services, including their names, and display names. It also indicates the default search service. 
+
+Example response:
+
+.. code-block:: json
+
+    {
+        "status": "OK",
+        "data": {
+         "services": [
+            {
+                "name": "solr",
+                "displayName": "Solr Search",
+            },
+            {
+                "name": "externalSearch",
+                "displayName": "External Search for Datasets",
+            }
+        ],
+        "defaultService": "solr"
+    }
+
+You can use the ``name`` values returned by this endpoint in the ``search_service`` parameter of the main search API to specify which search service to use for a particular query.
+
+Using Different Search Services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use a specific search service, include the ``search_service`` parameter in your search query and pass the ``name``. For example:
+
+https://demo.dataverse.org/api/search?q=trees&search_service=externalSearch
+
+This query will use the ``externalSearch`` service (assuming it exists) instead of the default search service (``solr``).
+
+.. note:: Other search services may not be complete replacements for the included ``solr`` service. For example, they may not support searching for collections or files (just datasets).
+
+Developing Search Services
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See :doc:`/developers/search-services` in the Developer Guide.

doc/sphinx-guides/source/developers/dev-environment.rst

@@ -18,6 +18,8 @@ After cloning the `dataverse repo <https://github.com/IQSS/dataverse>`_, run thi
 
 ``mvn -Pct clean package docker:run``
 
+(Note that if you are Windows, you must run the command above in `WSL <https://learn.microsoft.com/windows/wsl>`_ rather than cmd.exe. See :doc:`windows`.)
+
 After some time you should be able to log in:
 
 - url: http://localhost:8080

doc/sphinx-guides/source/developers/index.rst

@@ -46,4 +46,5 @@ Developer Guide
    workflows
    fontcustom
    classic-dev-env
+   search-services
 

doc/sphinx-guides/source/developers/search-services.rst

@@ -0,0 +1,141 @@
+Search Services
+===============
+
+Dataverse supports configurable search services, allowing developers to integrate additional search engines dynamically. This guide outlines the design and provides details on how to use the interfaces and classes involved.
+
+Design Overview
+---------------
+The configurable search services feature is designed to allow:
+
+1. Dynamic addition of new search engines
+2. Configuration of the Dataverse UI to use a specified search engine
+3. Use of different search engines via the API
+4. Discovery of installed search engines
+
+Key Components
+--------------
+
+1. SearchService Interface
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+The ``SearchService`` interface is the core of the configurable search services. It defines the methods that any search engine implementation must provide. (The methods below are accurate as of this writing.)
+
+.. code-block:: java
+
+   public interface SearchService {
+       String getServiceName();
+       String getDisplayName();
+       
+       SolrQueryResponse search(DataverseRequest dataverseRequest, List<Dataverse> dataverses, String query,
+               List<String> filterQueries, String sortField, String sortOrder, int paginationStart,
+               boolean onlyDatatRelatedToMe, int numResultsPerPage, boolean retrieveEntities, String geoPoint,
+               String geoRadius, boolean addFacets, boolean addHighlights) throws SearchException;
+
+       default void setSolrSearchService(SearchService solrSearchService);
+   }
+
+The interface allows you to provide a service name and display name, and to respond to the same search parameters that are normally sent to the Solr search engine.
+
+The ``setSolrSearchService`` method is used by Dataverse to give your class a reference to the ``SolrSearchService``, allowing your class to perform Solr queries as needed. (See the ``ExternalSearchServices`` for an example.)
+
+2. ConfigurableSearchService Interface
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``ConfigurableSearchService`` interface extends the ``SearchService`` interface and adds a method for Dataverse to set the ``SettingsServiceBean``. This allows search services to be configurable through Dataverse settings.
+
+.. code-block:: java
+
+   public interface ConfigurableSearchService extends SearchService {
+       void setSettingsService(SettingsServiceBean settingsService);
+   }
+
+The ``GetExternalSearchServiceBean`` and ``PostExternalSearchServiceBean`` classes provide a use case for this.
+
+3. JVM Options for Search Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Dataverse uses two JVM options to configure the search functionality:
+
+- ``dataverse.search.services.directory``: Specifies the local directory where jar files with search engines (classes implementing the ``SearchService`` interface) can be found. Dataverse will dynamically load engines from this directory.
+
+- ``dataverse.search.default-service``: The ``serviceName`` of the service that should be used in the Dataverse UI.
+
+Example configuration:
+
+.. code-block:: bash
+
+   ./asadmin create-jvm-options "-Ddataverse.search.services.directory=/var/lib/dataverse/searchServices"
+   ./asadmin create-jvm-options "-Ddataverse.search.default-service=solr"
+
+Remember to restart your Payara server after modifying these JVM options for the changes to take effect.
+
+4. Using Different Search Engines via API
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The loaded search services can be discovered using the ``/api/search/services`` endpoint.
+
+Queries can be made to different engines by including the optional ``search_service=<serviceName>`` query parameter.
+
+Use of these endpoints is described for end users in the API Guide under :ref:`search-services`.
+
+Available Search Services
+-------------------------
+
+The class definitions for four example search services are included in the Dataverse repository.
+They are not included in the Dataverse .war file but can be built as three separate .jar files using
+
+.. code-block:: bash 
+
+    mvn clean package -DskipTests=true -Pexternal-search-get -Pexternal-search-post
+
+or
+
+.. code-block:: bash 
+
+    mvn clean package -DskipTests=true -Ptrivial-search-examples
+
+1. GetExternalSearchServiceBean
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+2. PostExternalSearchServiceBean
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These classes implement the ``ConfigurableSearchService`` interface.
+They make a GET or POST call (respectively) to an external search engine that must return a JSON array of objects with "PID" (preferred) or "DOI" and "Distance" keys.
+The query sent to the external engine use the same query parameters as the Dataverse search API (GET) or have a JSON payload with those keys (POST).
+The results they return are then searched for using the solr search engine which enforces access control and provides the standard formatting expected by the Dataverse UI and API.
+The distance values are used to order the results, smallest distances first. 
+
+They can be configured via two settings each:
+
+- GET
+
+  - :GetExternalSearchUrl - the URL to send GET search queries to
+  - :GetExternalSearchName - the display name to use for this configuration
+
+- POST
+
+  - :PostExternalSearchUrl - the URL to send POST search queries to
+  - :PostExternalSearchName - the display name to use for this configuration
+
+As these classes use PIDs as identifiers, they cannot reference collections or, unless file PIDs are enabled, files.
+Similar classes, or extensions of these classes could search by database ids instead, etc. to support the additional types.
+
+3. GoldenOldiesSearchServiceBean
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+4. OddlyEnoughSearchServiceBean
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These classes implement the ``SearchService`` interface.
+They are intended only as code examples and simple tests of the design and are not intended for production use.
+The former simply replaces the user query with a query for entities with a db id < 1000. It demonstrates how a class can leverage the solr engine and achieve results solely by modifying/replacing the user query. 
+The latter only returns hits from the user's query that also have an odd database id. Since the filtering in the class changes the number of total hits available and pagination, this class demonstrates one way a developer can adjust those aspects of the Solr response.
+
+Notes
+-----
+
+1. Unless you use the Solr engine to provide access control, you must implement proper access control in your search engine
+2. The design currently limits search results to be in the format returned by Solr and the hits are expected to be collections, datasets, or files - other classes are not supported.
+3. Search services could be designed to completely replace Solr or to just support certain use cases (e.g. the external search classes only handling datasets).
+4. While search services can be deployed as independent jar files, they currently import multiple Dataverse classes and, unlike exporters, cannot be built using just the Dataverse SPI.
+5. As with other experimental features, we expect the ``SearchService`` interface may change over time as we learn about how people use it. Please keep in touch if you are developing search services.
+