Additional filter options in GN 4.x harvesters (geonetwork#9100)

Add configuration filter options for categories, metadata standard name, and group owners in 
GeoNetwork 4.x harvester.

Author: juanluisrp

core/src/main/java/org/fao/geonet/kernel/setting/HarvesterSettingsManager.java

@@ -221,7 +221,7 @@ public String add(String path, Object name, Object value) {
 
     public String add(String path, Object name, Object value, boolean encrypted) {
 
-            if (name == null)
+        if (name == null)
             throw new IllegalArgumentException("Name cannot be null");
 
         String sName = makeString(name);

docs/manual/docs/user-guide/harvesting/harvesting-geonetwork-4.md

@@ -1,40 +1,58 @@
-# GeoNetwork 4.X Harvester {#gn4_harvester}
-
-GeoNetwork 4.x changed the search engine to Elasticsearch, which is not compatible with previous versions. To harvest a catalogue based on GeoNetwork 4.x, this harvesting type is required.
-
-## Adding a GeoNetwork 4.x Harvester
-
-To create a GeoNetwork 4.X harvester, go to `Admin console` > `Harvesting` and select `Harvest from` > `GeoNetwork (from 4.x)`:
-
-![](img/add-geonetwork-3-harvester.png)
-
-Provide the following information:
-- **Identification** - Options describing the remote site.
-  - *Name* - A short description of the remote site. It will be shown on the harvesting main page as the name for this instance of the harvester.
-  - *Group* - Group that owns the harvested metadata.
-  - *User* - User that owns the harvested metadata.
-- **Schedule** - Schedule configuration to execute the harvester.
-- **Configure connection to GeoNetwork**:
-  - *Catalog URL* - The URL of the GeoNetwork server from which metadata will be harvested.
-  - *Node name* - GeoNetwork node name to harvest, by default `srv`.
-  - *Search filter* - This allows you to select metadata records for harvest based on certain criteria:
-    - *Full text*
-    - *Title*
-    - *Abstract*
-    - *Keyword*
-    - *Catalog* - Allows you to select a source to filter the metadata to harvest.
-
-- **Configure response processing**
-  - *Action on UUID collision* - Allows you to configure the action when a harvester finds the same UUID on a record collected by another method (another harvester, importer, dashboard editor, etc.).
-    - skipped (default)
-    - overridden
-    - generate a new UUID
-  - *Remote authentication*
-  - *Use full MEF format*
-  - *Use change date for comparison*
-  - *Set category if it exists locally*
-  - *Category for harvested records*
-  - *XSL filter name to apply*
-  - *Validate records before import*
-
-- **Privileges** - Assign privileges to harvested metadata.
+# GeoNetwork 4.x Harvester {#gn4_harvester}
+
+GeoNetwork 4.x uses **Elasticsearch** as its search engine, breaking compatibility with previous protocols 
+from older GeoNetwork releases. To harvest metadata from a GeoNetwork 4.x catalogue, you must use the 
+dedicated GeoNetwork 4.x harvester type.
+
+---
+
+## How to Add a GeoNetwork 4.x Harvester
+
+1. **Go to**: `Admin console` > `Harvesting`
+2. **Select**: `Harvest from` > `GeoNetwork (from 4.x)`
+3. **Complete the configuration panels as described below**
+
+![](img/add-geonetwork-4-harvester.png)
+
+### Identification
+
+- **Name**: Provide a short, descriptive name for your remote GeoNetwork instance. This name will appear in the main page as the name for this instance of the harvester.
+- **Group**: Assign the group that will own the harvested metadata records.
+- **User**: Select the user (owner) for the harvested records.
+
+### Schedule
+
+- **Schedule**: Set how often the harvester should run (e.g., daily, weekly), using cron syntax or the built-in scheduler.
+  You may also run the harvester manually from the interface.
+
+### Connection to GeoNetwork
+
+- **Catalog URL**: Enter the full URL of the remote GeoNetwork 4.x server.
+- **Node name**: Specify the catalogue node to harvest (typically `srv`).
+- **Search filter**: Define filters to limit harvested metadata using criteria like:
+  - **Full text**: Matches all text fields.
+  - **Title**: Filter by title.
+  - **Abstract**: Filter by abstract.
+  - **Keyword**: Filter by keyword.
+  - **Categories**: Combine multiple categories with `AND` or `OR` (e.g., `cat1 AND cat2`, `cat1 OR cat2`).
+  - **Metadata standard**: Specify the metadata standards (e.g., `iso19139 OR iso19115-3.2018`).
+  - **Groups**: List one or more groups (owners of the metadata) numeric IDs, comma-separated.
+  - **Catalog**: Identify a source sub-catalogue, if needed.
+
+### Response Processing
+
+- **Action on UUID collision**: Choose how to configure the action when a harvester finds the same UUID on a record collected by another method (another harvester, importer, dashboard editor, etc.):
+  - **Skipped** (default): Leave duplicates unchanged.
+  - **Overridden**: Replace with new record.
+  - **Generate a new UUID**: Assign a new unique identifier and keep both and old records.
+- **Remote Authentication**: Configure if credentials are required for the source catalogue.
+- **Use Full MEF Format**: Enable to transfer all metadata fields and resources.
+- **Use Change Date for Comparison**: Updates only if the change date differs.
+- **Set Category If Exists Locally**: Assign categories if matched in local catalogue.
+- **Category for Harvested Records**: Set the default category for imported entries.
+- **XSL Filter Name to Apply**: Specify custom XSL transformations if needed.
+- **Validate Records Before Import**: Toggle to validate metadata against the expected schema before import.
+
+### Privileges
+
+- Assign viewing, editing, or publishing privileges for harvested records.

docs/manual/docs/user-guide/harvesting/img/add-geonetwork-4-harvester.png

@@ -66,6 +66,9 @@ protected void storeNodeExtra(GeonetParams params, String path,
             harvesterSettingsManager.add("id:" + searchID, "abstract", s.abstractText);
             harvesterSettingsManager.add("id:" + searchID, "keywords", s.keywords);
             harvesterSettingsManager.add("id:" + searchID, "sourceUuid", s.sourceUuid);
+            harvesterSettingsManager.add("id:" + searchID, "categories", s.categories);
+            harvesterSettingsManager.add("id:" + searchID, "schemes", s.schemes);
+            harvesterSettingsManager.add("id:" + searchID, "groupOwners", s.groupOwners);
         }
 
         //--- store group mapping

harvesters/src/main/java/org/fao/geonet/kernel/harvest/harvester/geonet/v4/Geonet40Harvester.java

@@ -23,8 +23,15 @@
 
 package org.fao.geonet.kernel.harvest.harvester.geonet.v4;
 
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.fao.geonet.Util;
 import org.fao.geonet.constants.Geonet;
 import org.fao.geonet.exceptions.BadParameterEx;
 import org.fao.geonet.kernel.harvest.harvester.geonet.BaseSearch;
@@ -39,12 +46,19 @@
  */
 class Search extends BaseSearch {
 
+    public String categories;
+    public String schemes;
+    public String groupOwners;
+
     public Search() {
         super();
     }
 
     public Search(Element search) throws BadParameterEx {
         super(search);
+        categories = Util.getParam(search, "categories", "");
+        schemes = Util.getParam(search, "schemes", "");
+        groupOwners = Util.getParam(search, "groupOwners", "");
     }
 
     public static Search createEmptySearch(int from, int to) throws BadParameterEx {
@@ -63,6 +77,9 @@ public Search copy() {
         s.sourceUuid = sourceUuid;
         s.from = from;
         s.to = to;
+        s.categories = categories;
+        s.groupOwners = groupOwners;
+        s.schemes = schemes;
 
         return s;
     }
@@ -76,36 +93,60 @@ public Search copy() {
      * @return A string representation of the Elasticsearch query formatted as a JSON object.
      */
     public String createElasticsearchQuery() {
-        String sourceFilter = "";
+
+        List<String> filters = new ArrayList<>();
+
         if (StringUtils.isNotBlank(sourceUuid)) {
-            sourceFilter = String.format(",{\"term\": {\"sourceCatalogue\": \"%s\"}}", sourceUuid);
+            filters.add(String.format("{\"term\": {\"sourceCatalogue\": \"%s\"}}", sourceUuid));
         }
 
-        String freeTextFilter = "";
         if (StringUtils.isNotBlank(freeText)) {
-            freeTextFilter = String.format(",{\"query_string\": {\"query\": \"(any.\\\\*:(%s) OR any.common:(%s))\", \"default_operator\": \"AND\"}}", freeText, freeText);
+            filters.add(String.format("{\"query_string\": {\"query\": \"(any.\\\\*:(%s) OR any.common:(%s))\", \"default_operator\": \"AND\"}}", freeText, freeText));
         }
 
-        String titleFilter = "";
         if (StringUtils.isNotBlank(title)) {
-            titleFilter = String.format(",{\"query_string\": {\"query\": \"(resourceTitleObject.\\\\*:(%s))\", \"default_operator\": \"AND\"}}", title);
+            filters.add(String.format("{\"query_string\": {\"query\": \"(resourceTitleObject.\\\\*:(%s))\", \"default_operator\": \"AND\"}}", title));
         }
 
-        String abstractFilter = "";
         if (StringUtils.isNotBlank(abstractText)) {
-            abstractFilter = String.format(",{\"query_string\": {\"query\": \"(resourceAbstractObject.\\\\*:(%s))\", \"default_operator\": \"AND\"}}", abstractText);
+            filters.add(String.format("{\"query_string\": {\"query\": \"(resourceAbstractObject.\\\\*:(%s))\", \"default_operator\": \"AND\"}}", abstractText));
         }
 
-        String keywordFilter = "";
         if (StringUtils.isNotBlank(keywords)) {
-            abstractFilter = String.format(",{\"term\": {\"tag.default\": \"%s\"}}", keywords);
+             filters.add(String.format("{\"term\": {\"tag.default\": \"%s\"}}", keywords));
+        }
+
+        if (StringUtils.isNotBlank(categories)) {
+            filters.add(String.format("{\"query_string\": {\"query\": \"cat:(%s)\"}}", categories));
+        }
+
+        if (StringUtils.isNotBlank(schemes)) {
+            filters.add(String.format("{\"query_string\": {\"query\": \"documentStandard:(%s)\", \"default_operator\": \"OR\"}}", schemes));
+        }
+
+        if (StringUtils.isNotBlank(groupOwners)) {
+            try {
+                List<String> groupOwners = Arrays.stream(this.groupOwners.split(","))
+                    .map(String::trim)  // Remove extra spaces
+                    .collect(Collectors.toList());
+                String groupOwnersJson = new ObjectMapper().writeValueAsString(groupOwners); // Outputs: ["group1","group2"]
+                filters.add(String.format("{\"terms\": {\"groupOwner\": %s}}", groupOwnersJson));
+
+            } catch (JsonProcessingException e) {
+                Log.debug(Geonet.HARVEST_MAN, "Error creating criteria for ownerGroup. Ignoring this filter.");
+            }
+        }
+
+        String queryFilter = String.join(",", filters);
+        if (StringUtils.isNotBlank(queryFilter)) {
+            queryFilter = "," + queryFilter;
         }
 
         String queryBody = String.format("{\n" +
             "    \"from\": %d,\n" +
             "    \"size\": %d,\n" +
             "    \"sort\": [\"_score\"],\n" +
-            "    \"query\": {\"bool\": {\"must\": [{\"terms\": {\"isTemplate\": [\"n\"]}}%s%s%s%s%s]}},\n" +
+            "    \"query\": {\"bool\": {\"must\": [{\"terms\": {\"isTemplate\": [\"n\"]}}%s]}},\n" +
             "    \"_source\": {\"includes\": [\n" +
             "        \"uuid\",\n" +
             "        \"id\",\n" +
@@ -115,7 +156,8 @@ public String createElasticsearchQuery() {
             "        \"documentStandard\"\n" +
             "    ]},\n" +
             "    \"track_total_hits\": true\n" +
-            "}", from, to, sourceFilter, freeTextFilter, titleFilter, abstractFilter, keywordFilter);
+            "}",
+            from, to, queryFilter);
 
 
         if (Log.isDebugEnabled(Geonet.HARVEST_MAN)) {
@@ -139,6 +181,9 @@ public String toString() {
             .append("title", title)
             .append("abstrac", abstractText)
             .append("keywords", keywords)
+            .append("categories", categories)
+            .append("schemes", schemes)
+            .append("groupOwners", groupOwners)
             .append("sourceUuid", sourceUuid)
             .toString();
     }

harvesters/src/main/java/org/fao/geonet/kernel/harvest/harvester/geonet/v4/Search.java

@@ -120,6 +120,57 @@
         </div>
       </div>
 
+      <div id="gn-harvest-settings-gn4-basic-filter-category-row">
+        <label
+          id="gn-harvest-settings-gn4-basic-filter-category-label"
+          class="control-label col-lg-4"
+          data-translate=""
+          >categories</label
+        >
+        <div class="col-lg-8">
+          <input
+            id="gn-harvest-settings-gn4-basic-filter-category-input"
+            type="text"
+            class="form-control"
+            data-ng-model="harvesterSelected.searches[0].categories"
+          />
+        </div>
+      </div>
+
+      <div id="gn-harvest-settings-gn4-basic-filter-scheme-row">
+        <label
+          id="gn-harvest-settings-gn4-basic-filter-scheme-label"
+          class="control-label col-lg-4"
+          data-translate=""
+          >documentStandard</label
+        >
+        <div class="col-lg-8">
+          <input
+            id="gn-harvest-settings-gn4-basic-filter-scheme-input"
+            type="text"
+            class="form-control"
+            data-ng-model="harvesterSelected.searches[0].schemes"
+          />
+        </div>
+      </div>
+
+      <div id="gn-harvest-settings-gn4-basic-filter-groupOwners-row">
+        <label
+          id="gn-harvest-settings-gn4-basic-filter-groupOwners-label"
+          class="control-label col-lg-4"
+          data-translate=""
+          >groupOwners</label
+        >
+        <div class="col-lg-8">
+          <input
+            id="gn-harvest-settings-gn4-basic-filter-groupOwners-input"
+            type="text"
+            class="form-control"
+            data-ng-model="harvesterSelected.searches[0].groupOwners"
+          />
+        </div>
+      </div>
+
       <div id="gn-harvest-settings-gn4-basic-source-row">
         <label
           id="gn-harvest-settings-gn4-basic-source-label"

web-ui/src/main/resources/catalog/templates/admin/harvest/type/geonetwork40.html

@@ -41,7 +41,10 @@ var gnHarvestergeonetwork40 = {
         "source": {
           "uuid": [],
           "name": []
-        }
+        },
+        "categories": "",
+        "schemes": "",
+        "groupOwners": ""
       }],
       "ifRecordExistAppendPrivileges": false,
       "privileges": [{
@@ -87,6 +90,9 @@ var gnHarvestergeonetwork40 = {
       + '      <title>' + ((h.searches[0] && h.searches[0].title) || '') + '</title>'
       + '      <abstract>' + ((h.searches[0] && h.searches[0]['abstract']) || '') + '</abstract>'
       + '      <keywords>' + ((h.searches[0] && h.searches[0].keywords) || '') + '</keywords>'
+      + '      <categories>' + ((h.searches[0] && h.searches[0].categories) || '') + '</categories>'
+      + '      <schemes>' + ((h.searches[0] && h.searches[0].schemes) || '') + '</schemes>'
+      + '      <groupOwners>' + ((h.searches[0] && h.searches[0].groupOwners) || '') + '</groupOwners>'
       + '      <source>'
       + '        <uuid>' + ((h.searches[0] && h.searches[0].source.uuid) || '') + '</uuid>'
       + '        <name/>'

web-ui/src/main/resources/catalog/templates/admin/harvest/type/geonetwork40.js

@@ -49,6 +49,15 @@
           <keywords>
             <xsl:value-of select="children/keywords/value"/>
           </keywords>
+          <categories>
+            <xsl:value-of select="children/categories/value"/>
+          </categories>
+          <schemes>
+            <xsl:value-of select="children/schemes/value"/>
+          </schemes>
+          <groupOwners>
+            <xsl:value-of select="children/groupOwners/value"/>
+          </groupOwners>
           <source>
             <uuid>
               <xsl:value-of select="children/sourceUuid/value"/>