📝 Update documentation

simonwoerpel · simonwoerpel · commit a7322f44dbc7 · 2026-03-08T20:47:52.000+01:00
diff --git a/Makefile b/Makefile
@@ -35,7 +35,7 @@ clean:
 
 documentation:
 	mkdocs build
-	aws --endpoint-url https://s3.investigativedata.org s3 sync ./site s3://openaleph.org/docs/lib/openaleph-search
+	aws --profile nbg1 --endpoint-url https://s3.investigativedata.org s3 sync ./site s3://openaleph.org/docs/lib/openaleph-search
 
 elastic-build:
 	docker build -t ghcr.io/openaleph/elasticsearch:$(ELASTIC_TAG) .
diff --git a/docs/aggregations.md b/docs/aggregations.md
@@ -97,6 +97,36 @@ Aggregation type for special fields.
 --args "facet=properties.entity&facet_type:properties.entity=entity"
 ```
 
+### `metric:TYPE`
+
+Numeric metric aggregation. `TYPE` is one of `sum`, `avg`, `min`, `max`. Value is a FtM property name (the `numeric.` ES field prefix is resolved internally).
+
+```bash
+# Single metric
+--args "metric:sum=amount"
+
+# Multiple metrics on same or different fields
+--args "metric:sum=amount&metric:avg=amount&metric:min=registrationArea"
+```
+
+Response key format: `{field}.{type}` (e.g. `amount.sum`)
+
+### Metric aggregations
+
+Compute numeric metrics (sum, average, min, max) on numeric fields. Uses the `metric:` prefix with FtM property names — the `numeric.` ES field prefix is resolved internally.
+
+```bash
+# Sum of payment amounts
+openaleph-search search query-string "*" \
+  --args "filter:schema=Payment&metric:sum=amount"
+
+# Multiple metrics
+openaleph-search search query-string "*" \
+  --args "filter:schema=Payment&metric:sum=amount&metric:avg=amount&metric:min=amount&metric:max=amount"
+```
+
+Supported types: `sum`, `avg`, `min`, `max`
+
 ## Response format
 
 Aggregations appear in the `aggregations` section:
@@ -132,6 +162,12 @@ Aggregations appear in the `aggregations` section:
           "bg_count": 100
         }
       ]
+    },
+    "amount.sum": {
+      "value": 125000.0
+    },
+    "amount.avg": {
+      "value": 2500.0
     }
   }
 }
@@ -261,6 +297,13 @@ openaleph-search search query-string "company" \
   --args "facet=schema&facet=created_at&facet_interval:created_at=year&facet_size:schema=50"
 ```
 
+### Payment totals
+
+```bash
+openaleph-search search query-string "*" \
+  --args "filter:schema=Payment&filter:beneficiary=entity-id&metric:sum=amount&metric:avg=amount"
+```
+
 ## Error handling
 
 ### Invalid fields
diff --git a/docs/highlighting.md b/docs/highlighting.md
@@ -50,30 +50,26 @@ Reduce this value for better performance on large documents:
 
 The system uses different Elasticsearch highlighters optimized for each field type:
 
-### Fast Vector Highlighter (FVH)
-
-Used for: `content` field (full-text of source documents)
-
-Best for long text with accurate phrase highlighting. Requires term vectors to be stored in the index.
+### Unified Highlighter
 
-Configuration (via environment):
+Used for: `content` (document text), `text` (secondary text), `translation` (translated text), `name` (entity names)
 
-- `OPENALEPH_SEARCH_HIGHLIGHTER_FVH_ENABLED=true` (default)
-- Requires `OPENALEPH_SEARCH_CONTENT_TERM_VECTORS=true` (default)
+The default highlighter for all fields. Balanced performance with good support for mixed content.
 
-If fast vector highlighting is disabled, the unified highlighter is used for the `content` field.
+### Fast Vector Highlighter (FVH)
 
-### Unified Highlighter
+Optionally used for the `content` field. Provides more accurate phrase highlighting (wraps entire phrases in a single `<em>` tag) but requires term vectors. Disabled by default because it is incompatible with `copy_to` fields excluded from `_source` — for entities where multiple properties copy into `content` (e.g. HyperText with both `bodyHtml` and `indexText`), FVH causes term vector offset mismatches that drop hits from results.
 
-Used for: `name` field
+Configuration (via environment):
 
-Balanced performance for entity names and titles.
+- `OPENALEPH_SEARCH_HIGHLIGHTER_FVH_ENABLED=false` (default)
+- Requires `OPENALEPH_SEARCH_CONTENT_TERM_VECTORS=true` (default) when enabled
 
 ### Plain Highlighter
 
-Used for: `names` (keywords), `text`, and other fields
+Used for: `names` (keywords)
 
-Fast highlighting for simple matches.
+Fast highlighting for simple keyword matches.
 
 ## Configuration
 
@@ -84,10 +80,10 @@ Control highlighting behavior via environment variables:
 Use Fast Vector Highlighter for content field.
 
 ```bash
-export OPENALEPH_SEARCH_HIGHLIGHTER_FVH_ENABLED=true
+export OPENALEPH_SEARCH_HIGHLIGHTER_FVH_ENABLED=false
 ```
 
-When false, uses Unified Highlighter instead.
+Default: `false`. When false, uses Unified Highlighter instead. See [Highlighter types](#highlighter-types) for trade-offs.
 
 ### `highlighter_fragment_size`
 
@@ -146,10 +142,10 @@ Default: `300`
 Maximum characters to analyze.
 
 ```bash
-export OPENALEPH_SEARCH_HIGHLIGHTER_MAX_ANALYZED_OFFSET=999999
+export OPENALEPH_SEARCH_HIGHLIGHTER_MAX_ANALYZED_OFFSET=100000
 ```
 
-Default: `999999`
+Default: `100000`
 
 ## Response format
 
@@ -183,10 +179,15 @@ Matched terms are wrapped in `<em>` tags.
 
 Multiple fields are highlighted automatically:
 
-- `content` - Main document text
-- `name` - Entity names
-- `names` - Name keywords
-- `text` - Secondary text content
+- `content` - Main document text (primary highlight field for entities)
+- `names` - Normalized name keywords
+- `text` - Secondary text content (catch-all `copy_to` target)
+- `translation` - Translated text content
+
+The `text` and `translation` highlight fields can be disabled via settings:
+
+- `OPENALEPH_SEARCH_HIGHLIGHTER_TEXT_FIELD=false`
+- `OPENALEPH_SEARCH_HIGHLIGHTER_TRANSLATION_FIELD=false`
 
 ## Examples
 
@@ -280,4 +281,4 @@ Check that:
 - Reduce `highlight_count`
 - Lower `max_highlight_analyzed_offset`
 - Decrease `phrase_limit`
-- (Other than that the name suggests, the FVH seems to be _slower_ than the unified highlighter): Consider disabling FVH: `OPENALEPH_SEARCH_HIGHLIGHTER_FVH_ENABLED=false`
+- The unified highlighter (default) generally performs well; FVH is not recommended due to compatibility issues with `copy_to` fields
diff --git a/docs/more_like_this.md b/docs/more_like_this.md
@@ -35,7 +35,7 @@ Other entity types (Person, Company, etc.) are excluded.
 Minimum document frequency corpus-wide.
 
 - Type: `int`
-- Default: `0`
+- Default: `1`
 
 ```bash
 --args "mlt_min_doc_freq=2"
@@ -57,7 +57,7 @@ Minimum term frequency within source document.
 Maximum terms to use in query.
 
 - Type: `int`
-- Default: `25`
+- Default: `200`
 
 ```bash
 --args "mlt_max_query_terms=50"
@@ -74,6 +74,28 @@ Percentage of query terms that must match.
 --args "mlt_minimum_should_match=25%"
 ```
 
+### `mlt_min_word_length`
+
+Minimum word length for query terms.
+
+- Type: `int`
+- Default: `5`
+
+```bash
+--args "mlt_min_word_length=3"
+```
+
+### `mlt_max_doc_freq`
+
+Maximum document frequency for query terms. Terms appearing in more documents than this are ignored.
+
+- Type: `int`
+- Default: `500`
+
+```bash
+--args "mlt_max_doc_freq=1000"
+```
+
 ## Parameter effects
 
 ### `min_term_freq`
@@ -112,8 +134,10 @@ Affects query comprehensiveness:
     "fields": ["content", "text", "name"],
     "like": [{"_id": "doc-123"}],
     "min_term_freq": 1,
-    "max_query_terms": 25,
-    "min_doc_freq": 0,
+    "max_query_terms": 200,
+    "min_doc_freq": 1,
+    "min_word_length": 5,
+    "max_doc_freq": 500,
     "minimum_should_match": "10%"
   }
 }
diff --git a/docs/reference/parser.md b/docs/reference/parser.md
@@ -205,6 +205,28 @@ Specify facet aggregation type.
 --args "facet=properties.entity&facet_type:properties.entity=entity"
 ```
 
+## Metric aggregations
+
+Compute numeric metrics on numeric fields. [Read more](../aggregations.md#metric-aggregations)
+
+### `metric:TYPE`
+
+Format: `metric:TYPE=FIELD`
+
+Types: `sum`, `avg`, `min`, `max`
+
+Field is a FtM property name (the `numeric.` ES field prefix is resolved internally).
+
+```bash
+# Sum of amounts
+--args "metric:sum=amount"
+
+# Multiple metrics
+--args "metric:sum=amount&metric:avg=amount&metric:min=registrationArea"
+```
+
+Response keys follow `{field}.{type}` pattern (e.g. `amount.sum`).
+
 ## Significant terms
 
 Find unusual or interesting terms in search results. [Read more](../significant_terms.md)
@@ -304,31 +326,45 @@ Parameters for similarity search. [Read more](../more_like_this.md)
 Minimum document frequency for query terms.
 
 - Type: `int`
-- Default: `5`
+- Default: `1`
 
 ### `mlt_min_term_freq`
 
 Minimum term frequency within document.
 
 - Type: `int`
-- Default: `5`
+- Default: `1`
 
 ### `mlt_max_query_terms`
 
 Maximum number of query terms to use.
 
 - Type: `int`
-- Default: `50`
+- Default: `200`
 
 ### `mlt_minimum_should_match`
 
 Percentage of terms that must match.
 
 - Type: `str`
-- Default: `60%`
+- Default: `10%`
+
+### `mlt_min_word_length`
+
+Minimum word length for query terms.
+
+- Type: `int`
+- Default: `5`
+
+### `mlt_max_doc_freq`
+
+Maximum document frequency for query terms.
+
+- Type: `int`
+- Default: `500`
 
 ```bash
---args "mlt_min_doc_freq=3&mlt_max_query_terms=100&mlt_minimum_should_match=70%"
+--args "mlt_min_doc_freq=3&mlt_max_query_terms=100&mlt_minimum_should_match=25%"
 ```
 
 ## Performance
diff --git a/docs/reference/settings.md b/docs/reference/settings.md
