Document technical upgrades process (#205)

Author: Ndpnt

content/api/cli.md

@@ -14,7 +14,7 @@ In these commands:
 
 ## Tracking terms
 
-{{< refItem name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration." example="npx ota track" />}}
+{{< refItem name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration. This command automatically runs a technical upgrade pass first to apply any engine, dependency, or declaration updates before tracking." example="npx ota track" />}}
 
 > Note that the snapshots and versions will be recorded at the moment the command is executed, on top of the existing local history. If a shared history already exists and the goal is to add on top of it, that history has to be downloaded before executing that command.
 
@@ -26,6 +26,16 @@ In these commands:
 
 {{< refItem name="ota track --schedule [--services <service_id>...] [--types <terms_type>...]" description="Track terms on the schedule defined in the configuration" example="npx ota track --schedule" />}}
 
+## Applying technical upgrades
+
+{{< refItem name="ota apply-technical-upgrades" description="Apply technical upgrades by generating new versions from the latest snapshots using updated declarations, engine logic, or dependencies, and by retrieving any missing snapshots for newly added source documents. Versions created during this process are labeled as technical upgrades to avoid false notifications." example="npx ota apply-technical-upgrades" />}}
+
+{{< refItem name="ota apply-technical-upgrades --help" description="Show help and available options for apply-technical-upgrades command" example="npx ota apply-technical-upgrades --help" />}}
+
+{{< refItem name="ota apply-technical-upgrades [--services <service_id>...]" description="Apply technical upgrades to specific services only" example="npx ota apply-technical-upgrades --services \"Facebook\" \"LinkedIn\"" />}}
+
+{{< refItem name="ota apply-technical-upgrades [--services <service_id>...] [--types <terms_type>...]" description="Apply technical upgrades to specific terms types of specific services only" example="npx ota apply-technical-upgrades --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"" />}}
+
 ## Validating declarations
 
 {{< refItem name="ota validate declarations [--services <service_id>...] [--types <terms_type>...]" description="Check that all declarations allow recording a snapshot and a version properly. If service IDs are provided, check only those services." example="npx ota validate declarations --services \"Facebook\" --types \"Privacy Policy\"" />}}

content/terms/explanation/technical-upgrades.md

@@ -0,0 +1,129 @@
+---
+title: "Technical upgrades"
+weight: 5
+---
+
+# Technical upgrades
+
+## What is the technical upgrades process
+
+The technical upgrade process creates new versions by re-extracting content from the latest snapshots when there are changes not in service terms content, but in the system that extracts them (declarations, filters, engine, or dependencies).
+
+## Why technical upgrades are important
+
+Technical upgrades solve the critical problem of **distinguishing between actual content changes and extraction improvements**.
+
+Without technical upgrades, improving a declaration would trigger false notifications. For example, if terms have sections A, B, and C but the declaration only extracted A and B, adding section C would make the next version appear to include new content, triggering a notification even though the service's terms never changed.
+
+With technical upgrades, the system re-extracts from the current snapshot using the improved declaration and creates a new version, that includes section C, marked as a technical upgrade. Next regular tracking then compares against this upgraded version, so only actual content changes trigger notifications.
+
+## How technical upgrades work
+
+For each tracked terms:
+
+1. Retrieve the latest snapshot for each source document of the terms
+2. Re-extract content using latest declarations and engine code
+3. Create a new version marked as a technical upgrade
+
+## Types of changes handled
+
+1. Declaration changes: updates to selectors, filters, or removal rules
+2. Engine changes: updates to the core extraction logic
+3. Dependency changes: updates to libraries affecting extraction (e.g., HTML-to-Markdown conversion)
+
+## Behavior for different scenarios
+
+### Selector or filter changes
+
+Example:
+
+```json
+// Before: missing section C
+{
+  "Privacy Policy": {
+    "fetch": "https://example.com/privacy",
+    "select": ".section-a, .section-b"
+  }
+}
+
+// After: includes all relevant sections
+{
+  "Privacy Policy": {
+    "fetch": "https://example.com/privacy",
+    "select": ".section-a, .section-b, .section-c"
+  }
+}
+```
+
+What happens:
+
+- Retrieves the latest snapshot
+- Re-extracts content using updated selectors and/or filters
+- Creates a new version marked as a technical upgrade
+
+### Adding new source documents to combined terms
+
+Example:
+
+```json
+// Before: 2 source documents
+{
+  "Community Guidelines": {
+    "combine": [
+      { "id": "main", "fetch": "https://example.com/community" },
+      { "id": "hate-speech", "fetch": "https://example.com/community/hate-speech" }
+    ]
+  }
+}
+
+// After: 3 source documents
+{
+  "Community Guidelines": {
+    "combine": [
+      { "id": "main", "fetch": "https://example.com/community" },
+      { "id": "hate-speech", "fetch": "https://example.com/community/hate-speech" },
+      { "id": "violence", "fetch": "https://example.com/community/violence" }  // NEW
+    ]
+  }
+}
+```
+
+What happens:
+
+- Fetches and records snapshots **only for new source documents**
+- Retrieves latest snapshots for existing source documents
+- Extracts all documents and creates one new combined version marked as a technical upgrade
+
+### Location changes
+
+What happens:
+
+Nothing, technical upgrades do not fetch from new locations. Location changes represent a genuine change in how the service publishes their terms and should be tracked as a regular content change.
+
+### Engine and dependency changes
+
+When you upgrade the engine or dependencies, extraction logic may change even if declarations remain the same.
+
+Examples:
+
+- Engine improves HTML entity decoding so ` ` entities are converted to regular spaces instead of appearing literally in versions
+- Library improves table support so complex tables preserve their structure as Markdown tables instead of being converted to plain text
+
+What happens:
+
+- Retrieves the latest snapshot for each terms
+- Re-extracts using updated code
+- Creates a new version marked as a technical upgrade if output differs
+
+## Technical upgrade markers
+
+Versions created during technical upgrades are marked with:
+
+- `isTechnicalUpgrade: true` in version metadata
+- Commit message specify it by starting by `Apply technical or declaration upgrade on <service_id> <terms_type>`
+
+## Running technical upgrades
+
+Technical upgrades run automatically with `npx ota track`.
+
+To run them separately, see the [apply-technical-upgrades command]({{< relref "api/cli#applying-technical-upgrades" >}}).