From 634b433e303d2d62ff70a1b871c43329f1c6e483 Mon Sep 17 00:00:00 2001 From: Nicolas Dupont Date: Tue, 18 Nov 2025 15:52:08 +0100 Subject: [PATCH 1/5] Document technical upgrades process --- content/api/cli.md | 12 +- .../terms/explanation/technical-upgrades.md | 139 ++++++++++++++++++ 2 files changed, 150 insertions(+), 1 deletion(-) create mode 100644 content/terms/explanation/technical-upgrades.md diff --git a/content/api/cli.md b/content/api/cli.md index 3d1440fe..50de3ef3 100644 --- a/content/api/cli.md +++ b/content/api/cli.md @@ -14,7 +14,7 @@ In these commands: ## Tracking terms -{{< refItem name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration." example="npx ota track" />}} +{{< refItem name="ota track" description="Track the current terms of services according to provided declarations. The declarations, snapshots and versions paths are defined in the configuration. This command automatically runs a technical upgrade pass first to apply any engine, dependency, or declaration updates before tracking." example="npx ota track" />}} > Note that the snapshots and versions will be recorded at the moment the command is executed, on top of the existing local history. If a shared history already exists and the goal is to add on top of it, that history has to be downloaded before executing that command. @@ -26,6 +26,16 @@ In these commands: {{< refItem name="ota track --schedule [--services ...] [--types ...]" description="Track terms on the schedule defined in the configuration" example="npx ota track --schedule" />}} +## Applying technical upgrades + +{{< refItem name="ota apply-technical-upgrades" description="Apply technical upgrades by regenerating versions from existing snapshots with updated declarations/engine. This also fetches missing snapshots for newly added source documents in combined terms. Versions created during this process are labeled as technical upgrades to avoid false notifications." example="npx ota apply-technical-upgrades" />}} + +{{< refItem name="ota apply-technical-upgrades --help" description="Show help and available options for apply-technical-upgrades command" example="npx ota apply-technical-upgrades --help" />}} + +{{< refItem name="ota apply-technical-upgrades [--services ...]" description="Apply technical upgrades to specific services only" example="npx ota apply-technical-upgrades --services \"Facebook\" \"LinkedIn\"" />}} + +{{< refItem name="ota apply-technical-upgrades [--services ...] [--types ...]" description="Apply technical upgrades to specific terms types of specific services only" example="npx ota apply-technical-upgrades --services \"Facebook\" \"LinkedIn\" --types \"Privacy Policy\" \"Terms of Service\"" />}} + ## Validating declarations {{< refItem name="ota validate declarations [--services ...] [--types ...]" description="Check that all declarations allow recording a snapshot and a version properly. If service IDs are provided, check only those services." example="npx ota validate declarations --services \"Facebook\" --types \"Privacy Policy\"" />}} diff --git a/content/terms/explanation/technical-upgrades.md b/content/terms/explanation/technical-upgrades.md new file mode 100644 index 00000000..2b4200b9 --- /dev/null +++ b/content/terms/explanation/technical-upgrades.md @@ -0,0 +1,139 @@ +--- +title: "Technical upgrades" +weight: 5 +--- + +# Technical upgrades + +## What is the technical upgrades process + +The **technical upgrade** process creates new **versions** by re-extracting content from the **latest snapshots** when there are changes not in service terms content, but in the system that extracts them (declarations, filters, engine, or dependencies). + +## Why technical upgrades are important + +Technical upgrades solve the critical problem of **distinguishing between actual content changes and extraction improvements**. + +Without technical upgrades, improving a declaration would trigger false notifications. For example, if terms have sections A, B, and C but the declaration only extracted A and B, adding section C would make the next version appear to include new content, triggering a notification even though the service's terms never changed. + +With technical upgrades, the system re-extracts from the current snapshot using the improved declaration and creates a new version, that includes section C, marked as a technical upgrade. Next regular tracking then compares against this upgraded version, so only actual content changes trigger notifications. + +## How technical upgrades work + +For each tracked terms: + +1. Retrieve the latest snapshot for each source document of the terms +2. Re-extract content using latest declarations and engine code +3. Create a new version marked as a technical upgrade + +## Types of changes handled + +1. Declaration changes: updates to selectors, filters, or removal rules +2. Engine changes: updates to the core extraction logic +3. Dependency changes: updates to libraries affecting extraction (e.g., HTML-to-Markdown conversion) + +## Behavior for different scenarios + +### Selector or filter changes + +**Example:** + +```json +// Before: missing section C +{ + "Privacy Policy": { + "fetch": "https://example.com/privacy", + "select": ".section-a, .section-b" + } +} + +// After: includes all relevant sections +{ + "Privacy Policy": { + "fetch": "https://example.com/privacy", + "select": ".section-a, .section-b, .section-c" + } +} +``` + +**What happens:** + +- Retrieves the latest snapshot +- Re-extracts content using updated selectors and/or filters +- Creates a new version marked as a technical upgrade + +### Adding new source documents to combined terms + +**Example:** + +```json +// Before: 2 source documents +{ + "Community Guidelines": { + "combine": [ + { "id": "main", "fetch": "https://example.com/community" }, + { "id": "hate-speech", "fetch": "https://example.com/community/hate-speech" } + ] + } +} + +// After: 3 source documents +{ + "Community Guidelines": { + "combine": [ + { "id": "main", "fetch": "https://example.com/community" }, + { "id": "hate-speech", "fetch": "https://example.com/community/hate-speech" }, + { "id": "violence", "fetch": "https://example.com/community/violence" } // NEW + ] + } +} +``` + +**What happens:** + +- Fetches and records snapshots **only for new source documents** +- Retrieves latest snapshots for existing source documents +- Extracts all documents and creates one new combined version marked as a technical upgrade + +### Location changes + +**What happens:** + +Nothing, technical upgrades do not fetch from new locations. Location changes represent a genuine change in how the service publishes their terms and should be tracked as a regular content change. + +### Engine and dependency changes + +When you upgrade the engine or dependencies, extraction logic may change even if declarations remain the same. + +**Examples:** + +- Engine improves HTML entity decoding so ` ` entities are converted to regular spaces instead of appearing literally in versions +- Library improves table support so complex tables preserve their structure as Markdown tables instead of being converted to plain text + +**What happens:** + +- Retrieves the latest snapshot for each terms +- Re-extracts using updated code +- Creates a new version marked as a technical upgrade if output differs + +### Special cases + +**No existing version:** For terms that have never been tracked before, the technical upgrade is skipped because there is no existing version to upgrade. + +**Missing snapshots:** Technical upgrades cannot extract content without snapshots. If a source document has no snapshot, no version can be created. However, for combined terms, if you add a new source document to the declaration, technical upgrades will fetch and record snapshots only for these newly added documents, then combine them with existing snapshots to create the upgraded version. + +## Technical upgrade markers + +Versions created during technical upgrades are marked with: + +- `isTechnicalUpgrade: true` in version metadata +- Commit message specify it by starting by `Apply technical or declaration upgrade on …` + +## Running technical upgrades + +Technical upgrades run automatically with `npx ota track`. + +To run them separately: + +```bash +npx ota apply-technical-upgrades +``` From 2249cc1d8bc38855d0a898c12db831c3a7ecd925 Mon Sep 17 00:00:00 2001 From: Nicolas Dupont Date: Wed, 19 Nov 2025 09:39:45 +0100 Subject: [PATCH 2/5] Update documenation --- content/terms/explanation/technical-upgrades.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/content/terms/explanation/technical-upgrades.md b/content/terms/explanation/technical-upgrades.md index 2b4200b9..9c49f838 100644 --- a/content/terms/explanation/technical-upgrades.md +++ b/content/terms/explanation/technical-upgrades.md @@ -115,12 +115,6 @@ When you upgrade the engine or dependencies, extraction logic may change even if - Re-extracts using updated code - Creates a new version marked as a technical upgrade if output differs -### Special cases - -**No existing version:** For terms that have never been tracked before, the technical upgrade is skipped because there is no existing version to upgrade. - -**Missing snapshots:** Technical upgrades cannot extract content without snapshots. If a source document has no snapshot, no version can be created. However, for combined terms, if you add a new source document to the declaration, technical upgrades will fetch and record snapshots only for these newly added documents, then combine them with existing snapshots to create the upgraded version. - ## Technical upgrade markers Versions created during technical upgrades are marked with: @@ -132,8 +126,4 @@ Versions created during technical upgrades are marked with: Technical upgrades run automatically with `npx ota track`. -To run them separately: - -```bash -npx ota apply-technical-upgrades -``` +To run them separately, see the [apply-technical-upgrades command]({{< relref "api/cli#applying-technical-upgrades" >}}). From 81ca7c87b3473d96cc2616e873ece05174b64e4c Mon Sep 17 00:00:00 2001 From: Nicolas Dupont Date: Wed, 19 Nov 2025 14:19:41 +0100 Subject: [PATCH 3/5] Unify command doc with cli description --- content/api/cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/api/cli.md b/content/api/cli.md index 50de3ef3..3b8797f1 100644 --- a/content/api/cli.md +++ b/content/api/cli.md @@ -28,7 +28,7 @@ In these commands: ## Applying technical upgrades -{{< refItem name="ota apply-technical-upgrades" description="Apply technical upgrades by regenerating versions from existing snapshots with updated declarations/engine. This also fetches missing snapshots for newly added source documents in combined terms. Versions created during this process are labeled as technical upgrades to avoid false notifications." example="npx ota apply-technical-upgrades" />}} +{{< refItem name="ota apply-technical-upgrades" description="Apply technical upgrades by generating new versions from the latest snapshots using updated declarations, engine logic, or dependencies, and by retrieving any missing snapshots for newly added source documents. Versions created during this process are labeled as technical upgrades to avoid false notifications." example="npx ota apply-technical-upgrades" />}} {{< refItem name="ota apply-technical-upgrades --help" description="Show help and available options for apply-technical-upgrades command" example="npx ota apply-technical-upgrades --help" />}} From b9418dbc81fcaae4258e65cc0be2eaa18820d18f Mon Sep 17 00:00:00 2001 From: Nicolas Dupont Date: Wed, 19 Nov 2025 14:21:04 +0100 Subject: [PATCH 4/5] Unify styling with existing documentation --- content/terms/explanation/technical-upgrades.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/terms/explanation/technical-upgrades.md b/content/terms/explanation/technical-upgrades.md index 9c49f838..9315d825 100644 --- a/content/terms/explanation/technical-upgrades.md +++ b/content/terms/explanation/technical-upgrades.md @@ -7,7 +7,7 @@ weight: 5 ## What is the technical upgrades process -The **technical upgrade** process creates new **versions** by re-extracting content from the **latest snapshots** when there are changes not in service terms content, but in the system that extracts them (declarations, filters, engine, or dependencies). +The technical upgrade process creates new versions by re-extracting content from the latest snapshots when there are changes not in service terms content, but in the system that extracts them (declarations, filters, engine, or dependencies). ## Why technical upgrades are important @@ -35,7 +35,7 @@ For each tracked terms: ### Selector or filter changes -**Example:** +Example: ```json // Before: missing section C @@ -55,7 +55,7 @@ For each tracked terms: } ``` -**What happens:** +What happens: - Retrieves the latest snapshot - Re-extracts content using updated selectors and/or filters @@ -63,7 +63,7 @@ For each tracked terms: ### Adding new source documents to combined terms -**Example:** +Example: ```json // Before: 2 source documents @@ -88,7 +88,7 @@ For each tracked terms: } ``` -**What happens:** +What happens: - Fetches and records snapshots **only for new source documents** - Retrieves latest snapshots for existing source documents @@ -96,7 +96,7 @@ For each tracked terms: ### Location changes -**What happens:** +What happens: Nothing, technical upgrades do not fetch from new locations. Location changes represent a genuine change in how the service publishes their terms and should be tracked as a regular content change. @@ -104,12 +104,12 @@ Nothing, technical upgrades do not fetch from new locations. Location changes re When you upgrade the engine or dependencies, extraction logic may change even if declarations remain the same. -**Examples:** +Examples: - Engine improves HTML entity decoding so ` ` entities are converted to regular spaces instead of appearing literally in versions - Library improves table support so complex tables preserve their structure as Markdown tables instead of being converted to plain text -**What happens:** +What happens: - Retrieves the latest snapshot for each terms - Re-extracts using updated code From b43b6a8d8fc6cfcca9573bb02a58c7bc702e1afd Mon Sep 17 00:00:00 2001 From: Nicolas Dupont Date: Wed, 19 Nov 2025 14:23:36 +0100 Subject: [PATCH 5/5] Improve doc --- content/terms/explanation/technical-upgrades.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/terms/explanation/technical-upgrades.md b/content/terms/explanation/technical-upgrades.md index 9315d825..fafb81a3 100644 --- a/content/terms/explanation/technical-upgrades.md +++ b/content/terms/explanation/technical-upgrades.md @@ -120,7 +120,7 @@ What happens: Versions created during technical upgrades are marked with: - `isTechnicalUpgrade: true` in version metadata -- Commit message specify it by starting by `Apply technical or declaration upgrade on …` +- Commit message specify it by starting by `Apply technical or declaration upgrade on ` ## Running technical upgrades