feat(translation): complete translation parity review batches (#144)

* chore(i18n): update locale code maps and parity scripts

* feat(notion-fetch): enforce deterministic locale grouping and category generation

* feat(notion-fetch): improve locale section fallback and callout transforms

* feat(notion-fetch): preserve callout children and harden title extraction

* feat(notion-translate): preserve rich text and add conversion diagnostics

* test(translation): add locale parity harness and tracking docs

* fix(i18n): normalize es/pt translation key parity

* Codex-generated pull request (#146)

* fix(parity): harden tokenizer and icon stripping behavior

* docs(translation): audit resolved and remaining parity issues

* fix(callout): support locale punctuation separators

* fix(translation): close remaining parity harness and typing gaps

* fix(translation): resolve remaining merge-blocking type issues

* fix(test): align notion translation env bootstrap with runtime validation

* fix(parity): avoid nested-list and setext tokenization false positives

* fix(translation): correct callout delimiter and less language mapping

* fix(parity): preserve accented callouts and nested-list parent order

* fix(i18n): remove cross-language entries swapped into es and pt locale files

Portuguese strings were appended to i18n/es/code.json and Spanish strings to
i18n/pt/code.json. All the misplaced entries are duplicates of keys already
present in the correct locale file, so they can be removed outright.

* fix(i18n): remove self-referential translated keys from es and pt locale files

Six entries in es/code.json used Spanish text as both key and value, and six
entries in pt/code.json used Portuguese text as both key and value. Three of
each were duplicates of correctly English-keyed entries; the remaining three
were placeholder Notion labels (New Page, New Section Title, New Toggle,
Planejamento e Preparação para um Projeto) that had no English counterpart.

Removing all 12 reduces the cross-locale key diff from 12 to 0, unblocking
the verify-locale-output.test.ts parity check (max allowed: 3).

* chore(i18n): stop tracking locale files on feature branch

i18n/ is gitignored and belongs only on the content branch.
Previous commits on this branch incorrectly force-added and patched
i18n/es/code.json and i18n/pt/code.json directly. The content branch
does not have those bad entries, so no fix is needed there.

Remove the files from git tracking so the gitignore rule is respected.

* ci(test): fetch i18n content from content branch before running tests

The i18n/ directory is gitignored and lives only on the content branch.
Tests in verify-locale-output.test.ts require locale files to be present,
so mirror the pattern used by the deploy-pr-preview workflow:

  git checkout origin/content -- i18n/

Falls back gracefully (|| echo) if the content branch is unavailable.

* docs: update _category_.json creation to include all locales

Correction reflects code change in sectionProcessors.ts that removed
the English-only guard.

🤖 Generated with [Qoder][https://qoder.com]

* refactor: remove unused createAnnotations function

Dead code identified in PR review - function was defined but never called.

🤖 Generated with [Qoder][https://qoder.com]

* fix(translate): add table support and rate limit guard for block deletion

- Import remark-gfm so GFM tables are parsed (without it, table syntax
  falls through to plain-text paragraphs)
- Add TableNode/TableRowNode/TableCellNode types and a 'table' case in
  the markdown-to-Notion switch, producing a Notion table block with
  inline table_row children and has_column_header based on row count
- Add 100ms delay between individual block delete calls to avoid
  hitting Notion rate limits when updating pages with many blocks
- Add 3 tests covering full table conversion, single-row tables, and
  the no-paragraph-fallback invariant

* fix(ci,scripts): harden translation root/filter and frontmatter handling

Author: luandro

.github/workflows/test.yml

@@ -23,11 +23,32 @@ jobs:
         with:
           bun-version: latest
 
+      - name: Checkout i18n content from content branch
+        id: i18n_content
+        shell: bash
+        run: |
+          if git checkout origin/content -- i18n/; then
+            echo "has_i18n=true" >> "$GITHUB_OUTPUT"
+            echo "✅ Loaded i18n content from origin/content."
+          else
+            echo "has_i18n=false" >> "$GITHUB_OUTPUT"
+            echo "⚠️ origin/content checkout failed. Locale-dependent tests will be skipped."
+          fi
+
       - name: Install dependencies
         run: bun install
 
       - name: Rebuild sharp for CI environment
         run: npm rebuild sharp
 
       - name: Run tests
-        run: bun run test
+        shell: bash
+        run: |
+          if [ "${{ steps.i18n_content.outputs.has_i18n }}" = "true" ]; then
+            echo "✅ Running full test suite (including locale-dependent tests)."
+            bun run test
+          else
+            echo "⚠️ Skipping locale-dependent tests because origin/content checkout failed."
+            echo "Skipped: scripts/locale-parity.test.ts, scripts/verify-locale-output.test.ts"
+            bunx vitest run --pool=threads --exclude scripts/locale-parity.test.ts --exclude scripts/verify-locale-output.test.ts
+          fi

README.md

@@ -18,6 +18,7 @@ This repository uses a **two-branch architecture** to separate code from generat
 - **`content` branch**: Generated documentation from Notion (docs/, i18n/, static/images/ ~29MB)
 
 **Why separate branches?**
+
 - Keeps main branch clean for code review and development
 - Reduces repository clone time for contributors
 - Separates content syncs from code changes
@@ -28,14 +29,17 @@ This repository uses a **two-branch architecture** to separate code from generat
 Before local development, you need content files. Choose one of these methods:
 
 **Option 1: Fetch from content branch** (Recommended - Fast)
+
 ```bash
 git fetch origin content
 git checkout origin/content -- docs/ i18n/ static/images/
 ```
 
 **Option 2: Generate from Notion** (Requires API access)
+
 1. Copy `.env.example` to `.env` and add your Notion API key and Database ID
 2. Fetch content:
+
 ```bash
 bun notion:fetch
 ```
@@ -48,7 +52,7 @@ The `bun notion:fetch` script pulls structured content from Notion and rewrites
 - **Sub-page grouping**: Parents must link their language variants through the `Sub-item` relation. Each linked child should set `Language` to `English`, `Spanish`, or `Portuguese`. Any other language values are ignored.
 - **Element Type field drives layout**:
   - `Element Type = Page` exports markdown, regenerates frontmatter, rewrites remote images under `static/images/`, and tracks compression savings for the summary.
-  - `Element Type = Toggle` creates a folder (plus `_category_.json` for English) and increments the “section folders” counter.
+  - `Element Type = Toggle` creates a folder (plus `_category_.json` for all locales) and increments the “section folders” counter.
   - `Element Type = Heading` stores the heading for the next `Page` entry’s sidebar metadata and increments the “title sections applied” counter.
 - **Summary counters**: The totals printed at the end reflect the actions above. Zeros mean no matching work occurred (for example, no toggles, no headings, or no images to optimize).
 - **Translations**: When a non-English child page is processed, its title is written to `i18n/<locale>/code.json` using the parent’s English title as the key. Ensure those files exist before running the script.
@@ -68,6 +72,7 @@ bun dev
 This command opens your browser automatically and reflects changes immediately.
 
 **Full local setup from scratch:**
+
 ```bash
 # Clone repository
 git clone https://github.com/digidem/comapeo-docs.git
@@ -99,6 +104,7 @@ The resulting files are placed in the `build` directory for deployment via any s
 #### How Deployment Works
 
 Deployments use a **checkout strategy**:
+
 1. Checkout `main` branch (code and scripts)
 2. Overlay content files from `content` branch (docs, i18n, images)
 3. Build the site with merged content
@@ -221,24 +227,28 @@ The repository includes several automated workflows for content management:
 #### Content Workflows (Push to `content` branch)
 
 **Sync Notion Docs** (`sync-docs.yml`)
+
 - **Trigger**: Manual dispatch or repository dispatch
 - **Purpose**: Fetches content from Notion and commits to `content` branch
 - **Target Branch**: `content`
 - **Environment**: Requires `NOTION_API_KEY` and `DATABASE_ID` secrets
 
 **Translate Docs** (`translate-docs.yml`)
+
 - **Trigger**: Manual dispatch or repository dispatch
 - **Purpose**: Generates translations and commits to `content` branch
 - **Target Branch**: `content`
 - **Environment**: Requires `NOTION_API_KEY`, `DATABASE_ID`, `OPENAI_API_KEY`
 
 **Fetch All Content for Testing** (`notion-fetch-test.yml`)
+
 - **Trigger**: Manual dispatch with optional force mode
 - **Purpose**: Tests complete content fetch from Notion
 - **Target Branch**: `content`
 - **Features**: Retry logic, detailed statistics, content validation
 
 **Clean All Generated Content** (`clean-content.yml`)
+
 - **Trigger**: Manual dispatch with confirmation
 - **Purpose**: Removes all generated content from `content` branch
 - **Target Branch**: `content`
@@ -247,11 +257,13 @@ The repository includes several automated workflows for content management:
 #### Deployment Workflows (Read from both branches)
 
 **Deploy to Staging** (`deploy-staging.yml`)
+
 - **Trigger**: Push to `main`, manual dispatch, or after content sync
 - **Process**: Checkout `main` + overlay `content` → build → deploy to GitHub Pages
 - **URL**: https://digidem.github.io/comapeo-docs
 
 **Deploy to Production** (`deploy-production.yml`)
+
 - **Trigger**: Push to `main` or manual dispatch
 - **Process**: Checkout `main` + overlay `content` → build → deploy to Cloudflare Pages
 - **URL**: https://docs.comapeo.app

context/development/translation-improvements-progress.md

@@ -0,0 +1,129 @@
+# Translation Improvements Progress Tracker
+
+Date started: 2026-02-19  
+Branch: `feat/translation-parity-improvements`  
+Worktree: `.worktrees/translation-parity-improvements`
+
+## Goal
+
+Ensure auto-translation output is structurally equivalent across `en`, `pt`, and `es` markdown for every translated page family.
+
+Allowed differences:
+
+- media asset URLs/files/paths (images and other media links)
+
+Required exact parity (after excluding media only):
+
+- emojis (presence, position, and semantics)
+- markdown structure and formatting
+- heading hierarchy and order
+- admonitions/callouts
+- list structure and ordering
+- code fences and inline code tokens
+- non-media link structure and target parity
+- frontmatter key set consistency (translated values allowed for text fields)
+
+## Current Snapshot (Baseline)
+
+Baseline check date: 2026-02-19
+
+Verified failing family:
+
+- Root page: `2331b081-62d5-80a1-810e-dbb15a2e0f68`
+- EN: `docs/gathering-the-right-equipment-for-comapeo.md`
+- PT: `i18n/pt/docusaurus-plugin-content-docs/current/reunindo-o-equipamento-certo-para-o-comapeo.md`
+- ES: `i18n/es/docusaurus-plugin-content-docs/current/nueva-pgina.md`
+
+Observed mismatches:
+
+- heading levels/count/order mismatch
+- list nesting/ordering mismatch
+- non-media link mismatch
+- ES body effectively empty vs EN/PT content
+
+## Success Criteria
+
+The translation pipeline is considered fixed only when all criteria pass:
+
+1. For sampled and targeted translated families, parity checks pass for EN/PT/ES after media normalization.
+2. No generated PT/ES markdown files are empty when EN source contains non-empty content.
+3. Emoji conversion and placement is preserved across locales.
+4. Markdown formatting survives translation roundtrip without structural loss.
+5. CI/local test suite contains automated parity checks that fail on regressions.
+
+## Verification Workflow (Repeatable)
+
+### Step 1: Generate translation-child outputs from Notion
+
+Targeted family:
+
+```bash
+bun run notion:fetch-auto-translation-children -- --page-id <root_page_id>
+```
+
+Batch mode:
+
+```bash
+bun run notion:fetch-auto-translation-children
+```
+
+This writes:
+
+- `.cache/auto-translation-children-comparison.md`
+
+### Step 2: Build EN/PT/ES file triplets
+
+Source of truth:
+
+- `.cache/auto-translation-children-comparison.md`
+
+Select rows where EN/PT/ES all exist.
+
+### Step 3: Run parity comparison (media-normalized)
+
+Comparison must validate:
+
+- headings
+- lists
+- admonitions/callouts
+- code fences and inline code
+- non-media links
+- frontmatter key set
+- emoji retention
+
+Current implementation status:
+
+- manual/subagent-driven parity analysis exists
+- automated parity script and tests still needed (tracked in research + implementation backlog)
+
+### Step 4: Record results in this tracker
+
+For each run, append:
+
+- date/time
+- command used
+- families compared
+- pass/fail counts
+- failure categories
+- links to changed code/tests
+
+## Run Log
+
+| Date       | Scope                                                | Compared Families | Pass | Fail | Notes                                                      |
+| ---------- | ---------------------------------------------------- | ----------------: | ---: | ---: | ---------------------------------------------------------- |
+| 2026-02-19 | Targeted root `2331b081-62d5-80a1-810e-dbb15a2e0f68` |                 1 |    0 |    1 | ES output empty/partial mismatch; structural parity failed |
+
+## Implementation Backlog (Pipeline Hardening)
+
+- [ ] Add deterministic parity checker script for EN/PT/ES triplets (media-normalized).
+- [ ] Add Vitest coverage for parity checker with golden fixtures.
+- [ ] Harden `scripts/notion-translate/markdownToNotion.ts` against block loss.
+- [ ] Harden markdown generation path (`scripts/notion-fetch/*`) for locale consistency.
+- [ ] Add CI gate for parity regressions on sampled families.
+
+## Guardrails
+
+- Do not patch generated markdown by hand.
+- Fix generation code and rerun pipeline.
+- Treat empty translated markdown as critical failure.
+- Preserve existing media exception only; no broader exceptions.