Merge pull request #473 from atomantic/claim/extract-layout-doc-io-helper-from-migrations

refactor([extract-layout-doc-io-helper-from-migrations]): extract readLayoutsDoc/writeLayoutsDoc helper

Author: atomantic

.changelog/NEXT.md

@@ -22,6 +22,7 @@
 - `data/history.json` is now `data/history.jsonl`. Migration 037 converts the legacy `{ entries: [...] }` wrapper into one JSON record per line, renames the old file to `history.json.bak-037`, and the history service now appends new actions instead of rewriting the whole file on each log. `server/lib/fileUtils.js` also exposes shared `appendJSONLine`, `readJSONLines`, and `writeJSONLines` helpers for other JSONL logs.
 - `/work <task>` Claude Code slash command (`.claude/commands/work.md`) — slugifies the task, spins up a fresh git worktree under `.claude/worktrees/<slug>/` branched off local `main` (NOT current HEAD), `cd`s into it, carries out the work without disturbing the originating checkout's uncommitted edits / branch / HEAD, and on completion always chains `/simplify` → `/do:review --with codex,gemini` → `/do:pr --review-with copilot` so codex + gemini findings are addressed locally *before* the PR opens (otherwise Copilot kicks in immediately on unreviewed code and burns review cycles). Branch / dir collision auto-suffixes with `-2`, `-3`, etc. `.claude/worktrees/` is already gitignored.
 - **Fork-aware update flow.** The PortOS update tab now detects whether the local clone's `origin` remote points at upstream `atomantic/PortOS` or at a personal fork (`server/lib/gitRemote.js` parses SCP-style + HTTPS + ssh:// remote URLs, case-insensitive owner/repo match). When running from a fork the Update tab swaps the "Update Now" button for three explicit choices: **Sync Fork & Update** (runs `gh repo sync <owner>/<fork> --source atomantic/PortOS --branch main` then proceeds with the local update — fast-forward only, so it refuses to clobber divergent fork commits), **Sync Fork Only** (sync without applying), and **Update from Fork As-Is** (skip sync, pull from your fork's origin — useful if you already merged upstream into your fork via your own workflow). The server `/api/update/execute` endpoint now gates fork runs behind either a fresh `lastForkSync` record (≤10 min, same fullName) or an explicit `acknowledgeFork: true` to avoid the silent "I clicked Update and nothing happened" failure mode when a fork's main is behind upstream. The release-check still polls `atomantic/PortOS` so fork users continue to see upstream version notifications. `POST /api/update/sync-fork` exposes the gh-sync action; a 409 with `FORK_DIVERGED` is returned when the fork's main has commits not on upstream, with guidance pointing the user at PRs / feature branches / the explicit `--force` escape hatch they can run from a terminal. `update.sh` / `update.ps1` now log the active origin URL so users can confirm in the update log which repo they actually pulled from.
+- **[extract-layout-doc-io-helper-from-migrations] Shared helper for dashboard-layout seeding migrations.** Internal cleanup: the migrations that add new built-in dashboard widgets/layouts to existing installs now share one read/guard/write helper instead of each copying the same boilerplate, so future layout migrations stay consistent.
 
 ## Fixed
 

PLAN.md

@@ -6,7 +6,6 @@ For project goals, see [GOALS.md](./GOALS.md). For completed work, see [.changel
 
 - [ ] [extract-compare-helpers-once-eight-callers] **Extract `equalByKeys` / `equalListByKeys` once `useAutoRefetch`'s `compare` reaches ~8 widget callers.** With the dashboard rollout (PR #421) there are now 5 widget callers (Backup, Chief of Staff, Decision Log, Goal Progress, Proactive Alerts) on top of EpisodeVideoStage / Brain / Digital Twin from PR #425, using two emerging shapes — (a) scalar-object key equality (BackupWidget status, DecisionLogWidget last24Hours, ProactiveAlertsWidget counts) and (b) array equality by length + per-item key tuple (BackupWidget snapshots, GoalProgressWidget goals, ProactiveAlertsWidget alerts list, DecisionLogWidget impactfulDecisions). Inline forms are noisy but each has an explanatory comment that an `equalByKeys(prev, next, ['status', 'lastRun', …])` would lose. Defer until enough more callers appear that the abstraction pays for itself, then add helpers to `client/src/lib/compareHelpers.js` (register in `index.js` + `README.md`) and migrate the flat callers — leave CosDashboardWidget's nested-optional-chain shape inline. Surfaced by /simplify reuse review during `[useautorefetch-compare-caller-rollout-widgets]` claim 2026-05-21.
 - [ ] [codex5-onboarding-capability-map] **[P2][ONBOARDING]** Capability map of connected systems. One page showing each integration's status: Providers (per-provider configured/available/throttled), Calendar, Brain/memory embeddings, Voice (mic + TTS), Tailscale + HTTPS, Genome/health imports, Telegram/messages, App registry/PM2. Each row links to the relevant settings page. Doubles as a setup checklist and a runtime health overview.
-- [ ] [extract-layout-doc-io-helper-from-migrations] **Extract `readLayoutsDoc` / `writeLayoutsDoc` helper for migrations.** Migrations 029 and 030 both repeat the same `readFile → JSON.parse → Array.isArray(doc.layouts) guard → writeFile` shell with near-identical logging strings. `scripts/migrations/_lib.js` already exists as the shared library for prompt-replace migrations — add a layouts-doc helper there so a future migration that touches `data/dashboard-layouts.json` doesn't fork the pattern a third time. Surfaced by /simplify quality review during `[codex5-dashboard-intent-layouts]` claim 2026-05-20.
 - [ ] [migrate-two-remaining-local-formatters] **Migrate two remaining local formatters surfaced during the module-discovery PR.** Two additional local format definitions were found that aren't in the original tracker: (1) `client/src/components/calendar/EventDetail.jsx:10` defines `formatDateTime(dateStr, isAllDay)` with an all-day branch the shared `formatDateTime` lacks — extend the shared helper to take `{ allDay: true }` or keep this local and rename to `formatEventDateTime`. (2) `client/src/components/cos/tabs/TaskItem.jsx:78` defines `formatDurationMin(mins)` with a `~` approximation prefix (`~3h 30m`) — extend shared `formatDurationMin` with an `{ approximate: true }` option. Low priority; visual semantics need to be preserved exactly.
 - [ ] [wire-proactive-cos-speech-to-real-triggers] **Wire proactive CoS speech to real triggers.** Plumbing landed (`POST /api/voice/speak` + `voice:speak` socket event); hook to high-severity `errorEvents`, `task:ready`, and `notificationEvents` with per-source rate-limits.
 - [ ] [optimize-voice-ui-index-text-payload-lazy-only-run] **Optimize `voice:ui:index` text payload.** Lazy: only run `extractVisibleText` when server requests via `voice:ui:read-request`. Keep current behavior as fallback.

scripts/migrations/029-seed-network-exposure-widget.js

@@ -21,8 +21,7 @@
  *   come back.
  */
 
-import { readFile, writeFile } from 'fs/promises';
-import { join } from 'path';
+import { readLayoutsDoc, writeLayoutsDoc } from './_lib.js';
 
 const WIDGET_ID = 'network-exposure';
 const WIDGET_W = 3;
@@ -75,23 +74,9 @@ function applyToLayout(layout) {
 
 export default {
   async up({ rootDir }) {
-    const path = join(rootDir, 'data', 'dashboard-layouts.json');
-    const raw = await readFile(path, 'utf-8').catch((err) => {
-      if (err.code === 'ENOENT') return null;
-      throw err;
-    });
-    if (raw == null) {
-      console.log(`📦 migration 029: no dashboard-layouts.json yet — fresh install will seed from defaults.`);
-      return { updated: 0, reason: 'no-state' };
-    }
-    let doc;
-    try { doc = JSON.parse(raw); } catch {
-      console.log(`📦 migration 029: dashboard-layouts.json unreadable — skipping.`);
-      return { updated: 0, reason: 'unreadable' };
-    }
-    if (!doc || !Array.isArray(doc.layouts)) {
-      return { updated: 0, reason: 'no-layouts-array' };
-    }
+    const result = await readLayoutsDoc({ rootDir, label: 'migration 029' });
+    if (!result.ok) return { updated: 0, reason: result.reason };
+    const { doc, path } = result;
 
     let touched = 0;
     for (const layout of doc.layouts) {
@@ -104,7 +89,7 @@ export default {
       return { updated: 0, reason: 'already-applied' };
     }
 
-    await writeFile(path, JSON.stringify(doc, null, 2));
+    await writeLayoutsDoc(path, doc);
     console.log(`📦 migration 029: seeded network-exposure widget into ${touched} built-in layout(s).`);
     return { updated: touched };
   },

scripts/migrations/030-seed-intent-dashboard-layouts.js

@@ -20,29 +20,14 @@
  *   re-seed it. This matches the behavior of the fresh-install path.
  */
 
-import { readFile, writeFile } from 'fs/promises';
-import { join } from 'path';
 import { INTENT_LAYOUTS } from '../../server/services/dashboardLayouts.js';
+import { readLayoutsDoc, writeLayoutsDoc } from './_lib.js';
 
 export default {
   async up({ rootDir }) {
-    const path = join(rootDir, 'data', 'dashboard-layouts.json');
-    const raw = await readFile(path, 'utf-8').catch((err) => {
-      if (err.code === 'ENOENT') return null;
-      throw err;
-    });
-    if (raw == null) {
-      console.log(`📦 migration 030: no dashboard-layouts.json yet — fresh install will seed from defaults.`);
-      return { updated: 0, reason: 'no-state' };
-    }
-    let doc;
-    try { doc = JSON.parse(raw); } catch {
-      console.log(`📦 migration 030: dashboard-layouts.json unreadable — skipping.`);
-      return { updated: 0, reason: 'unreadable' };
-    }
-    if (!doc || !Array.isArray(doc.layouts)) {
-      return { updated: 0, reason: 'no-layouts-array' };
-    }
+    const result = await readLayoutsDoc({ rootDir, label: 'migration 030' });
+    if (!result.ok) return { updated: 0, reason: result.reason };
+    const { doc, path } = result;
 
     const existingIds = new Set(doc.layouts.map((l) => l?.id).filter(Boolean));
     const toAdd = INTENT_LAYOUTS.filter((l) => !existingIds.has(l.id));
@@ -52,7 +37,7 @@ export default {
     }
 
     doc.layouts.push(...toAdd.map((l) => ({ ...l, builtIn: true })));
-    await writeFile(path, JSON.stringify(doc, null, 2));
+    await writeLayoutsDoc(path, doc);
     console.log(`📦 migration 030: seeded ${toAdd.length} intent layout(s) (${toAdd.map((l) => l.id).join(', ')}).`);
     return { updated: toAdd.length };
   },

scripts/migrations/033-seed-quick-image-widget.js

@@ -13,8 +13,7 @@
  *   already present and skip.
  */
 
-import { readFile, writeFile } from 'fs/promises';
-import { join } from 'path';
+import { readLayoutsDoc, writeLayoutsDoc } from './_lib.js';
 
 const WIDGET_ID = 'quick-image';
 const WIDGET_W = 3;
@@ -83,23 +82,9 @@ function applyToLayout(layout) {
 
 export default {
   async up({ rootDir }) {
-    const path = join(rootDir, 'data', 'dashboard-layouts.json');
-    const raw = await readFile(path, 'utf-8').catch((err) => {
-      if (err.code === 'ENOENT') return null;
-      throw err;
-    });
-    if (raw == null) {
-      console.log(`📦 migration 033: no dashboard-layouts.json yet — fresh install will seed from defaults.`);
-      return { updated: 0, reason: 'no-state' };
-    }
-    let doc;
-    try { doc = JSON.parse(raw); } catch {
-      console.log(`📦 migration 033: dashboard-layouts.json unreadable — skipping.`);
-      return { updated: 0, reason: 'unreadable' };
-    }
-    if (!doc || !Array.isArray(doc.layouts)) {
-      return { updated: 0, reason: 'no-layouts-array' };
-    }
+    const result = await readLayoutsDoc({ rootDir, label: 'migration 033' });
+    if (!result.ok) return { updated: 0, reason: result.reason };
+    const { doc, path } = result;
 
     let touched = 0;
     for (const layout of doc.layouts) {
@@ -112,7 +97,7 @@ export default {
       return { updated: 0, reason: 'already-applied' };
     }
 
-    await writeFile(path, JSON.stringify(doc, null, 2));
+    await writeLayoutsDoc(path, doc);
     console.log(`📦 migration 033: seeded quick-image widget into ${touched} built-in layout(s).`);
     return { updated: touched };
   },

scripts/migrations/_lib.js

@@ -1,9 +1,13 @@
 /**
- * Shared scaffolding for hash-driven prompt-replace migrations.
+ * Shared scaffolding for migrations. Two families live here:
  *
- * Every prompt-replace migration from 003 onward uses
- * `makePromptReplaceMigration` to collapse onto ~50 lines (hash table + label
- * + customized-skip hint).
+ *   1. Hash-driven prompt-replace migrations — every one from 003 onward uses
+ *      `makePromptReplaceMigration` to collapse onto ~50 lines (hash table +
+ *      label + customized-skip hint).
+ *   2. Dashboard-layout seeding migrations — `readLayoutsDoc` /
+ *      `writeLayoutsDoc` collapse the read → JSON.parse → `Array.isArray`
+ *      guard → write shell shared by every migration that mutates built-in
+ *      layouts in `data/dashboard-layouts.json` (029, 030, 033, …).
  *
  * The runner (`scripts/run-migrations.js`) explicitly skips `_`-prefixed
  * files so this module is never imported as a migration.
@@ -179,3 +183,46 @@ export function makePromptReplaceMigration({
 
   return { applyMigration, up };
 }
+
+/**
+ * Read + parse + guard `data/dashboard-layouts.json` for a layout-seeding
+ * migration. Collapses the preamble every such migration repeats: resolve the
+ * path, read the file (absent → fresh install, nothing to do), JSON-parse it
+ * (unreadable → skip), and verify `doc.layouts` is an array.
+ *
+ * Returns a discriminated result:
+ * - `{ ok: false, reason: 'no-state' | 'unreadable' | 'no-layouts-array', path }`
+ *   — the caller short-circuits with `return { updated: 0, reason: result.reason }`.
+ * - `{ ok: true, doc, path }` — mutate `doc.layouts` in place, then persist
+ *   with `writeLayoutsDoc(path, doc)`.
+ *
+ * `label` is the migration's human tag (e.g. `'migration 029'`); it keeps the
+ * no-state / unreadable log lines per-migration identifiable.
+ */
+export async function readLayoutsDoc({ rootDir, label }) {
+  const path = join(rootDir, 'data', 'dashboard-layouts.json');
+  const raw = await readFile(path, 'utf-8').catch((err) => {
+    if (err.code === 'ENOENT') return null;
+    throw err;
+  });
+  if (raw == null) {
+    console.log(`📦 ${label}: no dashboard-layouts.json yet — fresh install will seed from defaults.`);
+    return { ok: false, reason: 'no-state', path };
+  }
+  let doc;
+  try {
+    doc = JSON.parse(raw);
+  } catch {
+    console.log(`📦 ${label}: dashboard-layouts.json unreadable — skipping.`);
+    return { ok: false, reason: 'unreadable', path };
+  }
+  if (!doc || !Array.isArray(doc.layouts)) {
+    return { ok: false, reason: 'no-layouts-array', path };
+  }
+  return { ok: true, doc, path };
+}
+
+/** Persist a layouts doc with the canonical 2-space indentation. */
+export async function writeLayoutsDoc(path, doc) {
+  await writeFile(path, JSON.stringify(doc, null, 2));
+}

scripts/migrations/_lib.test.js

@@ -13,7 +13,7 @@ import { mkdtempSync, rmSync, writeFileSync, readFileSync, mkdirSync, existsSync
 import { tmpdir } from 'os';
 import { join } from 'path';
 
-import { applyPromptReplaceMigration, md5 } from './_lib.js';
+import { applyPromptReplaceMigration, md5, readLayoutsDoc, writeLayoutsDoc } from './_lib.js';
 
 const FILENAME = 'pipeline-fake.md';
 const BODY_OLD = '# OLD\n';
@@ -108,3 +108,62 @@ describe('applyPromptReplaceMigration opt-ins', () => {
     });
   });
 });
+
+describe('readLayoutsDoc / writeLayoutsDoc', () => {
+  let rootDir;
+  let dataDir;
+  let layoutsPath;
+
+  beforeEach(() => {
+    rootDir = mkdtempSync(join(tmpdir(), 'migration-layouts-'));
+    dataDir = join(rootDir, 'data');
+    mkdirSync(dataDir, { recursive: true });
+    layoutsPath = join(dataDir, 'dashboard-layouts.json');
+  });
+
+  afterEach(() => {
+    rmSync(rootDir, { recursive: true, force: true });
+  });
+
+  it('reports no-state when the file is absent', async () => {
+    const result = await readLayoutsDoc({ rootDir, label: 'migration test' });
+    expect(result.ok).toBe(false);
+    expect(result.reason).toBe('no-state');
+    expect(result.path).toBe(layoutsPath);
+  });
+
+  it('reports unreadable for malformed JSON', async () => {
+    writeFileSync(layoutsPath, 'not json');
+    const result = await readLayoutsDoc({ rootDir, label: 'migration test' });
+    expect(result.ok).toBe(false);
+    expect(result.reason).toBe('unreadable');
+  });
+
+  it('reports no-layouts-array when the layouts key is missing or non-array', async () => {
+    writeFileSync(layoutsPath, JSON.stringify({ activeLayoutId: 'default' }));
+    expect((await readLayoutsDoc({ rootDir, label: 'x' })).reason).toBe('no-layouts-array');
+    writeFileSync(layoutsPath, JSON.stringify({ layouts: 'nope' }));
+    expect((await readLayoutsDoc({ rootDir, label: 'x' })).reason).toBe('no-layouts-array');
+    writeFileSync(layoutsPath, 'null');
+    expect((await readLayoutsDoc({ rootDir, label: 'x' })).reason).toBe('no-layouts-array');
+  });
+
+  it('returns the parsed doc + path when valid', async () => {
+    const doc = { activeLayoutId: 'default', layouts: [{ id: 'default', widgets: [] }] };
+    writeFileSync(layoutsPath, JSON.stringify(doc));
+    const result = await readLayoutsDoc({ rootDir, label: 'migration test' });
+    expect(result.ok).toBe(true);
+    expect(result.path).toBe(layoutsPath);
+    expect(result.doc).toEqual(doc);
+  });
+
+  it('round-trips through writeLayoutsDoc with 2-space indentation', async () => {
+    const doc = { activeLayoutId: 'default', layouts: [{ id: 'default', widgets: ['cos'] }] };
+    await writeLayoutsDoc(layoutsPath, doc);
+    const raw = readFileSync(layoutsPath, 'utf-8');
+    expect(raw).toBe(JSON.stringify(doc, null, 2));
+    const reread = await readLayoutsDoc({ rootDir, label: 'migration test' });
+    expect(reread.ok).toBe(true);
+    expect(reread.doc).toEqual(doc);
+  });
+});