diff --git a/.changelog/NEXT.md b/.changelog/NEXT.md index a85b84df0..a33408012 100644 --- a/.changelog/NEXT.md +++ b/.changelog/NEXT.md @@ -1,6 +1,7 @@ # Unreleased Changes ## Added +- **Federated media collections + per-category sync integrity.** Media collections are now a first-class peer-sync record kind (`mediaCollection` in `PEER_SUBSCRIBABLE_KINDS`, schema-version-gated at `mediaCollections: 1`): they auto-subscribe on create, push per-record on edit, and propagate deletes via tombstones — collections now reach parity between federated machines the way universes/series already did, instead of only riding along as `linkedCollection` cargo or the opt-in 60s snapshot. Required adding soft-delete to collections (`deleted`/`deletedAt`; `deleteCollection` marks instead of splicing; `listCollections({ includeDeleted })`; tombstone-aware LWW merge; tombstone GC), so a delete on one node can't be resurrected by a reverse-subscribed peer. Synced images now carry their **generation prompts**: the peer asset manifest advertises a canonical gen-params-only `sidecarSha256` (machine-local hash-cache stripped so it converges across peers), and the asset-pull worker fetches each image's `.metadata.json` sidecar alongside the bytes — fixing images that previously landed in "Unsorted" with no prompts. A new per-category **sync integrity** surface (`GET /api/peer-sync/manifest`, `GET /api/peer-sync/integrity`, pure `computeRecordIntegrity` diff) drives a `SyncBadge` on every Universe / Series / Media-Collection row (in-sync / diverged / assets-missing / local-only / on-peer-only / "not syncing — enable?") plus a deep-linkable `SyncDetailDrawer` (`/…/:id/sync`) showing the per-peer breakdown, previews, and manual actions: **Sync to peer** (`POST /sync-record`, force-push bypassing the unchanged-hash short-circuit), **Sync now** (`POST /sync-now`, backfill-subscribe + retry a peer's pending pushes), and **Pull missing prompts** (`POST /pull-metadata` + a button on the Unsorted view that backfills sidecars from online peers). `mediaCollections` sync stays **opt-in** per peer (default off); the badge surfaces a clear "enable?" state rather than flipping the default. - **Manual tombstone GC trigger.** The 60s `syncOrchestrator` sweep buffers tombstones for 24h to give every peer time to ack a deletion before the bytes leave disk — fine for normal use, painful right after a mass-delete. New `POST /api/sync/tombstones/sweep` (Zod-validated optional `{ graceMs }` clamped to `[0, 24h]`) calls `sweepTombstones({ graceMs })` so callers can shrink the buffer; the per-kind null-cutoff refusal (snapshot-mode peer with no per-record subscription) still fires independently of `graceMs` to preserve resurrection safety. Companion `GET /api/sync/tombstones/status` returns the dry-run refusal status without invoking the prune helpers. A new "GC tombstones now" section on the Instances page (between SelfCard/AddPeer and the peers list) POSTs `graceMs: 0` and surfaces per-kind prune counts + any refusals in a toast; the button is disabled only when every kind is refused. `scripts/gc-tombstones-now.js` is the equivalent CLI path (dry-run by default; `--apply` to prune) for headless cleanups — uses the same "stop the server first, single-process write-tail isn't cross-process safe" wording as `cleanup-test-data.js`. `sweepTombstones` now returns `{ universes, series, issues, refused }` so the manual trigger can explain why a kind didn't shrink. The orchestrator path keeps the 24h default behavior unchanged. - `server/lib/collectionStore.js` — per-type, per-record JSON storage helper with explicit type-level `schemaVersion` stamping. `createCollectionStore({ dir, type, schemaVersion, sanitizeRecord })` returns `loadOne` / `saveOne` / `listIds` / `loadAll` / `deleteOne` / `loadTypeIndex` / `saveTypeIndex` / `verifySchemaVersion`. Per-id write queue means writes to different records run in parallel — the win over a single-file write queue. `verifyCollectionVersions([store, ...])` is called at server boot after `runMigrations()` so a missed migration produces a loud, single-line log per collection (does not crash — PortOS is single-user). - New layout `data/universes/{id}/index.json` for the universe builder, plus a type-level `data/universes/index.json` carrying `{ schemaVersion: 5, type, updatedAt, config: { runs } }`. Migration 034 (`scripts/migrations/034-split-universe-builder-to-per-uuid.js`) splits the legacy monolithic `data/universe-builder.json` (1.4MB / ~30 universes that previously rewrote in full on every mutation) into per-record files at boot. The legacy file is renamed to `universe-builder.json.bak-034` (not deleted) as a recovery path. Idempotent — partial-completion re-runs finish the split; full-completion re-runs are no-ops. diff --git a/PLAN.md b/PLAN.md index 39cfe5a0e..72bae502b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -118,6 +118,17 @@ Items here need a research / design pass, an explicit decision, or a preconditio - [ ] [cos-on-demand-mark-app-review-started-dedupe] **`evaluateTasks` on-demand loop — dedupe `markAppReviewStarted` calls per app.** Surfaced by gemini review (2026-05-23) at `server/services/cos.js:730`. When multiple on-demand requests for the same app are queued in a single evaluation cycle, `markAppReviewStarted` is called once per request — wasted state writes (single-user / single-instance so not a correctness bug, just churn). Collect unique `appId`s before the loop and call once. Nit-level priority. - [ ] [subagent-spawner-top-level-side-effects] **`subAgentSpawner.js` top-level side effects — move init into an explicit `initSpawner()`.** Surfaced by gemini review (2026-05-23) at `server/services/subAgentSpawner.js:153`. Module-load wires event listeners + timers, which interferes with test isolation (each test import re-arms them). Wrap the init block in `export function initSpawner()` and call it from `server/index.js` after dependencies are ready. Treat as a refactor — needs care that no consumer of the module relies on the import-time wiring. - [ ] [ref-watch-phosphene-long-clip-temporal-boost-12fps] **"Long Clip Temporal Boost" (render at 12fps, interpolate to 24fps).** From `reference-watch` review of phosphene (commit `53585fc`, 2026-05-22). Phosphene added a 12fps render path that uses ffmpeg's `minterpolate` / RIFE downstream to fill back to 24fps, doubling the effective clip length per memory budget. PortOS doesn't have an interpolation post-step in `server/services/videoGen/local.js` — adding one is a feature decision, not a bug fix. The math helpers (`_duration_to_8k_frames`, LTX `8k+1` frame-count snapping) are useful regardless; PortOS's `DEFAULT_NUM_FRAMES = 121` is already 8k+1-aligned but UI-side validation could borrow this pattern. **Decision needed:** is this worth a feature surface? No clear user pain motivating it today. + +### Federated media sync — deferred follow-ups (2026-05-23) + +Surfaced during code review of the federated-media-collection-sync feature and consciously deferred (out of scope for that PR). Each is independently pickable. + +- [ ] [media-sync-snapshot-tombstones] **dataSync `mediaCollections` snapshot omits tombstones.** `getMediaCollectionsSnapshot` (`server/services/dataSync.js`) lists live-only collections, so a peer with the `mediaCollections` category enabled but no per-record subscription yet (transient window before auto-subscribe lands) won't receive collection tombstones via the 60s snapshot — only via the push path. Either include tombstones in the snapshot (bounded by a horizon, like the universe/series snapshots) or document that first-class push is the authoritative tombstone path for collections. +- [ ] [media-sync-integrity-unreachable-reason-ui] **Integrity: surface the `peer-unreachable` vs `peer-too-old` reason in the UI.** Server-side is done — `getPeerIntegrity` (`server/services/sharing/integrity.js`) now returns `reason: 'peer-unreachable'` for a network failure, `'peer-too-old'` for a 404, and `'fetch-failed'` for other non-OK statuses. The remaining gap is client-side: `useSyncIntegrity` only checks `available` and silently drops the `reason`, so the badge/drawer can't tell the user "peer is offline" vs "update the peer's PortOS." Thread `reason` through the hook's per-peer payload and render distinct messaging in `SyncBadge`/`SyncDetailDrawer`. +- [ ] [media-sync-series-issue-assets-manifest] **Series integrity manifest omits child-issue assets (v1 limitation).** `assetShaListForRecord('series', …)` (`server/services/sharing/peerSync.js`) uses `buildAssetManifest(series)`, capturing only the series' own asset refs, not its issues' — so a series whose images all live in issues reports assets in-parity even when a peer is missing them. Extend to fold in child-issue asset hashes (mirror `buildAssetManifestForSeries`). +- [ ] [media-sync-metadata-missing-status] **(Enhancement) `metadata-missing` integrity status.** Surface records whose images lack gen-params sidecars as a distinct status so the badge directly flags the "no prompts" condition (today repaired via the Unsorted "Pull missing prompts" action; byte-divergence shows as `assets-missing`). Requires the peer manifest to carry per-record sidecar-presence. `server/lib/syncIntegrity.js` + the manifest builder. +- [ ] [media-sync-drawer-test-act-warning] **Test polish: `SyncDetailDrawer.test.jsx` act() warning.** The universe-kind "shows per-peer breakdown" test renders without awaiting the async record-name fetch, emitting a React `act()` warning (test still passes). Wrap the render/assertions in `waitFor`/`act`. Test-quality only. + ## Future Ideas - **Identity Context Injection** — per-task-type digital twin preamble toggle. diff --git a/client/src/App.jsx b/client/src/App.jsx index 1a7afba9b..8ec809d5d 100644 --- a/client/src/App.jsx +++ b/client/src/App.jsx @@ -37,6 +37,8 @@ const VideoGen = lazyWithReload(() => import('./pages/VideoGen')); const MediaHistory = lazyWithReload(() => import('./pages/MediaHistory')); const MediaCollections = lazyWithReload(() => import('./pages/MediaCollections')); const MediaCollectionDetail = lazyWithReload(() => import('./pages/MediaCollectionDetail')); +const MediaCollectionSyncView = lazyWithReload(() => import('./pages/MediaCollectionSyncView')); +const SyncView = lazyWithReload(() => import('./pages/SyncView')); const MediaModels = lazyWithReload(() => import('./pages/MediaModels')); const Loras = lazyWithReload(() => import('./pages/Loras')); const UniverseBuilder = lazyWithReload(() => import('./pages/UniverseBuilder')); @@ -218,6 +220,7 @@ export default function App() { } /> } /> } /> + } /> } /> } /> } /> @@ -246,6 +249,7 @@ export default function App() { } /> } /> } /> + } /> } /> {/* Legacy /universe-builder* → /universes* (route renamed when the index landed). Keeps old bookmarks + in-app deep-links working. */} @@ -258,6 +262,7 @@ export default function App() { } /> } /> } /> + } /> } /> } /> } /> diff --git a/client/src/components/sync/SyncBadge.jsx b/client/src/components/sync/SyncBadge.jsx new file mode 100644 index 000000000..2a34b962e --- /dev/null +++ b/client/src/components/sync/SyncBadge.jsx @@ -0,0 +1,80 @@ +/** + * SyncBadge — presentational-only sync status badge. + * + * Props: + * status — 'in-parity' | 'diverged' | 'assets-missing' | 'local-only' | + * 'peer-only' | 'not-syncing' | 'unknown' | null | undefined + * onClick — called when the badge button is clicked (e.g. open detail drawer) + */ + +import { CheckCircle2, AlertTriangle, WifiOff, HelpCircle } from 'lucide-react'; + +const STATUS_CONFIG = { + 'in-parity': { + label: 'In sync', + className: 'bg-port-success/15 text-port-success hover:bg-port-success/25', + Icon: CheckCircle2, + title: 'All peers in parity', + }, + diverged: { + label: 'Diverged', + className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25', + Icon: AlertTriangle, + title: 'Content differs from at least one peer', + }, + 'assets-missing': { + label: 'Assets missing', + className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25', + Icon: AlertTriangle, + title: 'Record present but associated files are missing on a peer', + }, + 'local-only': { + label: 'Local only', + className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25', + Icon: AlertTriangle, + title: 'Not present on at least one peer', + }, + 'peer-only': { + label: 'On peer only', + className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25', + Icon: AlertTriangle, + title: 'Present on peer but not local', + }, + 'not-syncing': { + label: 'Not syncing', + className: 'bg-gray-600/20 text-gray-400 hover:bg-gray-600/30', + Icon: WifiOff, + // True when no online peer is syncing THIS category — either no peers are + // sync-enabled at all, or they are but have this category turned off. + title: 'No peers syncing this category — enable it for a peer?', + }, + unknown: { + label: 'Sync unknown', + className: 'bg-gray-600/20 text-gray-400 hover:bg-gray-600/30', + Icon: HelpCircle, + // Sync IS configured, but every eligible peer was unreachable / too-old / + // errored, so we couldn't compute parity. Distinct from 'not-syncing'. + title: 'Sync status unavailable — peer offline, unreachable, or on an older PortOS', + }, +}; + +export default function SyncBadge({ status, onClick }) { + if (!status) return null; + + const config = STATUS_CONFIG[status]; + if (!config) return null; + + const { label, className, Icon, title } = config; + + return ( + + ); +} diff --git a/client/src/components/sync/SyncBadge.test.jsx b/client/src/components/sync/SyncBadge.test.jsx new file mode 100644 index 000000000..cbd97fd62 --- /dev/null +++ b/client/src/components/sync/SyncBadge.test.jsx @@ -0,0 +1,79 @@ +import { describe, it, expect, vi } from 'vitest'; +import { render, screen, fireEvent } from '@testing-library/react'; +import SyncBadge from './SyncBadge'; + +describe('SyncBadge', () => { + it('renders "In sync" for in-parity status', () => { + render( {}} />); + expect(screen.getByRole('button', { name: /in sync/i })).toBeInTheDocument(); + expect(screen.getByRole('button').className).toMatch(/port-success/); + }); + + it('renders "Diverged" for diverged status', () => { + render( {}} />); + expect(screen.getByRole('button', { name: /diverged/i })).toBeInTheDocument(); + expect(screen.getByRole('button').className).toMatch(/port-warning/); + }); + + it('renders "Assets missing" for assets-missing status', () => { + render( {}} />); + expect(screen.getByText('Assets missing')).toBeInTheDocument(); + expect(screen.getByRole('button').className).toMatch(/port-warning/); + }); + + it('renders "Local only" for local-only status', () => { + render( {}} />); + expect(screen.getByText('Local only')).toBeInTheDocument(); + expect(screen.getByRole('button').className).toMatch(/port-warning/); + }); + + it('renders "On peer only" for peer-only status', () => { + render( {}} />); + expect(screen.getByText('On peer only')).toBeInTheDocument(); + expect(screen.getByRole('button').className).toMatch(/port-warning/); + }); + + it('renders "Not syncing" with distinct neutral styling for not-syncing status', () => { + render( {}} />); + const btn = screen.getByRole('button', { name: /not syncing/i }); + expect(btn).toBeInTheDocument(); + // Must not use warning or success colors — visually distinct/neutral + expect(btn.className).not.toMatch(/port-warning/); + expect(btn.className).not.toMatch(/port-success/); + expect(btn.className).toMatch(/gray/); + }); + + it('renders "Sync unknown" with neutral styling for unknown status', () => { + render( {}} />); + const btn = screen.getByRole('button', { name: /sync unknown/i }); + expect(btn).toBeInTheDocument(); + // Neutral, like not-syncing — not a warning/success state. + expect(btn.className).not.toMatch(/port-warning/); + expect(btn.className).not.toMatch(/port-success/); + expect(btn.className).toMatch(/gray/); + }); + + it('calls onClick when clicked', () => { + const onClick = vi.fn(); + render(); + fireEvent.click(screen.getByRole('button')); + expect(onClick).toHaveBeenCalledTimes(1); + }); + + it('renders nothing for null status', () => { + const { container } = render( {}} />); + expect(container.firstChild).toBeNull(); + }); + + it('renders nothing for undefined status', () => { + const { container } = render( {}} />); + expect(container.firstChild).toBeNull(); + }); + + it('has a descriptive title attribute', () => { + render( {}} />); + const btn = screen.getByRole('button'); + expect(btn.title).toBeTruthy(); + expect(btn.title.length).toBeGreaterThan(5); + }); +}); diff --git a/client/src/components/sync/SyncDetailDrawer.jsx b/client/src/components/sync/SyncDetailDrawer.jsx new file mode 100644 index 000000000..11a18a0b9 --- /dev/null +++ b/client/src/components/sync/SyncDetailDrawer.jsx @@ -0,0 +1,401 @@ +/** + * SyncDetailDrawer — deep-linkable right-side panel showing per-peer sync + * status for a single record, plus action buttons. + * + * Reusable across kinds: 'mediaCollection', 'universe', 'series'. + * + * Props: + * kind — record kind ('mediaCollection' | 'universe' | 'series') + * recordId — the record's id (from URL param) + * onClose — called to navigate back (e.g. useNavigate() back to list) + * + * Deep-linkability: mounted under kind-specific routes via SyncView. + * Fetches all its own data so it loads standalone from a direct URL. + */ + +import { useEffect, useState, useCallback, useRef } from 'react'; +import { X, RefreshCw, ArrowUpCircle, Download, CheckCircle2, AlertTriangle, WifiOff, Loader2 } from 'lucide-react'; +import toast from '../ui/Toast'; +import { useSyncIntegrity } from '../../hooks/useSyncIntegrity'; +import { useAsyncAction } from '../../hooks/useAsyncAction'; +import { getMediaCollection, getUniverse, getPipelineSeries, syncRecordToPeer, pullMissingMetadata } from '../../services/api'; +import MediaImage from '../MediaImage'; + +// ── Per-kind record fetcher ────────────────────────────────────────────────── +// Returns a promise resolving to { name, ...rest } for use in the drawer header. +// Only mediaCollection additionally exposes an `items` array for the thumbnail grid. +const KIND_FETCHER = { + mediaCollection: getMediaCollection, + universe: getUniverse, + series: getPipelineSeries, +}; + +// ── Per-status display config ──────────────────────────────────────────────── +const STATUS_CONFIG = { + 'in-parity': { + label: 'In parity', + className: 'text-port-success', + Icon: CheckCircle2, + }, + diverged: { + label: 'Diverged', + className: 'text-port-warning', + Icon: AlertTriangle, + }, + 'assets-missing': { + label: 'Assets missing', + className: 'text-port-warning', + Icon: AlertTriangle, + }, + 'local-only': { + label: 'Local only', + className: 'text-port-warning', + Icon: AlertTriangle, + }, + 'peer-only': { + label: 'On peer only', + className: 'text-port-warning', + Icon: AlertTriangle, + }, +}; + +function StatusPill({ status }) { + const config = STATUS_CONFIG[status]; + if (!config) return {status ?? '—'}; + const { label, className, Icon } = config; + return ( + + + ); +} + +// ── Collection preview (mediaCollection kind) ──────────────────────────────── +// Presentational only — the collection state is owned by the drawer (fetched +// once there) and passed down, so the "Pull missing metadata" action can read +// the same already-loaded record without a second fetch. +function CollectionPreview({ collection, loading }) { + if (loading) { + return ( +
+ + Loading… +
+ ); + } + if (!collection) { + return

Collection not found.

; + } + + const items = collection.items ?? []; + const imageItems = items.filter((it) => it.kind === 'image').slice(0, 8); + + return ( +
+
+

{collection.name}

+

{items.length} item{items.length !== 1 ? 's' : ''}

+
+ {imageItems.length > 0 && ( +
+ {imageItems.map((it) => ( + + ))} +
+ )} + {imageItems.length === 0 && ( +

No image thumbnails available.

+ )} +
+ ); +} + +// Friendly labels for the `{ pushed:false, reason }` shapes the server's +// forcePushRecord → pushRecordToPeer can return with HTTP 200 (the push was +// accepted as a request but no bytes actually went out). Unmapped reasons +// (e.g. `http-409`) fall back to the raw string. +const PUSH_SKIP_LABELS = { + 'category-disabled': 'this category is not enabled for that peer', + 'peer-disallows-outbound': 'that peer does not accept outbound sync', + 'peer-not-found': 'peer not found', + 'record-not-found': 'record missing locally', + 'peer-schema-behind': 'peer is on an older PortOS version', + 'peer-schema-behind-cooldown': 'peer is on an older PortOS version', + 'invalid-subscription': 'subscription is invalid', + unchanged: 'already up to date', + network: 'network error reaching the peer', +}; + +// ── Peer row with per-peer sync action ─────────────────────────────────────── +function PeerRow({ entry, kind, recordId, onRefresh }) { + const { peerId, peerName, status } = entry; + const needsAction = status !== 'in-parity'; + + const [syncToPeer, syncing] = useAsyncAction(async () => { + // The endpoint returns 200 even when nothing was pushed ({ pushed:false, + // reason }) — toasting success unconditionally would mislead the user. + const result = await syncRecordToPeer(peerId, kind, recordId, { silent: true }); + if (result?.pushed) { + toast.success(`Synced to ${peerName}`); + } else { + const reason = result?.reason; + const detail = reason ? ` — ${PUSH_SKIP_LABELS[reason] ?? reason}` : ''; + toast.error(`Nothing synced to ${peerName}${detail}`); + } + onRefresh(); + }, { errorMessage: `Failed to sync to ${peerName}` }); + + return ( +
+
+

{peerName}

+ +
+ {needsAction && ( + + )} +
+ ); +} + +// ── Main drawer ────────────────────────────────────────────────────────────── +export default function SyncDetailDrawer({ kind, recordId, onClose }) { + const { byPeer, noSyncingPeers, integrityUnavailable, loading, error, refresh } = useSyncIntegrity(kind); + const peerEntries = byPeer.get(recordId) ?? []; + + // Record state is owned here (fetched once) so the preview and the + // "Pull missing metadata" action share the same record — no second fetch + // and no double-toast (kind fetchers have no {silent} support). + // For non-mediaCollection kinds we still fetch to show the record name in + // the header, but skip the thumbnail grid and pull-metadata button. + const fetcher = KIND_FETCHER[kind] ?? null; + const [record, setRecord] = useState(null); + const [recordLoading, setRecordLoading] = useState(!!fetcher); + + // Drop async results that resolve after the drawer unmounts (fast route + // change / close while a fetch is in flight) to avoid setState-on-unmounted + // warnings. Never reset to true — handles dev-mode double-mount cleanly. + const mountedRef = useRef(true); + useEffect(() => () => { mountedRef.current = false; }, []); + // Generation counter so only the LATEST in-flight fetch commits state — a + // rapid recordId change (or switch to empty) bumps this, and an older fetch + // that resolves afterward fails the equality check and is dropped, instead + // of overwriting the newer record with a stale name/preview. + const loadGenRef = useRef(0); + + const loadRecord = useCallback(() => { + if (!fetcher) return; + const gen = ++loadGenRef.current; // invalidates any prior in-flight fetch + const fresh = () => mountedRef.current && gen === loadGenRef.current; + // An empty recordId (e.g. a param-less route mount) would fetch + // `/media/collections/` and 404/toast — skip the request, and clear any + // previously-loaded record so a stale name/preview can't linger. + if (!recordId) { setRecord(null); setRecordLoading(false); return; } + setRecordLoading(true); + fetcher(recordId) + .then((data) => { if (fresh()) setRecord(data); }) + .catch(() => { if (fresh()) setRecord(null); }) + .finally(() => { if (fresh()) setRecordLoading(false); }); + }, [fetcher, recordId]); + + useEffect(() => { loadRecord(); }, [loadRecord]); + + // Keep the mediaCollection-specific alias in scope so the pull action below + // can read it without changing its reference to `collection`. + const collection = kind === 'mediaCollection' ? record : null; + const collectionLoading = kind === 'mediaCollection' ? recordLoading : false; + + // Convenience alias so the header can show `record?.name` regardless of kind. + const recordName = record?.name ?? null; + + // Read the image filenames from the already-loaded collection state. + // Only runs for mediaCollection — the button is gated to that kind below. + const [pullMissing, pulling] = useAsyncAction(async () => { + if (!collection) { toast.error('Collection not loaded yet'); return; } + const filenames = (collection.items ?? []) + .filter((it) => it.kind === 'image') + .map((it) => it.ref); + if (filenames.length === 0) { toast('No image files to pull'); return; } + const result = await pullMissingMetadata(filenames, { silent: true }); + const recovered = result?.recovered ?? 0; + const attempted = result?.attempted ?? filenames.length; + // Mirror MediaCollectionDetail's Unsorted "Pull missing prompts": only + // claim success when something was actually recovered — recovered=0 (or + // attempted=0) is a neutral "nothing to do", not a win. + if (recovered > 0) { + toast.success(`Pulled ${recovered}/${attempted} metadata item${attempted === 1 ? '' : 's'}`); + } else { + toast(`No missing metadata found (${attempted} checked)`); + } + refresh(); + loadRecord(); // refresh the preview thumbnails post-pull + }, { errorMessage: 'Failed to pull missing metadata' }); + + // Esc key support + const handleKeyDown = useCallback((e) => { + if (e.key === 'Escape') onClose(); + }, [onClose]); + useEffect(() => { + window.addEventListener('keydown', handleKeyDown); + return () => window.removeEventListener('keydown', handleKeyDown); + }, [handleKeyDown]); + + // Lock the body scroll while the drawer is open (matches Drawer.jsx). + useEffect(() => { + const prevOverflow = document.body.style.overflow; + document.body.style.overflow = 'hidden'; + return () => { document.body.style.overflow = prevOverflow; }; + }, []); + + return ( + <> + {/* Backdrop */} + diff --git a/client/src/pages/MediaCollectionDetail.test.jsx b/client/src/pages/MediaCollectionDetail.test.jsx new file mode 100644 index 000000000..887c0cf18 --- /dev/null +++ b/client/src/pages/MediaCollectionDetail.test.jsx @@ -0,0 +1,241 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { MemoryRouter, Route, Routes } from 'react-router-dom'; + +// ── Mocks must be declared before any imports that use them ────────────────── + +const mockPullMissingMetadata = vi.fn(); +const mockListImageGallery = vi.fn(); +const mockListVideoHistory = vi.fn(); +const mockListMediaCollections = vi.fn(); +const mockGetMediaCollection = vi.fn(); + +vi.mock('../services/api', () => ({ + listImageGallery: (...args) => mockListImageGallery(...args), + listVideoHistory: (...args) => mockListVideoHistory(...args), + listMediaCollections: (...args) => mockListMediaCollections(...args), + getMediaCollection: (...args) => mockGetMediaCollection(...args), + updateMediaCollection: vi.fn(), + addMediaCollectionItem: vi.fn(), + removeMediaCollectionItem: vi.fn(), + deleteImage: vi.fn(), + deleteVideoHistoryItem: vi.fn(), + pullMissingMetadata: (...args) => mockPullMissingMetadata(...args), +})); + +vi.mock('../components/ui/Toast', () => ({ + default: Object.assign(vi.fn(), { + success: vi.fn(), + error: vi.fn(), + warning: vi.fn(), + }), +})); + +vi.mock('../hooks/useMediaAnnotations', () => ({ + useMediaAnnotations: () => ({ + annotations: {}, + toggleStar: vi.fn(), + updateAnnotation: vi.fn(), + getCardProps: () => ({}), + }), +})); + +vi.mock('../hooks/useMediaPreviewActions', () => ({ + default: () => ({ + handleRemix: vi.fn(), + handleSendToVideo: vi.fn(), + handleContinue: vi.fn(), + handleClean: vi.fn(), + }), +})); + +vi.mock('../hooks/usePreviewRoute', () => ({ + default: () => [null, vi.fn()], +})); + +vi.mock('../components/media/MediaCard', () => ({ + default: ({ item }) =>
{item.filename || item.key}
, +})); + +vi.mock('../components/media/MediaPreview', () => ({ + default: () => null, +})); + +vi.mock('../components/media/BulkTargetPicker', () => ({ + default: () => null, +})); + +vi.mock('../components/sharing/ShareToButton', () => ({ + default: () => null, +})); + +vi.mock('../components/media/normalize', () => ({ + normalizeImage: (i) => ({ + kind: 'image', + key: `image:${i.filename}`, + filename: i.filename, + ref: i.filename, + }), + normalizeVideo: (v) => ({ + kind: 'video', + key: `video:${v.id}`, + id: v.id, + ref: v.id, + }), +})); + +import toast from '../components/ui/Toast'; +import MediaCollectionDetail from './MediaCollectionDetail'; +import { UNSORTED_ID } from '../lib/unsorted'; + +// ── Fixtures ───────────────────────────────────────────────────────────────── + +const IMAGE_A = { filename: 'a.png', createdAt: '2024-01-02' }; +const IMAGE_B = { filename: 'b.png', createdAt: '2024-01-01' }; +const VIDEO_C = { id: 'vid-c', createdAt: '2024-01-03' }; + +// A collection that contains IMAGE_A (so IMAGE_B and VIDEO_C are "unsorted"). +const REAL_COLLECTION = { + id: 'col-real', + name: 'My Collection', + items: [{ kind: 'image', ref: IMAGE_A.filename, addedAt: '2024-01-02' }], +}; + +function renderUnsorted() { + return render( + + + } /> + + , + ); +} + +function renderReal() { + return render( + + + } /> + + , + ); +} + +// ── Setup default mock return values ───────────────────────────────────────── + +beforeEach(() => { + vi.clearAllMocks(); + // Default: all three images + video; one real collection that contains IMAGE_A + mockListImageGallery.mockResolvedValue([IMAGE_A, IMAGE_B]); + mockListVideoHistory.mockResolvedValue([VIDEO_C]); + mockListMediaCollections.mockResolvedValue([REAL_COLLECTION]); + mockGetMediaCollection.mockResolvedValue(REAL_COLLECTION); + mockPullMissingMetadata.mockResolvedValue({ attempted: 1, recovered: 1 }); +}); + +// ── Unsorted view tests ─────────────────────────────────────────────────────── + +describe('MediaCollectionDetail — Unsorted view', () => { + it('renders "Pull missing prompts" button on the unsorted view', async () => { + renderUnsorted(); + await waitFor(() => { + expect(screen.getByRole('button', { name: /pull missing prompts/i })).toBeInTheDocument(); + }); + }); + + it('button is disabled when there are no unsorted images', async () => { + // Make every image belong to the real collection so nothing is unsorted. + mockListMediaCollections.mockResolvedValue([ + { + ...REAL_COLLECTION, + items: [ + { kind: 'image', ref: IMAGE_A.filename, addedAt: '2024-01-02' }, + { kind: 'image', ref: IMAGE_B.filename, addedAt: '2024-01-01' }, + { kind: 'video', ref: VIDEO_C.id, addedAt: '2024-01-03' }, + ], + }, + ]); + renderUnsorted(); + await waitFor(() => { + const btn = screen.getByRole('button', { name: /pull missing prompts/i }); + expect(btn).toBeDisabled(); + }); + }); + + it('calls pullMissingMetadata with only image filenames (not video ids)', async () => { + const user = userEvent.setup(); + renderUnsorted(); + await waitFor(() => screen.getByRole('button', { name: /pull missing prompts/i })); + + await user.click(screen.getByRole('button', { name: /pull missing prompts/i })); + + await waitFor(() => expect(mockPullMissingMetadata).toHaveBeenCalledOnce()); + // IMAGE_B is unsorted (IMAGE_A is in col-real); VIDEO_C should not appear. + const [filenames] = mockPullMissingMetadata.mock.calls[0]; + expect(filenames).toContain(IMAGE_B.filename); + expect(filenames).not.toContain(IMAGE_A.filename); + expect(filenames).not.toContain(VIDEO_C.id); + }); + + it('toasts success when prompts are recovered', async () => { + mockPullMissingMetadata.mockResolvedValue({ attempted: 1, recovered: 1 }); + const user = userEvent.setup(); + renderUnsorted(); + await waitFor(() => screen.getByRole('button', { name: /pull missing prompts/i })); + + await user.click(screen.getByRole('button', { name: /pull missing prompts/i })); + + await waitFor(() => expect(toast.success).toHaveBeenCalledWith( + expect.stringMatching(/recovered prompts for 1\/1/i), + )); + }); + + it('toasts neutral message when no prompts are found', async () => { + mockPullMissingMetadata.mockResolvedValue({ attempted: 1, recovered: 0 }); + const user = userEvent.setup(); + renderUnsorted(); + await waitFor(() => screen.getByRole('button', { name: /pull missing prompts/i })); + + await user.click(screen.getByRole('button', { name: /pull missing prompts/i })); + + await waitFor(() => expect(toast).toHaveBeenCalledWith( + expect.stringMatching(/no missing prompts found/i), + )); + }); + + it('refreshes the image list after a successful pull', async () => { + // First load: IMAGE_B unsorted. After pull: IMAGE_B is now in a collection. + const updatedCollection = { + ...REAL_COLLECTION, + items: [ + ...REAL_COLLECTION.items, + { kind: 'image', ref: IMAGE_B.filename, addedAt: '2024-01-01' }, + ], + }; + mockPullMissingMetadata.mockResolvedValue({ attempted: 1, recovered: 1 }); + mockListMediaCollections + .mockResolvedValueOnce([REAL_COLLECTION]) // initial load + .mockResolvedValue([updatedCollection]); // refresh after pull + + const user = userEvent.setup(); + renderUnsorted(); + await waitFor(() => screen.getByRole('button', { name: /pull missing prompts/i })); + await user.click(screen.getByRole('button', { name: /pull missing prompts/i })); + + // listMediaCollections should be called a second time (the refresh). + await waitFor(() => { + expect(mockListMediaCollections.mock.calls.length).toBeGreaterThanOrEqual(2); + }); + }); +}); + +// ── Non-unsorted view: button must NOT appear ───────────────────────────────── + +describe('MediaCollectionDetail — regular collection view', () => { + it('does NOT render "Pull missing prompts" on a real collection', async () => { + renderReal(); + await waitFor(() => screen.getByText('My Collection')); + expect(screen.queryByRole('button', { name: /pull missing prompts/i })).toBeNull(); + }); +}); diff --git a/client/src/pages/MediaCollectionSyncView.jsx b/client/src/pages/MediaCollectionSyncView.jsx new file mode 100644 index 000000000..97e654348 --- /dev/null +++ b/client/src/pages/MediaCollectionSyncView.jsx @@ -0,0 +1,29 @@ +/** + * MediaCollectionSyncView — route page for /media/collections/:id/sync. + * + * Deep-linkable: renders the SyncDetailDrawer overlaid on top of the + * MediaGen layout. `onClose` navigates back to the collections list so the + * user lands on /media/collections after dismissing the drawer. + */ + +import { useParams, useNavigate } from 'react-router-dom'; +import SyncDetailDrawer from '../components/sync/SyncDetailDrawer'; + +export default function MediaCollectionSyncView() { + const { id } = useParams(); + const navigate = useNavigate(); + + const handleClose = () => { + navigate('/media/collections'); + }; + + return ( + + ); +} diff --git a/client/src/pages/MediaCollections.jsx b/client/src/pages/MediaCollections.jsx index 06af0c0b8..7553e4591 100644 --- a/client/src/pages/MediaCollections.jsx +++ b/client/src/pages/MediaCollections.jsx @@ -1,5 +1,5 @@ import { useEffect, useMemo, useState } from 'react'; -import { Link } from 'react-router-dom'; +import { Link, useNavigate } from 'react-router-dom'; import { Plus, FolderOpen, Inbox, Trash2, Image as ImageIcon, Film } from 'lucide-react'; import toast from '../components/ui/Toast'; import { @@ -7,6 +7,8 @@ import { listVideoHistory, listImageGallery, } from '../services/api'; import { buildUnsortedCollection } from '../lib/unsorted'; +import SyncBadge from '../components/sync/SyncBadge'; +import { useSyncIntegrity, syncBadgeStatus } from '../hooks/useSyncIntegrity'; // Resolve a collection's cover-thumbnail URL. Default = newest item by // addedAt; user-pinned coverKey wins when set. We need full image/video @@ -47,6 +49,7 @@ const resolveCover = (collection, imagesByName, videosById) => { }; export default function MediaCollections() { + const navigate = useNavigate(); const [collections, setCollections] = useState([]); const [loading, setLoading] = useState(true); const [creating, setCreating] = useState(false); @@ -54,6 +57,10 @@ export default function MediaCollections() { const [imagesByName, setImagesByName] = useState(new Map()); const [videosById, setVideosById] = useState(new Map()); + // Sync integrity — no peers prop (the page doesn't fetch peers itself), + // so the hook fetches instances internally. + const sync = useSyncIntegrity('mediaCollection'); + const refresh = async () => { setLoading(true); const [cols, images, videos] = await Promise.all([ @@ -167,16 +174,24 @@ export default function MediaCollections() { {c.counts.image === 0 && c.counts.video === 0 && Empty}
- {!c.synthetic && ( - - )} +
+ {!c.synthetic && ( + navigate(`/media/collections/${encodeURIComponent(c.id)}/sync`)} + /> + )} + {!c.synthetic && ( + + )} +
))} diff --git a/client/src/pages/MediaCollections.test.jsx b/client/src/pages/MediaCollections.test.jsx new file mode 100644 index 000000000..45ca479f8 --- /dev/null +++ b/client/src/pages/MediaCollections.test.jsx @@ -0,0 +1,84 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { render, screen, waitFor } from '@testing-library/react'; +import { MemoryRouter } from 'react-router-dom'; + +// ── Mock API calls ─────────────────────────────────────────────────────────── +vi.mock('../services/api', () => ({ + listMediaCollections: vi.fn().mockResolvedValue([ + { id: 'col-1', name: 'Alpha', items: [{ kind: 'image', ref: 'img1.png', addedAt: '2024-01-01' }] }, + { id: 'col-2', name: 'Beta', items: [] }, + ]), + createMediaCollection: vi.fn(), + deleteMediaCollection: vi.fn(), + listVideoHistory: vi.fn().mockResolvedValue([]), + listImageGallery: vi.fn().mockResolvedValue([]), +})); + +// ── Mock useSyncIntegrity ──────────────────────────────────────────────────── +const statusById = new Map([['col-1', 'in-parity'], ['col-2', 'diverged']]); +vi.mock('../hooks/useSyncIntegrity', () => ({ + useSyncIntegrity: () => ({ + statusById, + noSyncingPeers: false, + integrityUnavailable: false, + loading: false, + error: null, + refresh: vi.fn(), + byPeer: new Map(), + }), + // Mirror the real precedence helper so badge-status assertions stay valid. + syncBadgeStatus: (sync, recordId) => ( + sync.noSyncingPeers + ? 'not-syncing' + : (sync.statusById.get(recordId) ?? (sync.integrityUnavailable ? 'unknown' : undefined)) + ), +})); + +// ── Mock buildUnsortedCollection ───────────────────────────────────────────── +vi.mock('../lib/unsorted', () => ({ + buildUnsortedCollection: () => ({ + id: '__unsorted__', + name: 'Unsorted', + items: [], + synthetic: true, + }), +})); + +import MediaCollections from './MediaCollections'; + +function renderPage() { + return render( + + + , + ); +} + +describe('MediaCollections', () => { + beforeEach(() => { vi.clearAllMocks(); }); + + it('renders collection names after loading', async () => { + renderPage(); + await waitFor(() => { + expect(screen.getByText('Alpha')).toBeInTheDocument(); + expect(screen.getByText('Beta')).toBeInTheDocument(); + }); + }); + + it('renders a SyncBadge per non-synthetic collection row', async () => { + renderPage(); + await waitFor(() => screen.getByText('Alpha')); + // 'in-parity' badge on col-1, 'diverged' on col-2 + expect(screen.getByText('In sync')).toBeInTheDocument(); + expect(screen.getByText('Diverged')).toBeInTheDocument(); + }); + + it('does not render a SyncBadge for the synthetic Unsorted collection', async () => { + renderPage(); + await waitFor(() => screen.getByText('Alpha')); + // Unsorted is synthetic — only 2 badges for the 2 real collections + const badges = screen.getAllByRole('button', { name: /in sync|diverged|assets missing|local only|on peer only|not syncing/i }); + // Should be exactly 2 (one per real collection) + expect(badges.length).toBe(2); + }); +}); diff --git a/client/src/pages/Pipeline.jsx b/client/src/pages/Pipeline.jsx index 113095718..5b68b37a5 100644 --- a/client/src/pages/Pipeline.jsx +++ b/client/src/pages/Pipeline.jsx @@ -8,12 +8,14 @@ */ import { useEffect, useState } from 'react'; -import { Link } from 'react-router-dom'; +import { Link, useNavigate } from 'react-router-dom'; import { Plus, Workflow as WorkflowIcon, Trash2, Loader2, Globe2 } from 'lucide-react'; import toast from '../components/ui/Toast'; import ShareToButton from '../components/sharing/ShareToButton'; import SyncToPeerButton from '../components/sharing/SyncToPeerButton'; import OriginBadge from '../components/sharing/OriginBadge'; +import SyncBadge from '../components/sync/SyncBadge'; +import { useSyncIntegrity, syncBadgeStatus } from '../hooks/useSyncIntegrity'; import { listPipelineSeries, createPipelineSeries, @@ -39,9 +41,12 @@ const emptyForm = () => ({ }); export default function Pipeline() { + const navigate = useNavigate(); const [series, setSeries] = useState([]); const [universes, setWorlds] = useState([]); const [loading, setLoading] = useState(true); + + const sync = useSyncIntegrity('series'); const [creating, setCreating] = useState(false); const [showForm, setShowForm] = useState(false); const [form, setForm] = useState(emptyForm); @@ -354,6 +359,10 @@ export default function Pipeline() { ) : null} + navigate(`/pipeline/series/${encodeURIComponent(s.id)}/sync`)} + /> + + ); + }, +})); + +import SyncView from './SyncView'; + +function renderSyncView({ path, route, kind, param, backPath }) { + return render( + + + } /> + + , + ); +} + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe('SyncView', () => { + it('passes kind="universe" and decoded universeId to SyncDetailDrawer', () => { + renderSyncView({ + path: '/universes/uni-abc/sync', + route: '/universes/:universeId/sync', + kind: 'universe', + param: 'universeId', + backPath: '/universes', + }); + expect(screen.getByTestId('kind').textContent).toBe('universe'); + expect(screen.getByTestId('record-id').textContent).toBe('uni-abc'); + }); + + it('passes kind="series" and decoded seriesId to SyncDetailDrawer', () => { + renderSyncView({ + path: '/pipeline/series/ser-xyz/sync', + route: '/pipeline/series/:seriesId/sync', + kind: 'series', + param: 'seriesId', + backPath: '/pipeline', + }); + expect(screen.getByTestId('kind').textContent).toBe('series'); + expect(screen.getByTestId('record-id').textContent).toBe('ser-xyz'); + }); + + it('passes kind="mediaCollection" and decoded id to SyncDetailDrawer', () => { + renderSyncView({ + path: '/media/collections/col-123/sync', + route: '/media/collections/:id/sync', + kind: 'mediaCollection', + param: 'id', + backPath: '/media/collections', + }); + expect(screen.getByTestId('kind').textContent).toBe('mediaCollection'); + expect(screen.getByTestId('record-id').textContent).toBe('col-123'); + }); + + it('decodes percent-encoded record ids from the URL', () => { + renderSyncView({ + path: '/universes/my%20universe/sync', + route: '/universes/:universeId/sync', + kind: 'universe', + param: 'universeId', + backPath: '/universes', + }); + expect(screen.getByTestId('record-id').textContent).toBe('my universe'); + }); + + it('navigates to backPath when onClose is called', async () => { + // Use a two-route setup so navigate has somewhere to go. + const { getByRole, queryByTestId } = render( + + + universes} /> + } + /> + + , + ); + expect(queryByTestId('sync-detail-drawer')).toBeInTheDocument(); + fireEvent.click(getByRole('button', { name: /close/i })); + expect(queryByTestId('sync-detail-drawer')).not.toBeInTheDocument(); + expect(screen.getByTestId('universes-page')).toBeInTheDocument(); + }); +}); diff --git a/client/src/pages/Universes.jsx b/client/src/pages/Universes.jsx index 63c6ace89..4313b5f69 100644 --- a/client/src/pages/Universes.jsx +++ b/client/src/pages/Universes.jsx @@ -9,14 +9,16 @@ */ import { useEffect, useMemo, useState } from 'react'; -import { Link } from 'react-router-dom'; +import { Link, useNavigate } from 'react-router-dom'; import { Plus, Globe, Trash2, Users, Workflow as WorkflowIcon } from 'lucide-react'; import toast from '../components/ui/Toast'; import ShareToButton from '../components/sharing/ShareToButton'; import SyncToPeerButton from '../components/sharing/SyncToPeerButton'; import OriginBadge from '../components/sharing/OriginBadge'; +import SyncBadge from '../components/sync/SyncBadge'; import { timeAgo } from '../utils/formatters'; import { listUniverses, deleteUniverse, listPipelineSeries } from '../services/api'; +import { useSyncIntegrity, syncBadgeStatus } from '../hooks/useSyncIntegrity'; // Named canon entities across all trunks — the "Canon" column reflects the // characters/places/objects the user has registered, not the looser variation @@ -44,10 +46,13 @@ function DeleteButton({ universe, armed, onDelete }) { } export default function Universes() { + const navigate = useNavigate(); const [universes, setUniverses] = useState([]); const [series, setSeries] = useState([]); const [loading, setLoading] = useState(true); + const sync = useSyncIntegrity('universe'); + useEffect(() => { // Guard setState against a navigate-away before the fetch resolves — // mirrors the editor's load effect (UniverseBuilder.jsx). @@ -158,15 +163,21 @@ export default function Universes() { {universes.map((u) => ( - -
- {u.name || '(untitled universe)'} - {u.origin ? : null} -
- {u.logline ? ( -
{u.logline}
- ) : null} - +
+ +
+ {u.name || '(untitled universe)'} + {u.origin ? : null} +
+ {u.logline ? ( +
{u.logline}
+ ) : null} + + navigate(`/universes/${encodeURIComponent(u.id)}/sync`)} + /> +
{canonCount(u)} {seriesCountByUniverse[u.id] || 0} @@ -208,6 +219,10 @@ export default function Universes() {
+ navigate(`/universes/${encodeURIComponent(u.id)}/sync`)} + />
))} diff --git a/client/src/services/apiPeerSync.js b/client/src/services/apiPeerSync.js index cbda541c1..75fe66b0e 100644 --- a/client/src/services/apiPeerSync.js +++ b/client/src/services/apiPeerSync.js @@ -16,7 +16,7 @@ import { request } from './apiCore.js'; -export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series']); +export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series', 'mediaCollection']); export const listPeerSubscriptions = (filter = {}, options) => { const qs = new URLSearchParams(); @@ -53,3 +53,72 @@ export const sweepTombstonesNow = ({ graceMs } = {}, options) => body: JSON.stringify(graceMs !== undefined ? { graceMs } : {}), ...options, }); + +// --------------------------------------------------------------------------- +// Integrity checking + manual sync (Group 4 — federated media sync integrity) +// --------------------------------------------------------------------------- + +/** + * Fetch integrity diff for a single kind against a specific peer. + * Uses `silent: true` because the hook caller owns the failure UI (it just + * marks the peer as unavailable rather than toasting on every poll tick). + */ +export const fetchSyncIntegrity = (peerId, kind) => + request( + `/peer-sync/integrity?peerId=${encodeURIComponent(peerId)}&kind=${encodeURIComponent(kind)}`, + { silent: true }, + ); + +/** + * Trigger a one-record sync push to a specific peer. + * Accepts an optional `options` spread so callers that own their error UI can + * pass `{ silent: true }`; defaults to letting the helper toast on failure. + */ +export const syncRecordToPeer = (peerId, recordKind, recordId, options = {}) => + request('/peer-sync/sync-record', { + method: 'POST', + body: JSON.stringify({ peerId, recordKind, recordId }), + ...options, + }); + +/** + * Trigger a full sync-now for all subscribed records to a peer. + * Same silent-capable pattern as `syncRecordToPeer`. + */ +export const syncNowForPeer = (peerId, options = {}) => + request('/peer-sync/sync-now', { + method: 'POST', + body: JSON.stringify({ peerId }), + ...options, + }); + +// The server validates each /pull-metadata request at ≤5000 filenames +// (peerPullMetadataSchema). Chunk larger lists so big libraries don't hard-400. +const PULL_METADATA_BATCH = 5000; + +/** + * Request the server to pull metadata for a list of filenames from peers. + * Same silent-capable pattern. Transparently batches lists larger than the + * server's per-request cap and aggregates the `{ attempted, recovered }` counts, + * so callers can pass an unbounded filename list (e.g. a large Unsorted library). + */ +export const pullMissingMetadata = async (filenames, options = {}) => { + const list = Array.isArray(filenames) ? filenames : []; + const postChunk = (chunk) => + request('/peer-sync/pull-metadata', { + method: 'POST', + body: JSON.stringify({ filenames: chunk }), + ...options, + }); + + if (list.length <= PULL_METADATA_BATCH) return postChunk(list); + + let attempted = 0; + let recovered = 0; + for (let i = 0; i < list.length; i += PULL_METADATA_BATCH) { + const res = await postChunk(list.slice(i, i + PULL_METADATA_BATCH)); + attempted += res?.attempted ?? 0; + recovered += res?.recovered ?? 0; + } + return { attempted, recovered }; +}; diff --git a/client/src/services/apiPeerSync.test.js b/client/src/services/apiPeerSync.test.js new file mode 100644 index 000000000..06891ce09 --- /dev/null +++ b/client/src/services/apiPeerSync.test.js @@ -0,0 +1,122 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +// Mock apiCore before importing the module under test. +vi.mock('./apiCore.js', () => ({ + request: vi.fn(), +})); + +let request; +let fetchSyncIntegrity; +let syncRecordToPeer; +let syncNowForPeer; +let pullMissingMetadata; + +beforeEach(async () => { + vi.resetModules(); + ({ request } = await import('./apiCore.js')); + ({ + fetchSyncIntegrity, + syncRecordToPeer, + syncNowForPeer, + pullMissingMetadata, + } = await import('./apiPeerSync.js')); + request.mockReset(); + request.mockResolvedValue({ ok: true }); +}); + +describe('fetchSyncIntegrity', () => { + it('calls the correct GET path with encoded params', async () => { + await fetchSyncIntegrity('peer-a', 'universe'); + expect(request).toHaveBeenCalledWith( + '/peer-sync/integrity?peerId=peer-a&kind=universe', + { silent: true }, + ); + }); + + it('URL-encodes peerId and kind that contain special characters', async () => { + await fetchSyncIntegrity('peer a/b', 'media collection'); + expect(request).toHaveBeenCalledWith( + '/peer-sync/integrity?peerId=peer%20a%2Fb&kind=media%20collection', + { silent: true }, + ); + }); + + it('always passes silent:true so the hook owns the failure UI', async () => { + await fetchSyncIntegrity('x', 'series'); + const [, opts] = request.mock.calls[0]; + expect(opts.silent).toBe(true); + }); +}); + +describe('syncRecordToPeer', () => { + it('calls POST /peer-sync/sync-record with the correct body', async () => { + await syncRecordToPeer('peer-b', 'universe', 'rec-1'); + expect(request).toHaveBeenCalledWith('/peer-sync/sync-record', { + method: 'POST', + body: JSON.stringify({ peerId: 'peer-b', recordKind: 'universe', recordId: 'rec-1' }), + }); + }); + + it('spreads caller options (e.g. silent:true) into the request', async () => { + await syncRecordToPeer('p', 'series', 'r', { silent: true }); + const [, opts] = request.mock.calls[0]; + expect(opts.silent).toBe(true); + expect(opts.method).toBe('POST'); + }); +}); + +describe('syncNowForPeer', () => { + it('calls POST /peer-sync/sync-now with peerId in body', async () => { + await syncNowForPeer('peer-c'); + expect(request).toHaveBeenCalledWith('/peer-sync/sync-now', { + method: 'POST', + body: JSON.stringify({ peerId: 'peer-c' }), + }); + }); + + it('spreads caller options', async () => { + await syncNowForPeer('peer-c', { silent: true }); + const [, opts] = request.mock.calls[0]; + expect(opts.silent).toBe(true); + }); +}); + +describe('pullMissingMetadata', () => { + it('calls POST /peer-sync/pull-metadata with filenames array', async () => { + await pullMissingMetadata(['a.json', 'b.json']); + expect(request).toHaveBeenCalledWith('/peer-sync/pull-metadata', { + method: 'POST', + body: JSON.stringify({ filenames: ['a.json', 'b.json'] }), + }); + }); + + it('spreads caller options', async () => { + await pullMissingMetadata([], { silent: true }); + const [, opts] = request.mock.calls[0]; + expect(opts.silent).toBe(true); + }); + + it('sends a single request when at/under the 5000 cap', async () => { + await pullMissingMetadata(Array.from({ length: 5000 }, (_, i) => `f${i}.json`)); + expect(request).toHaveBeenCalledTimes(1); + }); + + it('chunks lists over 5000 into multiple requests and aggregates counts', async () => { + request.mockReset(); + request + .mockResolvedValueOnce({ attempted: 5000, recovered: 10 }) + .mockResolvedValueOnce({ attempted: 1, recovered: 1 }); + + const result = await pullMissingMetadata( + Array.from({ length: 5001 }, (_, i) => `f${i}.json`), + { silent: true }, + ); + + expect(request).toHaveBeenCalledTimes(2); + // Each chunk is ≤ 5000 (server's peerPullMetadataSchema cap). + expect(JSON.parse(request.mock.calls[0][1].body).filenames).toHaveLength(5000); + expect(JSON.parse(request.mock.calls[1][1].body).filenames).toHaveLength(1); + // Aggregated counts across batches. + expect(result).toEqual({ attempted: 5001, recovered: 11 }); + }); +}); diff --git a/docs/superpowers/plans/2026-05-23-federated-media-sync-integrity.md b/docs/superpowers/plans/2026-05-23-federated-media-sync-integrity.md new file mode 100644 index 000000000..c0d7811d7 --- /dev/null +++ b/docs/superpowers/plans/2026-05-23-federated-media-sync-integrity.md @@ -0,0 +1,1221 @@ +# Federated Media Sync Parity + Per-Category Sync Integrity — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make media collections federate as reliably as universes/series, ensure synced images carry their generation prompts, and surface per-category sync integrity (badge + detail drawer + manual sync) on existing pages. + +**Architecture:** Promote media collections to a first-class peer-sync record kind (`mediaCollection`) reusing the existing subscribe/push/tombstone pipeline; extend the asset-pull worker to also pull the `.metadata.json` sidecar; add a Tailnet-only peer manifest endpoint + a pure local-vs-peer integrity diff; render the diff as per-record badges + a deep-linkable drawer with manual-sync actions. + +**Tech Stack:** Node/Express, Zod validation, Vitest (server + client), React/Vite, Socket.IO. No new dependencies. + +**Spec:** `docs/superpowers/specs/2026-05-23-federated-media-sync-integrity-design.md` + +**Conventions (read before starting):** +- No `try/catch` in request-lifecycle code (errors bubble to middleware); the `.catch(() => …)` pattern is used at fire-and-forget boundaries throughout `peerSync.js` — match it. +- Single-line emoji-prefixed logs. +- Every new `server/lib` / `client/src/lib` / hook / `apiX.js` file gets a barrel re-export + README row (enforced by `server/lib/index.test.js`). +- Record-creating Vitest suites mock `mockNoPeers()` AND `mockNoPeerSync()` from `server/lib/mockPathsDataRoot.js`. +- Tests live beside source as `.test.js`. Run server tests with `cd server && npm test`, client with `cd client && npm test`. + +**Pre-existing groundwork to reuse (do NOT re-create):** +- `peerSyncPushBase.linkedCollection` already in the push schema (`server/lib/validation.js:1310-1323`). +- `sanitizeRecordForWire` already has a `case 'mediaCollection'` (`server/lib/syncWire.js`) — extend it, don't add a duplicate. +- `mergeMediaCollectionsFromSync` (`server/services/mediaCollections.js:738`) is the receiver merge — reuse it. +- `KIND_TO_CATEGORY` (`peerSync.js:264`) maps kind→sync-category. `peerHasCategory` gates pushes on `syncCategories[cat]`. +- Universe/series create paths auto-subscribe via `import('./sharing/peerSync.js').then(({ autoSubscribeRecordToAllPeers }) => …)` (`universeBuilder.js:940`, `pipeline/series.js:213`) — mirror this fire-and-forget dynamic-import pattern. + +--- + +## File Structure + +**Group 1 — Collections as first-class sync records (server)** +- Modify: `server/services/mediaCollections.js` — soft-delete shape, `listCollections({includeDeleted})`, soft-delete `deleteCollection`, merge respects `deleted`, `createCollection` auto-subscribe + own-kind event, new `collectCollectionAssetReferences`. +- Modify: `server/lib/syncWire.js` — `mediaCollection` wire case applies soft-delete tail. +- Modify: `server/services/sharing/peerSync.js` — add `'mediaCollection'` to `PEER_SUBSCRIBABLE_KINDS`, `KIND_TO_CATEGORY`, `buildPushPayload` branch, `applyIncomingPush` branch. +- Modify: `server/lib/validation.js` — add `mediaCollectionPushSchema` to the discriminated union. +- Modify: `server/lib/schemaVersions.js` — `mediaCollections: 1`. +- Modify: `server/services/syncOrchestrator.js` — map `mediaCollection` sub → skip `mediaCollections` snapshot category. +- Modify: `server/services/sharing/tombstoneGc.js` — `mediaCollections` sweep. +- Modify: `server/services/mediaCollections.js` (`pruneTombstonedCollections` export). +- Tests: `mediaCollections.test.js`, `peerSync.test.js`, `syncWire.test.js`, `tombstoneGc.test.js`. + +**Group 2 — Sidecar metadata sync (server)** +- Modify: `server/services/sharing/peerSync.js` — `buildAssetManifest` adds `sidecarSha256`; `diffAssetManifestAgainstLocal` re-pulls on sidecar mismatch; `doPullOneAsset` pulls the sidecar. +- Create: `server/services/sharing/sidecarSync.js` — `pullSidecarForImage(peer, base, filename)`, `backfillMissingSidecars(filenames)` (pure-ish helpers, file-IO). +- Modify: `server/services/sharing/index.js` barrel + README. +- Tests: `peerSync.test.js`, `sidecarSync.test.js`. + +**Group 3 — Integrity API + peer manifest (server)** +- Create: `server/lib/syncIntegrity.js` — pure `computeRecordIntegrity(localList, remoteList, assetState)` + status constants. +- Create: `server/services/sharing/integrity.js` — orchestration: build local manifest, fetch peer manifest, run the pure diff. +- Modify: `server/routes/peerSync.js` — `GET /manifest`, `GET /integrity`. +- Modify: `server/lib/validation.js` — query schemas if needed. +- Modify: `server/lib/index.js` barrel + README. +- Tests: `syncIntegrity.test.js`, `peerSync.routes` (new `integrity` route test). + +**Group 4 — Manual sync + client UI** +- Modify: `server/services/sharing/peerSync.js` — `forcePushRecord(peerId, recordKind, recordId)` (bypass unchanged-hash), `syncNowForPeer(peerId)`. +- Modify: `server/routes/peerSync.js` — `POST /sync-record`, `POST /sync-now`, `POST /pull-metadata`. +- Modify: `server/lib/validation.js` — schemas for the three POSTs. +- Create: `client/src/services/apiPeerSync.js` (or extend existing) — wrappers. +- Create: `client/src/components/sync/SyncBadge.jsx`, `client/src/components/sync/SyncDetailDrawer.jsx`. +- Create: `client/src/hooks/useSyncIntegrity.js`. +- Modify: `client/src/pages/MediaCollections.jsx`, `Universes`/`Pipeline` list pages, `App.jsx` routes for the `/.../:id/sync` drawer. +- Tests: client Vitest for `SyncBadge`, `useSyncIntegrity`, `computeRecordIntegrity` already covered server-side (mirror the pure fn if duplicated client-side). + +--- + +# GROUP 1 — Collections as first-class sync records + +### Task 1.1: Soft-delete fields on the collection shape + +**Files:** +- Modify: `server/services/mediaCollections.js:86-128` (`sanitizeCollection`) +- Test: `server/services/mediaCollections.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +import { describe, it, expect } from 'vitest'; +// sanitizeCollection is module-private; test via the public round-trip: +// createCollection → deleteCollection → listCollections({includeDeleted}). +// For the pure-shape assertion, export sanitizeCollection for tests OR assert +// through mergeMediaCollectionsFromSync (preferred — it calls sanitizeCollection). +import { mergeMediaCollectionsFromSync, listCollections } from './mediaCollections.js'; +import { mockNoPeers } from '../lib/mockPathsDataRoot.js'; + +describe('collection soft-delete shape', () => { + it('preserves deleted + deletedAt through sync merge', async () => { + mockNoPeers(); + await mergeMediaCollectionsFromSync([{ + id: 'c1', name: 'C1', items: [], + deleted: true, deletedAt: '2026-05-23T00:00:00.000Z', + updatedAt: '2026-05-23T00:00:00.000Z', + }]); + const live = await listCollections(); + expect(live.find((c) => c.id === 'c1')).toBeUndefined(); + const all = await listCollections({ includeDeleted: true }); + const c = all.find((x) => x.id === 'c1'); + expect(c?.deleted).toBe(true); + expect(c?.deletedAt).toBe('2026-05-23T00:00:00.000Z'); + }); +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server && npm test -- mediaCollections.test.js -t "soft-delete shape"` +Expected: FAIL — `deleted` is stripped by `sanitizeCollection` and `listCollections` has no `includeDeleted`. + +- [ ] **Step 3: Implement — preserve soft-delete in `sanitizeCollection`** + +In `sanitizeCollection` (before the `return { … }` at line 128), compute: + +```js +const deleted = raw.deleted === true; +const deletedAt = deleted && typeof raw.deletedAt === 'string' ? raw.deletedAt : (deleted ? createdAt : null); +``` + +and add `deleted, deletedAt` to the returned object (tail position, after `updatedAt`): + +```js +return { id: raw.id, name, description, coverKey, universeId, seriesId, items, createdAt, updatedAt, deleted, deletedAt }; +``` + +- [ ] **Step 4: (Task 1.2 makes this test pass — `includeDeleted` filter needed.) Run after 1.2.** + +--- + +### Task 1.2: `listCollections({ includeDeleted })` filter + +**Files:** +- Modify: `server/services/mediaCollections.js:131-144` (`listCollections`) + +- [ ] **Step 1: Implement the filter** + +```js +export async function listCollections({ includeDeleted = false } = {}) { + await ensureDir(PATHS.data); + const raw = await readJSONFile(statePath(), DEFAULT_STATE, { logError: false }); + if (!Array.isArray(raw.collections)) return []; + const seen = new Set(); + const out = []; + for (const c of raw.collections) { + const s = sanitizeCollection(c); + if (!s || seen.has(s.id)) continue; + seen.add(s.id); + if (!includeDeleted && s.deleted === true) continue; + out.push(s); + } + return out; +} +``` + +**IMPORTANT:** `mergeMediaCollectionsFromSync` and `deleteCollection` internally call `listCollections()` to load *all* records for read-modify-write. Audit every internal caller (`grep -n "listCollections(" server/services/mediaCollections.js`) and pass `{ includeDeleted: true }` to the **mutation** read paths (merge, delete, addItem, updateCollection, bulkUpdate) so a tombstone isn't silently dropped from the rewrite (which would resurrect it). Read-only/display callers keep the default (live-only). + +- [ ] **Step 2: Run the Task 1.1 test** + +Run: `cd server && npm test -- mediaCollections.test.js -t "soft-delete shape"` +Expected: PASS + +- [ ] **Step 3: Commit** + +```bash +git add server/services/mediaCollections.js server/services/mediaCollections.test.js +git commit -m "feat(sync): soft-delete shape + includeDeleted on media collections" +``` + +--- + +### Task 1.3: `deleteCollection` soft-deletes + emits `mediaCollection` events + +**Files:** +- Modify: `server/services/mediaCollections.js:534-550` (`deleteCollection`) +- Test: `server/services/mediaCollections.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +import { createCollection, deleteCollection, listCollections } from './mediaCollections.js'; +import { recordEvents } from './sharing/recordEvents.js'; + +it('deleteCollection soft-deletes and emits mediaCollection deleted event', async () => { + mockNoPeers(); + const c = await createCollection({ name: 'ToDelete' }); + const events = []; + const handler = (e) => events.push(e); + recordEvents.on('deleted', handler); + await deleteCollection(c.id); + recordEvents.off('deleted', handler); + expect((await listCollections()).find((x) => x.id === c.id)).toBeUndefined(); + const all = await listCollections({ includeDeleted: true }); + expect(all.find((x) => x.id === c.id)?.deleted).toBe(true); + expect(events).toContainEqual({ recordKind: 'mediaCollection', recordId: c.id }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd server && npm test -- mediaCollections.test.js -t "soft-deletes and emits"` +Expected: FAIL — record is spliced out (hard delete), no `mediaCollection` event. + +- [ ] **Step 3: Implement soft-delete** + +Replace the body of `deleteCollection`: + +```js +import { emitRecordUpdated, emitRecordDeleted } from './sharing/recordEvents.js'; // emitRecordDeleted is new to the import + +export async function deleteCollection(id) { + const { universeId: deletedUniverseId, seriesId: deletedSeriesId } = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + const idx = all.findIndex((c) => c.id === id); + if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); + const target = all[idx]; + const now = new Date().toISOString(); + // Soft-delete: keep the row, drop its items + parent links so a tombstone + // push ships no asset cargo and the bucket can't keep re-publishing. + const next = [...all]; + next[idx] = { ...target, deleted: true, deletedAt: now, updatedAt: now, items: [], universeId: null, seriesId: null }; + await writeAll(next); + return { universeId: target.universeId || null, seriesId: target.seriesId || null }; + }); + // Parent universe/series re-export (membership changed) + own-kind delete + // so the peer-sync delete listener pushes the tombstone. + if (deletedUniverseId) emitRecordUpdated('universe', deletedUniverseId); + if (deletedSeriesId) emitRecordUpdated('series', deletedSeriesId); + emitRecordDeleted('mediaCollection', id); + return { id }; +} +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `cd server && npm test -- mediaCollections.test.js -t "soft-deletes and emits"` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add server/services/mediaCollections.js server/services/mediaCollections.test.js +git commit -m "feat(sync): media collection deleteCollection soft-deletes + emits tombstone event" +``` + +--- + +### Task 1.4: Merge respects `deleted` (LWW tombstone) + +**Files:** +- Modify: `server/services/mediaCollections.js:738-792` (`mergeMediaCollectionsFromSync`) + `collectionsEqual` +- Test: `server/services/mediaCollections.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +it('a newer remote tombstone deletes a local live collection', async () => { + mockNoPeers(); + const c = await createCollection({ name: 'Live' }); // updatedAt = t0 + await new Promise((r) => setTimeout(r, 5)); + await mergeMediaCollectionsFromSync([{ + id: c.id, name: 'Live', items: [], deleted: true, + deletedAt: '2999-01-01T00:00:00.000Z', updatedAt: '2999-01-01T00:00:00.000Z', + }]); + expect((await listCollections()).find((x) => x.id === c.id)).toBeUndefined(); + const all = await listCollections({ includeDeleted: true }); + expect(all.find((x) => x.id === c.id)?.deleted).toBe(true); +}); + +it('an older remote tombstone does NOT delete a newer local collection', async () => { + mockNoPeers(); + const c = await createCollection({ name: 'Fresh' }); // updatedAt = now (newer) + await mergeMediaCollectionsFromSync([{ + id: c.id, name: 'Fresh', items: [], deleted: true, + deletedAt: '2000-01-01T00:00:00.000Z', updatedAt: '2000-01-01T00:00:00.000Z', + }]); + expect((await listCollections()).find((x) => x.id === c.id)).toBeDefined(); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd server && npm test -- mediaCollections.test.js -t "tombstone"` +Expected: FAIL — `next` object never carries `deleted`/`deletedAt`, so the tombstone is dropped. + +- [ ] **Step 3: Implement — carry soft-delete through the scalar-LWW branch** + +In `mergeMediaCollectionsFromSync`, the `next` object (line 774-783) takes scalars from `scalarSource` (the newer side). Add the soft-delete pair from the scalar source and, when the winning side is deleted, blank items: + +```js +const scalarDeleted = scalarSource.deleted === true; +const next = { + ...local, + name: scalarSource.name, + description: scalarSource.description, + coverKey: scalarDeleted ? null : coverKey, + universeId: scalarDeleted ? null : scalarSource.universeId, + seriesId: scalarDeleted ? null : scalarSource.seriesId, + items: scalarDeleted ? [] : mergedItems, + updatedAt: remoteWins ? remoteTs : localTs, + deleted: scalarDeleted, + deletedAt: scalarDeleted ? (scalarSource.deletedAt || (remoteWins ? remoteTs : localTs)) : null, +}; +``` + +Also handle the **local-missing** branch (line 748-751): a remote tombstone for a collection we've never seen must still be recorded (so it can't resurrect). `sanitizeCollection` already preserves `deleted`, so `localById.set(sanitized.id, sanitized)` is correct as-is — confirm the test for "remote tombstone, no local" passes (add one if missing). + +`collectionsEqual` uses `JSON.stringify` — since both sides go through `sanitizeCollection` (canonical key order incl. the new tail fields), no change needed. + +- [ ] **Step 4: Run to verify passes** + +Run: `cd server && npm test -- mediaCollections.test.js -t "tombstone"` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add server/services/mediaCollections.js server/services/mediaCollections.test.js +git commit -m "feat(sync): LWW merge honors media collection tombstones" +``` + +--- + +### Task 1.5: `createCollection` auto-subscribes + emits own-kind event + +**Files:** +- Modify: `server/services/mediaCollections.js:163-189` (`createCollection`) +- Test: `server/services/mediaCollections.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +it('createCollection emits a mediaCollection updated event', async () => { + mockNoPeers(); + const events = []; + const handler = (e) => events.push(e); + recordEvents.on('updated', handler); + const c = await createCollection({ name: 'New' }); + recordEvents.off('updated', handler); + expect(events).toContainEqual({ recordKind: 'mediaCollection', recordId: c.id }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd server && npm test -- mediaCollections.test.js -t "createCollection emits"` +Expected: FAIL — `createCollection` emits nothing today. + +- [ ] **Step 3: Implement — emit + auto-subscribe (mirror universe/series)** + +After the `serializeFileWrite` returns `next` in `createCollection`, before returning, add: + +```js + const created = await serializeFileWrite(async () => { /* …existing… */ }); + emitRecordUpdated('mediaCollection', created.id); + // Auto-subscribe to peers with mediaCollections enabled. Dynamic import + + // fire-and-forget mirrors universeBuilder.js:940 / pipeline/series.js:213 + // so the heavy peerSync graph stays off mediaCollections' module-load path. + import('./sharing/peerSync.js') + .then(({ autoSubscribeRecordToAllPeers }) => autoSubscribeRecordToAllPeers('mediaCollection', created.id)) + .catch(() => {}); + return created; +``` + +(Refactor the existing inline `return next` so the function returns `created` after the side effects.) + +- [ ] **Step 4: Run to verify passes** + +Run: `cd server && npm test -- mediaCollections.test.js -t "createCollection emits"` +Expected: PASS. Confirm the existing `mockNoPeers()` suites still pass (the mock stubs `autoSubscribeRecordToAllPeers`). + +- [ ] **Step 5: Commit** + +```bash +git add server/services/mediaCollections.js server/services/mediaCollections.test.js +git commit -m "feat(sync): createCollection emits mediaCollection event + auto-subscribes peers" +``` + +--- + +### Task 1.6: Register `mediaCollection` as a subscribable kind + category map + +**Files:** +- Modify: `server/services/sharing/peerSync.js:73` (`PEER_SUBSCRIBABLE_KINDS`), `:264-267` (`KIND_TO_CATEGORY`) +- Modify: `server/lib/schemaVersions.js:38` +- Test: `server/services/sharing/peerSync.test.js` + +- [ ] **Step 1: Edit constants** + +```js +// peerSync.js:73 +export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series', 'mediaCollection']); + +// peerSync.js:264 +const KIND_TO_CATEGORY = Object.freeze({ + universe: 'universe', + series: 'pipeline', + mediaCollection: 'mediaCollections', +}); +``` + +```js +// schemaVersions.js — replace the "future:" comment line + pipelineSeries: 1, + mediaCollections: 1, +}); +``` + +- [ ] **Step 2: Write/extend a guard test** + +```js +it('mediaCollection is a subscribable kind mapped to the mediaCollections category', () => { + expect(PEER_SUBSCRIBABLE_KINDS).toContain('mediaCollection'); +}); +``` + +Run: `cd server && npm test -- peerSync.test.js -t "subscribable kind"` → PASS. + +- [ ] **Step 3: Commit** + +```bash +git add server/services/sharing/peerSync.js server/lib/schemaVersions.js server/services/sharing/peerSync.test.js +git commit -m "feat(sync): register mediaCollection as a peer-subscribable kind (schemaVersion 1)" +``` + +--- + +### Task 1.7: Collection asset-reference collector + push payload branch + +**Files:** +- Modify: `server/services/sharing/peerSync.js` — add `collectCollectionAssetReferences`, `buildCollectionAssetManifest`, and a `buildPushPayload` branch (`:850-941`) +- Test: `server/services/sharing/peerSync.test.js` + +- [ ] **Step 1: Write the failing test (asset references)** + +```js +it('collectCollectionAssetReferences maps items to image/video refs', () => { + const refs = collectCollectionAssetReferences({ + items: [ + { kind: 'image', ref: 'a.png' }, + { kind: 'video', ref: 'vid123' }, + { kind: 'image', ref: 'b.png' }, + ], + }); + expect(refs.directImageFilenames).toEqual(['a.png', 'b.png']); + expect(refs.directVideoFilenames).toEqual(['vid123']); +}); +``` + +- [ ] **Step 2: Run to verify fails** + +Run: `cd server && npm test -- peerSync.test.js -t "collectCollectionAssetReferences"` +Expected: FAIL — function not exported. + +- [ ] **Step 3: Implement** + +```js +// Collections store items as { kind: 'image'|'video', ref, addedAt }. Map to +// the same shape buildAssetManifest's hashers consume. Videos in collections +// reference a job/file id (no image-ref kind in collections). +export function collectCollectionAssetReferences(collection) { + const items = Array.isArray(collection?.items) ? collection.items : []; + const directImageFilenames = []; + const directVideoFilenames = []; + for (const it of items) { + if (it?.kind === 'image' && typeof it.ref === 'string') directImageFilenames.push(it.ref); + else if (it?.kind === 'video' && typeof it.ref === 'string') directVideoFilenames.push(it.ref); + } + return { directImageFilenames, directImageRefFilenames: [], directVideoFilenames }; +} + +async function buildCollectionAssetManifest(collection) { + const refs = collectCollectionAssetReferences(collection); + const out = []; + for (const filename of refs.directImageFilenames) { + const entry = await hashImageForManifest(filename); + if (entry) out.push(entry); + } + for (const filename of refs.directVideoFilenames) { + const entry = await hashSimpleAsset(filename, 'video', PATHS.videos); + if (entry) out.push(entry); + } + return out; +} +``` + +Add the `buildPushPayload` branch (after the `series` branch, before `return null`): + +```js + if (sub.recordKind === 'mediaCollection') { + const record = await getCollection(sub.recordId, { includeDeleted: true }).catch(() => null); + if (!record) return null; + const sanitized = sanitizeRecordForWire('mediaCollection', record); + if (!sanitized) return null; + const assetManifest = record.deleted === true ? [] : await buildCollectionAssetManifest(record); + return { kind: 'mediaCollection', record: sanitized, assetManifest, sourceInstanceId, portosMeta }; + } +``` + +**Note:** `getCollection` (`mediaCollections.js:146`) currently throws on not-found and has no `includeDeleted`. Add an `{ includeDeleted }` option that reads `listCollections({ includeDeleted })` and returns `null`-friendly via the caller's `.catch`. Import `getCollection` into `peerSync.js`. + +- [ ] **Step 4: Run to verify passes** + +Run: `cd server && npm test -- peerSync.test.js -t "collectCollectionAssetReferences"` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add server/services/sharing/peerSync.js server/services/mediaCollections.js server/services/sharing/peerSync.test.js +git commit -m "feat(sync): build push payload + asset manifest for mediaCollection records" +``` + +--- + +### Task 1.8: Receiver applies `mediaCollection` pushes + +**Files:** +- Modify: `server/services/sharing/peerSync.js:1126-1159` (`applyIncomingPush`) +- Modify: `server/lib/syncWire.js` (`mediaCollection` case → soft-delete tail) +- Modify: `server/lib/validation.js:1324-1336` (push schema union) +- Test: `server/services/sharing/peerSync.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +it('applies an incoming mediaCollection push into local collections', async () => { + mockNoPeers(); + await applyIncomingPush({ + kind: 'mediaCollection', + record: { id: 'col-x', name: 'Synced', items: [], updatedAt: '2026-05-23T00:00:00.000Z' }, + assetManifest: [], + sourceInstanceId: 'peer-abc', + }); + const all = await listCollections(); + expect(all.find((c) => c.id === 'col-x')?.name).toBe('Synced'); +}); +``` + +- [ ] **Step 2: Run to verify fails** + +Run: `cd server && npm test -- peerSync.test.js -t "incoming mediaCollection"` +Expected: FAIL — push schema rejects `kind: 'mediaCollection'` (not in the union) and `applyIncomingPush` has no branch. + +- [ ] **Step 3a: Add the push schema variant** (`validation.js`, after `seriesPushSchema`): + +```js +const mediaCollectionPushSchema = z.object({ + kind: z.literal('mediaCollection'), + ...peerSyncPushBase, +}).strict(); +export const peerSyncPushSchema = z.discriminatedUnion('kind', [ + universePushSchema, + seriesPushSchema, + mediaCollectionPushSchema, +]); +``` + +- [ ] **Step 3b: Add the apply branch** in `applyIncomingPush`, after the `series` block (~line 1159): + +```js + } else if (kind === 'mediaCollection') { + await mergeMediaCollectionsFromSync([record]); + } +``` + +`mergeMediaCollectionsFromSync` is already imported. The existing asset-diff/pull tail (`diffAssetManifestAgainstLocal` + `pullMissingAssetsFromPeer`) and `maybeCreateReverseSubscription` run unchanged for the new kind. The `localEphemeral` block (universe/series only) leaves `localEphemeral=false` for collections — correct, collections have no ephemeral flag. + +- [ ] **Step 3c: Tail the soft-delete in the wire sanitizer** (`syncWire.js`, the `case 'mediaCollection'`): + +```js + case 'mediaCollection': { + // Strip + re-add soft-delete at tail for byte-stable checksums, mirroring + // the universe/series case. Collections have no `ephemeral` flag. + const { deleted: _d, deletedAt: _da, ...rest } = record; + return { ...rest, ...sanitizeSoftDeleteFields(record) }; + } +``` + +- [ ] **Step 4: Run to verify passes** + +Run: `cd server && npm test -- peerSync.test.js -t "incoming mediaCollection"` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add server/services/sharing/peerSync.js server/lib/syncWire.js server/lib/validation.js server/services/sharing/peerSync.test.js +git commit -m "feat(sync): receiver applies mediaCollection pushes + push schema variant" +``` + +--- + +### Task 1.9: Orchestrator skips snapshot for peer-subbed collections + +**Files:** +- Modify: `server/services/syncOrchestrator.js:356-372` (`categoriesCoveredByPeerSync`) +- Test: `server/services/syncOrchestrator.test.js` + +- [ ] **Step 1: Implement the mapping** + +In `categoriesCoveredByPeerSync`, add to the loop: + +```js + if (sub.recordKind === 'mediaCollection') skip.add('mediaCollections'); +``` + +- [ ] **Step 2: Test** + +```js +it('a mediaCollection subscription skips the mediaCollections snapshot category', async () => { + // mock listPeerSubscriptions to return one mediaCollection sub for the peer + // (follow the existing mocking pattern in this suite), then assert the + // returned skip set contains 'mediaCollections'. +}); +``` + +Run: `cd server && npm test -- syncOrchestrator.test.js -t "mediaCollection subscription skips"` +Expected: PASS + +- [ ] **Step 3: Commit** + +```bash +git add server/services/syncOrchestrator.js server/services/syncOrchestrator.test.js +git commit -m "feat(sync): orchestrator skips snapshot for peer-subbed collections" +``` + +--- + +### Task 1.10: Tombstone GC for collections + +**Files:** +- Modify: `server/services/mediaCollections.js` — add `pruneTombstonedCollections({ olderThanMs })` +- Modify: `server/services/sharing/tombstoneGc.js` — sweep collections +- Test: `server/services/mediaCollections.test.js`, `server/services/sharing/tombstoneGc.test.js` + +- [ ] **Step 1: Write the failing test for the prune helper** + +```js +it('pruneTombstonedCollections removes tombstones older than the cutoff', async () => { + mockNoPeers(); + await mergeMediaCollectionsFromSync([{ + id: 'old', name: 'Old', items: [], deleted: true, + deletedAt: '2000-01-01T00:00:00.000Z', updatedAt: '2000-01-01T00:00:00.000Z', + }]); + const pruned = await pruneTombstonedCollections({ olderThanMs: Date.now() }); + expect(pruned).toBe(1); + expect((await listCollections({ includeDeleted: true })).find((c) => c.id === 'old')).toBeUndefined(); +}); +``` + +- [ ] **Step 2: Run to verify fails** + +Run: `cd server && npm test -- mediaCollections.test.js -t "pruneTombstonedCollections"` +Expected: FAIL — not exported. + +- [ ] **Step 3: Implement the prune helper** (`mediaCollections.js`): + +```js +// Hard-remove tombstoned collections whose deletedAt is older than the cutoff. +// Called by tombstoneGc once every subscribed peer has acked the deletion. +export async function pruneTombstonedCollections({ olderThanMs = 0 } = {}) { + return serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + const keep = all.filter((c) => { + if (c.deleted !== true) return true; + const ms = Date.parse(c.deletedAt || ''); + return !(Number.isFinite(ms) && ms <= olderThanMs); + }); + if (keep.length === all.length) return 0; + await writeAll(keep); + return all.length - keep.length; + }); +} +``` + +- [ ] **Step 4: Wire into `tombstoneGc.js`** — mirror the universe/series sweep. Add `import { pruneTombstonedCollections } from '../mediaCollections.js';`, add `mediaCollection` to `snapshotCategoryForKind` (returns `'mediaCollections'`), and add a sweep call alongside the others, gated by the same `getMinAckAcrossPeers` + grace logic. Return a `collections` count in the sweep result; surface it in the `syncOrchestrator.runTombstoneSweep` log line. + +- [ ] **Step 5: Run both test files** + +Run: `cd server && npm test -- mediaCollections.test.js tombstoneGc.test.js` +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add server/services/mediaCollections.js server/services/sharing/tombstoneGc.js server/services/syncOrchestrator.js server/services/sharing/*.test.js server/services/mediaCollections.test.js +git commit -m "feat(sync): tombstone GC prunes acked media collection deletions" +``` + +--- + +### Task 1.11: Full Group-1 regression + +- [ ] Run: `cd server && npm test -- mediaCollections peerSync syncOrchestrator tombstoneGc syncWire` +- [ ] Run the barrel/README guard: `cd server && npm test -- index.test.js` (no new public lib files in Group 1, but `getCollection` signature changed — confirm no other caller breaks: `grep -rn "getCollection(" server/ | grep -v node_modules`). +- [ ] Expected: all PASS. + +--- + +# GROUP 2 — Sidecar metadata sync + +### Task 2.1: Manifest carries `sidecarSha256` for images + +**Files:** +- Modify: `server/services/sharing/peerSync.js:481-493` (`hashImageForManifest`) +- Modify: `server/lib/validation.js` (`peerAssetManifestEntrySchema` — allow optional `sidecarSha256`) +- Test: `server/services/sharing/peerSync.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +it('image manifest entry includes sidecarSha256 when a sidecar exists', async () => { + // Arrange: write a fake image + its .metadata.json sidecar under PATHS.images + // (use the test data-root from mockPathsDataRoot). Then: + const entry = await hashImageForManifest('test.png'); + expect(entry.sidecarSha256).toBeTypeOf('string'); +}); +``` + +- [ ] **Step 2: Run to verify fails** → FAIL (`sidecarSha256` undefined). + +- [ ] **Step 3: Implement** — in `hashImageForManifest`, after computing the image hash: + +```js +import { imageSidecarName } from './buckets.js'; // add to imports +// …inside hashImageForManifest, before return: +const sidecarPath = join(PATHS.images, imageSidecarName(safeName)); +const sidecarSha256 = existsSync(sidecarPath) ? await sha256File(sidecarPath).catch(() => null) : null; +return { filename: safeName, kind: 'image', sha256: result.hash, ...(sidecarSha256 ? { sidecarSha256 } : {}) }; +``` + +Add `sidecarSha256: z.string().min(1).max(128).optional()` to `peerAssetManifestEntrySchema` in `validation.js`. + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): image manifest carries sidecar hash` + +--- + +### Task 2.2: Pull the sidecar when pulling an image + +**Files:** +- Create: `server/services/sharing/sidecarSync.js` +- Modify: `server/services/sharing/peerSync.js:1427-1495` (`doPullOneAsset`) — call the sidecar pull after a successful image write +- Modify: `server/services/sharing/index.js` barrel + `README.md` +- Test: `server/services/sharing/sidecarSync.test.js` + +- [ ] **Step 1: Write the failing test** for `pullSidecarForImage` — mock `peerFetch` to return a JSON body, assert the file lands at `PATHS.images/.metadata.json`, and that a 404 is swallowed (no throw, no file). + +- [ ] **Step 2: Run** → FAIL (module missing). + +- [ ] **Step 3: Implement `sidecarSync.js`** + +```js +import { join } from 'path'; +import { atomicWrite, ensureDir, PATHS } from '../../lib/fileUtils.js'; +import { imageSidecarName } from './buckets.js'; +import { peerFetch } from '../../lib/httpClient.js'; // match doPullOneAsset's import +import { peerBaseUrl } from '../../lib/peerUrl.js'; + +const SIDECAR_MAX_BYTES = 256 * 1024; // gen-params JSON is tiny; cap defensively + +// Pull `.metadata.json` from a peer's /data/images mount and +// write it alongside the image. Best-effort: a 404 (no sidecar on the sender) +// is normal and silently ignored. Filename is sanitized by the caller. +export async function pullSidecarForImage(peer, base, imageFilename) { + const sidecarName = imageSidecarName(imageFilename); + const url = `${base}/data/images/${encodeURIComponent(sidecarName)}`; + const res = await peerFetch(url, { maxBytes: SIDECAR_MAX_BYTES }).catch(() => null); + if (!res || !res.ok) return false; + const buf = Buffer.from(await res.arrayBuffer()); + if (buf.length === 0 || buf.length > SIDECAR_MAX_BYTES) return false; + await ensureDir(PATHS.images); + await atomicWrite(join(PATHS.images, sidecarName), buf); + console.log(`📥 peerSync: pulled sidecar ${sidecarName} from ${peer.name || peer.instanceId}`); + return true; +} +``` + +In `doPullOneAsset`, after the successful image `atomicWrite` + `asset-arrived` emit, for `entry.kind === 'image'`: + +```js + if (entry.kind === 'image') { + await pullSidecarForImage(peer, base, safeName).catch(() => {}); + } +``` + +Add `pullSidecarForImage` (and Task 2.3's `backfillMissingSidecars`) to `server/services/sharing/index.js` barrel + a README row. + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): pull image sidecar metadata alongside image bytes` + +--- + +### Task 2.3: Manual backfill for bare Unsorted images + +**Files:** +- Modify: `server/services/sharing/sidecarSync.js` — `backfillMissingSidecars({ filenames })` +- Test: `server/services/sharing/sidecarSync.test.js` + +- [ ] **Step 1: Write the failing test** — given two local images (one with a sidecar, one without), `backfillMissingSidecars` only attempts the bare one against online peers (mock `getPeers` to return one online peer + mock `peerFetch`). + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** + +```js +import { existsSync } from 'fs'; +import { getPeers } from '../instances.js'; +import { peerBaseUrl } from '../../lib/peerUrl.js'; + +// For each local image filename lacking a sidecar, try each online peer until +// one yields the sidecar. Returns { attempted, recovered }. +export async function backfillMissingSidecars({ filenames }) { + const peers = (await getPeers().catch(() => [])).filter((p) => p?.status === 'online' && p.instanceId); + let attempted = 0, recovered = 0; + for (const filename of Array.isArray(filenames) ? filenames : []) { + const sidecarPath = join(PATHS.images, imageSidecarName(filename)); + if (existsSync(sidecarPath)) continue; + attempted++; + for (const peer of peers) { + const ok = await pullSidecarForImage(peer, peerBaseUrl(peer), filename).catch(() => false); + if (ok) { recovered++; break; } + } + } + console.log(`🔄 sidecar backfill: ${recovered}/${attempted} recovered from ${peers.length} peer(s)`); + return { attempted, recovered }; +} +``` + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): backfill missing image sidecars from online peers` + +--- + +# GROUP 3 — Integrity API + peer manifest + +### Task 3.1: Pure integrity diff + +**Files:** +- Create: `server/lib/syncIntegrity.js` +- Modify: `server/lib/index.js` barrel + `README.md` +- Test: `server/lib/syncIntegrity.test.js` + +- [ ] **Step 1: Write the failing test** + +```js +import { describe, it, expect } from 'vitest'; +import { computeRecordIntegrity, INTEGRITY_STATUS } from './syncIntegrity.js'; + +describe('computeRecordIntegrity', () => { + const local = [{ id: 'a', updatedAt: '2026-01-02', assetHashes: ['h1'] }]; + it('flags peer-only records', () => { + const r = computeRecordIntegrity([], [{ id: 'a', updatedAt: '2026-01-02', assetHashes: ['h1'] }]); + expect(r.find((x) => x.id === 'a').status).toBe(INTEGRITY_STATUS.PEER_ONLY); + }); + it('flags local-only records (peer has no row, not a tombstone)', () => { + const r = computeRecordIntegrity(local, []); + expect(r.find((x) => x.id === 'a').status).toBe(INTEGRITY_STATUS.LOCAL_ONLY); + }); + it('flags diverged on updatedAt mismatch', () => { + const r = computeRecordIntegrity(local, [{ id: 'a', updatedAt: '2026-01-09', assetHashes: ['h1'] }]); + expect(r.find((x) => x.id === 'a').status).toBe(INTEGRITY_STATUS.DIVERGED); + }); + it('flags assets-missing when hashes differ but record matches', () => { + const r = computeRecordIntegrity(local, [{ id: 'a', updatedAt: '2026-01-02', assetHashes: ['h1', 'h2'] }]); + expect(r.find((x) => x.id === 'a').status).toBe(INTEGRITY_STATUS.ASSETS_MISSING); + }); + it('flags in-parity on full match', () => { + const r = computeRecordIntegrity(local, [{ id: 'a', updatedAt: '2026-01-02', assetHashes: ['h1'] }]); + expect(r.find((x) => x.id === 'a').status).toBe(INTEGRITY_STATUS.IN_PARITY); + }); +}); +``` + +- [ ] **Step 2: Run** → FAIL (module missing). + +- [ ] **Step 3: Implement `syncIntegrity.js`** + +```js +export const INTEGRITY_STATUS = Object.freeze({ + IN_PARITY: 'in-parity', + LOCAL_ONLY: 'local-only', + PEER_ONLY: 'peer-only', + DIVERGED: 'diverged', + ASSETS_MISSING: 'assets-missing', +}); + +const sortedHashes = (a) => [...(Array.isArray(a) ? a : [])].sort(); +const hashesEqual = (a, b) => { + const sa = sortedHashes(a), sb = sortedHashes(b); + return sa.length === sb.length && sa.every((h, i) => h === sb[i]); +}; + +// Pure diff of two manifest lists. Each entry: { id, name?, updatedAt, deleted?, assetHashes }. +// Tombstones (deleted) are excluded from "missing on the other side" so a deleted +// record doesn't read as local-only/peer-only. +export function computeRecordIntegrity(localList, remoteList) { + const byId = new Map(); + for (const l of localList || []) byId.set(l.id, { id: l.id, name: l.name, local: l, remote: null }); + for (const r of remoteList || []) { + const cur = byId.get(r.id) || { id: r.id, name: r.name, local: null, remote: null }; + cur.remote = r; + if (!cur.name) cur.name = r.name; + byId.set(r.id, cur); + } + const out = []; + for (const { id, name, local, remote } of byId.values()) { + const localLive = local && local.deleted !== true; + const remoteLive = remote && remote.deleted !== true; + let status; + if (localLive && !remoteLive) status = INTEGRITY_STATUS.LOCAL_ONLY; + else if (!localLive && remoteLive) status = INTEGRITY_STATUS.PEER_ONLY; + else if (!localLive && !remoteLive) continue; // both tombstoned/absent → nothing to show + else if (local.updatedAt !== remote.updatedAt) status = INTEGRITY_STATUS.DIVERGED; + else if (!hashesEqual(local.assetHashes, remote.assetHashes)) status = INTEGRITY_STATUS.ASSETS_MISSING; + else status = INTEGRITY_STATUS.IN_PARITY; + out.push({ id, name: name || id, status }); + } + return out; +} +``` + +Add `export * from './syncIntegrity.js';` (or named) to `server/lib/index.js` + a README row. + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): pure record-integrity diff` + +--- + +### Task 3.2: Local manifest builder + peer manifest fetch (orchestration) + +**Files:** +- Create: `server/services/sharing/integrity.js` +- Test: `server/services/sharing/integrity.test.js` + +- [ ] **Step 1: Write the failing test** — `buildLocalManifest('mediaCollection')` returns `[{ id, name, updatedAt, deleted, assetHashes }]` for local collections (seed two via `createCollection` + `addItem`, mock peers). Assert shape + that `assetHashes` reflects item refs. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement `integrity.js`** + +```js +import { computeRecordIntegrity } from '../../lib/syncIntegrity.js'; +import { listCollections } from '../mediaCollections.js'; +import { listUniverses } from '../universeBuilder.js'; +import { listSeries } from '../pipeline/series.js'; +import { findPeerById } from './peerSync.js'; // or instances.js — use the existing helper +import { peerBaseUrl } from '../../lib/peerUrl.js'; +import { peerFetch } from '../../lib/httpClient.js'; + +// One row per record: id, display name, updatedAt, deleted, and the set of +// asset hashes the record references (so the diff can detect byte divergence +// without shipping bytes). Asset hashes reuse buildAssetManifest's sha256s. +export async function buildLocalManifest(kind) { + if (kind === 'mediaCollection') { + const cols = await listCollections({ includeDeleted: true }); + return Promise.all(cols.map(async (c) => ({ + id: c.id, name: c.name, updatedAt: c.updatedAt, deleted: c.deleted === true, + assetHashes: await collectionAssetHashes(c), + }))); + } + // universe / series: reuse listUniverses/listSeries + buildAssetManifest hashes. + // (Implement parallel branches; for the first cut, mediaCollection is the priority.) +} + +export async function getPeerIntegrity({ peerId, kind }) { + const peer = await findPeerById(peerId); + if (!peer) return { available: false, reason: 'peer-not-found', records: [] }; + const res = await peerFetch(`${peerBaseUrl(peer)}/api/peer-sync/manifest?kind=${encodeURIComponent(kind)}`).catch(() => null); + if (!res || res.status === 404) return { available: false, reason: 'peer-too-old', records: [] }; + if (!res.ok) return { available: false, reason: 'fetch-failed', records: [] }; + const body = await res.json().catch(() => null); + const remote = Array.isArray(body?.records) ? body.records : []; + const local = await buildLocalManifest(kind); + return { available: true, records: computeRecordIntegrity(local, remote) }; +} +``` + +`collectionAssetHashes(c)` reuses the Group-2 manifest hashers (export a small helper from peerSync or compute via `buildCollectionAssetManifest(c).then(m => m.map(e => e.sha256))`). + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): local manifest builder + peer integrity fetch` + +--- + +### Task 3.3: HTTP routes — `GET /manifest`, `GET /integrity` + +**Files:** +- Modify: `server/routes/peerSync.js` +- Test: `server/routes/peerSync.test.js` (create if absent — follow `palette.test.js` route-test style) + +- [ ] **Step 1: Write the failing test** — `GET /api/peer-sync/manifest?kind=mediaCollection` returns `{ records: [...] }`; `GET /api/peer-sync/integrity?peerId=…&kind=…` returns `{ available, records }`. Invalid `kind` → 400. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** in `routes/peerSync.js`: + +```js +import { buildLocalManifest, getPeerIntegrity } from '../services/sharing/integrity.js'; +import { PEER_SUBSCRIBABLE_KINDS } from '../services/sharing/peerSync.js'; + +const validKind = (k) => typeof k === 'string' && PEER_SUBSCRIBABLE_KINDS.includes(k); + +router.get('/manifest', asyncHandler(async (req, res) => { + if (!validKind(req.query.kind)) throw new ServerError('invalid kind', { status: 400, code: 'VALIDATION_ERROR' }); + res.json({ records: await buildLocalManifest(req.query.kind) }); +})); + +router.get('/integrity', asyncHandler(async (req, res) => { + if (typeof req.query.peerId !== 'string') throw new ServerError('peerId required', { status: 400, code: 'VALIDATION_ERROR' }); + if (!validKind(req.query.kind)) throw new ServerError('invalid kind', { status: 400, code: 'VALIDATION_ERROR' }); + res.json(await getPeerIntegrity({ peerId: req.query.peerId, kind: req.query.kind })); +})); +``` + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): peer manifest + integrity HTTP routes` + +--- + +# GROUP 4 — Manual sync + client UI + +### Task 4.1: Force-push + sync-now service functions + +**Files:** +- Modify: `server/services/sharing/peerSync.js` — `forcePushRecord`, `syncNowForPeer` +- Test: `server/services/sharing/peerSync.test.js` + +- [ ] **Step 1: Write the failing test** — `forcePushRecord` pushes even when `lastPushedHash` matches (mock `peerFetch`, assert a POST fired). `syncNowForPeer` subscribes-all + retries pending. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** + +```js +// Force a push regardless of the unchanged-hash short-circuit. Used by the +// manual "Sync to peers now" action. Resolves/creates the subscription first. +export async function forcePushRecord(peerId, recordKind, recordId) { + const sub = (await findPeerSubscription(peerId, recordKind, recordId)) + || await subscribePeer({ peerId, recordKind, recordId }); + return pushRecordToPeer({ ...sub, lastPushedHash: null }, { bypassSchemaCooldown: true }); +} + +// Backfill-subscribe + retry every pending push for one peer's enabled kinds. +export async function syncNowForPeer(peerId) { + const peer = await findPeerById(peerId); + if (!peer?.instanceId) return { ok: false }; + for (const kind of PEER_SUBSCRIBABLE_KINDS) { + if (peerHasCategory(peer, kind)) await autoSubscribePeerToAllRecords(peer.instanceId, kind).catch(() => {}); + } + await retryPendingPushesForPeer(peer.instanceId).catch(() => {}); + return { ok: true }; +} +``` + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): forcePushRecord + syncNowForPeer service fns` + +--- + +### Task 4.2: Manual-sync HTTP routes + +**Files:** +- Modify: `server/routes/peerSync.js`, `server/lib/validation.js` +- Test: `server/routes/peerSync.test.js` + +- [ ] **Step 1: Write the failing test** — `POST /api/peer-sync/sync-record` with `{peerId, recordKind, recordId}` returns `{pushed}`; `POST /sync-now {peerId}` returns `{ok}`; `POST /pull-metadata {filenames}` returns `{attempted, recovered}`. Bad body → 400. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement schemas** (`validation.js`): + +```js +export const peerSyncRecordSchema = z.object({ + peerId: z.string().trim().min(1).max(120), + recordKind: z.enum(['universe', 'series', 'mediaCollection']), + recordId: z.string().trim().min(1).max(200), +}).strict(); +export const peerSyncNowSchema = z.object({ peerId: z.string().trim().min(1).max(120) }).strict(); +export const peerPullMetadataSchema = z.object({ + peerId: z.string().trim().min(1).max(120).optional(), + filenames: z.array(z.string().min(1).max(300)).max(5000), +}).strict(); +``` + +**Routes** (`routes/peerSync.js`): + +```js +router.post('/sync-record', asyncHandler(async (req, res) => { + const { peerId, recordKind, recordId } = validateRequest(peerSyncRecordSchema, req.body || {}); + res.json(await forcePushRecord(peerId, recordKind, recordId).catch(mapAndRethrow)); +})); +router.post('/sync-now', asyncHandler(async (req, res) => { + const { peerId } = validateRequest(peerSyncNowSchema, req.body || {}); + res.json(await syncNowForPeer(peerId).catch(mapAndRethrow)); +})); +router.post('/pull-metadata', asyncHandler(async (req, res) => { + const { filenames } = validateRequest(peerPullMetadataSchema, req.body || {}); + const { backfillMissingSidecars } = await import('../services/sharing/sidecarSync.js'); + res.json(await backfillMissingSidecars({ filenames })); +})); +``` + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): manual sync + pull-metadata routes` + +--- + +### Task 4.3: Client API wrappers + +**Files:** +- Create: `client/src/services/apiPeerSync.js` +- Modify: `client/src/services/api.js` (re-export per the services barrel rule) +- Test: `client/src/services/apiPeerSync.test.js` (mock fetch) + +- [ ] **Step 1: Write the failing test** — each wrapper calls the right path/method. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** using the existing `request()` helper from `apiCore.js` (pass `{ silent: true }` for calls whose callers own error UI): + +```js +import { request } from './apiCore.js'; +export const fetchSyncIntegrity = (peerId, kind, opts) => + request(`/api/peer-sync/integrity?peerId=${encodeURIComponent(peerId)}&kind=${encodeURIComponent(kind)}`, { ...opts }); +export const syncRecordToPeer = (peerId, recordKind, recordId) => + request('/api/peer-sync/sync-record', { method: 'POST', body: { peerId, recordKind, recordId } }); +export const syncNowForPeer = (peerId) => + request('/api/peer-sync/sync-now', { method: 'POST', body: { peerId } }); +export const pullMissingMetadata = (filenames) => + request('/api/peer-sync/pull-metadata', { method: 'POST', body: { filenames } }); +``` + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): client peer-sync API wrappers` + +--- + +### Task 4.4: `useSyncIntegrity` hook + +**Files:** +- Create: `client/src/hooks/useSyncIntegrity.js` +- Modify: `client/src/hooks/index.js` barrel + `README.md` +- Test: `client/src/hooks/useSyncIntegrity.test.js` + +- [ ] **Step 1: Write the failing test** — hook fetches integrity for enabled online peers, returns a `Map` + a `byPeer` breakdown. Mock the API wrapper. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** — fetch integrity per online peer (peers come from the existing instances/peers source), reduce to a worst-case status per record id (precedence: `peer-only`/`local-only`/`diverged`/`assets-missing` worse than `in-parity`). Expose `{ statusById, byPeer, loading, refresh }`. Reads from already-fetched peer state where possible (no duplicate peers fetch). + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): useSyncIntegrity hook` + +--- + +### Task 4.5: `SyncBadge` component + +**Files:** +- Create: `client/src/components/sync/SyncBadge.jsx` +- Test: `client/src/components/sync/SyncBadge.test.jsx` + +- [ ] **Step 1: Write the failing test** — renders the right label/color per status, including the distinct "not syncing — enable?" state when no peer has the category enabled. Uses the design tokens (`port-success`/`port-warning`/`port-error`). Clicking calls `onOpenDetail`. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** a small presentational component: prop `status` (one of `INTEGRITY_STATUS` + `'not-syncing'`), maps to `{ label, className }`, renders a button that calls `onOpenDetail`. No data fetching inside. + +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): SyncBadge component` + +--- + +### Task 4.6: `SyncDetailDrawer` + deep-linkable routes + +**Files:** +- Create: `client/src/components/sync/SyncDetailDrawer.jsx` +- Modify: `client/src/App.jsx` — routes `/universes/:id/sync`, `/pipeline/:id/sync`, `/media/collections/:id/sync` +- Modify: `client/src/pages/MediaCollections.jsx` (+ universe/series list pages) — render `` per row, wire navigation to the `/:id/sync` sub-route +- Test: `client/src/components/sync/SyncDetailDrawer.test.jsx` + +- [ ] **Step 1: Write the failing test** — drawer shows per-peer breakdown, thumbnails for the record's items, the diff, and action buttons ("Sync to peers", "Pull metadata", "Re-pull from peer") wired to the API wrappers. Closing navigates back to the parent route. + +- [ ] **Step 2: Run** → FAIL. + +- [ ] **Step 3: Implement** the drawer (reads `useSyncIntegrity` `byPeer` for the record id; reuses existing thumbnail components; buttons call `syncRecordToPeer` / `pullMissingMetadata` then `refresh()`). Add the three sub-routes in `App.jsx`. Render `` in each list page's rows using `statusById.get(record.id)`. + +**No `NAV_COMMANDS` change** — these are sub-routes of already-registered pages (per the spec); confirm `server/lib/navManifest.test.js` still passes untouched. + +- [ ] **Step 4: Run** → PASS. Run `cd client && npm test`. +- [ ] **Step 5: Commit** `feat(sync): SyncDetailDrawer + deep-linkable sync sub-routes + per-row badges` + +--- + +### Task 4.7: "Pull missing prompts" on the Unsorted view + +**Files:** +- Modify: the Unsorted collection view component (where `buildUnsortedCollection` is consumed) +- Test: client Vitest for the button → `pullMissingMetadata` call + +- [ ] **Step 1: Write the failing test** — button collects the Unsorted image filenames and calls `pullMissingMetadata`, then refreshes the gallery. +- [ ] **Step 2: Run** → FAIL. +- [ ] **Step 3: Implement** — a "Pull missing prompts" button visible on the Unsorted view; on click, `pullMissingMetadata(unsortedImageFilenames)` then refetch images. Toast the `{recovered}/{attempted}` result. +- [ ] **Step 4: Run** → PASS. +- [ ] **Step 5: Commit** `feat(sync): manual Pull missing prompts action on Unsorted view` + +--- + +# Final verification + +- [ ] `cd server && npm test` (full server suite green, incl. `index.test.js` barrel guard, `navManifest.test.js`). +- [ ] `cd client && npm test` (full client suite green). +- [ ] `npm run build` (client builds clean). +- [ ] Run `/simplify` on the changed code before the PR (per CLAUDE.md). +- [ ] Manual federation smoke test: with two installs and `mediaCollections` toggled ON for the peer — create a collection on A, confirm it appears on B with items + prompts; delete on A, confirm tombstone removes it on B; check the SyncBadge reflects parity; run "Pull missing prompts" on B's Unsorted. +- [ ] Add a `.changelog/NEXT.md` entry. + +# Self-review notes (addressed) + +- **Spec coverage:** Piece 1 → Group 1; Piece 2 → Group 2; Piece 3 → Group 3 (Tasks 3.1-3.3); Piece 4 → Group 4. Opt-in default preserved (no change to `DEFAULT_SYNC_CATEGORIES.mediaCollections`). Auto+manual sidecar backfill → Tasks 2.2 (auto) + 2.3/4.7 (manual). +- **Type consistency:** `INTEGRITY_STATUS` constants used identically across syncIntegrity.js, the hook, and SyncBadge. `forcePushRecord(peerId, recordKind, recordId)` signature matches the route + client wrapper. `buildLocalManifest(kind)` / `getPeerIntegrity({peerId, kind})` consistent across integrity.js and routes. +- **Open verification during impl:** confirm `peerWireRecordSchema` accepts a sanitized collection (id + passthrough); confirm `getCollection` signature change has no un-migrated callers; confirm universe/series branches of `buildLocalManifest` before shipping their badges (mediaCollection is the priority kind). diff --git a/docs/superpowers/specs/2026-05-23-federated-media-sync-integrity-design.md b/docs/superpowers/specs/2026-05-23-federated-media-sync-integrity-design.md new file mode 100644 index 000000000..47d4d41b1 --- /dev/null +++ b/docs/superpowers/specs/2026-05-23-federated-media-sync-integrity-design.md @@ -0,0 +1,242 @@ +# Federated Media Sync Parity + Per-Category Sync Integrity + +**Date:** 2026-05-23 +**Status:** Approved design — ready for implementation plan +**Branch:** `feat/federated-media-sync-integrity` + +## Problem + +PortOS installs federate as peers over Tailscale. Universes and series reach +parity between nodes, but **media collections do not**, and synced images +arrive without their generation metadata. Concretely: + +1. **Collections aren't first-class sync records.** `PEER_SUBSCRIBABLE_KINDS = + ['universe', 'series']` (`server/services/sharing/peerSync.js:73`). + Collections cross only as (a) `linkedCollection` cargo bundled when a parent + universe/series pushes, or (b) the `mediaCollections` snapshot category — + which is **off by default** (`DEFAULT_SYNC_CATEGORIES.mediaCollections = + false`, `server/services/instances.js:169`). Standalone collections and + collection edits therefore never reliably propagate. + +2. **Image gen-params never travel.** The peer-sync asset manifest carries only + `{ filename, kind, sha256 }` (`peerSync.js:458-478`) and `pullOneAsset` + writes just the image bytes (`peerSync.js:1488`). The `.metadata.json` + sidecar (prompt / model / seed) stays on the sender. Synced images land in + `data/images/` with no provenance and, because the referencing collection + didn't merge, surface in the client-side synthetic **"Unsorted"** bucket + (`client/src/lib/unsorted.js`). + +3. **The "live-pushed records" panel is opaque.** `Instances.jsx:573-634` shows + only a record-kind label + truncated id — no name, no preview, no detail — + so the user can't tell what they'd be unsubscribing/reversing. + +4. **No manual sync.** Sync is automatic only (60s interval + `peer:online`, + `server/services/syncOrchestrator.js`). + +## Goal + +Make media collections federate as reliably as universes/series, ensure synced +images carry their prompts, and give each category (Universes / Series / Media) +an at-a-glance **sync integrity badge** + deep-linkable detail drawer + manual +sync — so divergence is visible and actionable with confidence. + +## Approved decisions + +- **UI placement:** per-category badges on existing pages (`/universes`, + `/pipeline`, `/media/collections`) — **no new top-level page**. +- **Collection sync fix:** make collections first-class peer-sync records + (subscriptions + per-record push + tombstones), the same robust path + universes/series use. +- **Scope:** all four pieces ship together (visibility + collection sync + + sidecar metadata + manual trigger), as four logical commits. +- **Collections default:** stays **opt-in** (`mediaCollections: false` in + `DEFAULT_SYNC_CATEGORIES`). The integrity badge surfaces a clear + "not syncing — enable?" state rather than flipping the default. +- **Unsorted/sidecar backfill:** **both** — auto-pull sidecars going forward + during sync, plus a manual "Pull missing prompts" repair action for images + already sitting bare in Unsorted. + +--- + +## Piece 1 — Media collections as first-class sync records + +### Sender side +- Add `'mediaCollection'` to `PEER_SUBSCRIBABLE_KINDS` (`peerSync.js:73`). +- `buildPushPayload` (`peerSync.js:850`) gains a `sub.recordKind === + 'mediaCollection'` branch: load the collection (`getCollection`, with an + `includeDeleted` option), `sanitizeRecordForWire('mediaCollection', record)`, + build its asset manifest, and (for a live record) emit a payload + `{ kind: 'mediaCollection', record, assetManifest, sourceInstanceId, + portosMeta }`. Tombstone push sends an empty asset manifest, matching the + universe/series posture at `peerSync.js:865-877`. +- **Asset references for a collection.** `collectAssetReferences` + (`server/services/sharing/exporter.js`, used at `peerSync.js:459`) expects the + universe/series shape (`imageRefs` / `imageJobIds` / `videoPath`). Collections + store `items: [{ kind: 'image'|'video', ref, addedAt }]`. Add + `collectCollectionAssetReferences(collection)` that maps `image` refs → + `PATHS.images`, `video` refs → `PATHS.videos`. Feed the existing + `buildAssetManifest` machinery (or a thin collection wrapper). +- **recordEvents.** `mediaCollections.js` currently emits only + `emitRecordUpdated('universe'|'series', …)` for linked parents + (`mediaCollections.js:547-548, 605-606, 686-687, 715-716`). Add + `emitRecordUpdated('mediaCollection', id)` on create/update/delete for the + collection's *own* kind. Register a listener for `'mediaCollection'` in + `peerSync.js:1626-1648`. +- **Auto-subscribe.** `createCollection` fires + `autoSubscribeRecordToAllPeers('mediaCollection', id)`. This is gated by + `syncCategories.mediaCollections` (default off) via the existing + `peerHasCategory` check — opt-in is preserved automatically. + +### Soft-delete / tombstones (new scope) +Collections currently **hard-delete** — `deleteCollection` +(`mediaCollections.js:534-548`) splices the record out, with no `deleted` field. +First-class sync requires tombstones, otherwise a delete can't propagate and a +reverse-subscribed peer re-pushes the record back (resurrection). + +- Add `deleted: boolean` + `deletedAt: string|null` to the collection shape; + preserve them through `sanitizeCollection` (`mediaCollections.js:86-128`). +- `deleteCollection` **marks** `deleted: true, deletedAt: now` instead of + splicing. It still unlinks from any parent universe/series and emits the + parent + own-kind record events. +- `listCollections` (`mediaCollections.js:131`) filters `deleted` records unless + called with `{ includeDeleted: true }`. The synthetic Unsorted builder and all + read paths use the filtered list. +- `mergeMediaCollectionsFromSync` (`mediaCollections.js:738`) LWW must respect + `deleted` (a tombstone with a newer `updatedAt`/`deletedAt` wins over a live + incumbent). +- `tombstoneGc.js` gains a `mediaCollections` sweep so acked tombstones are + eventually pruned (parallel to universes/series/issues), keyed off + `peerTombstoneCursors`. + +### Receiver side +- The push receiver routes an incoming `mediaCollection` record through + `mergeMediaCollectionsFromSync([record])` (already serialized on the + collections write tail) + `diffAssetManifestAgainstLocal` + + `pullMissingAssetsFromPeer` — reusing the existing apply path. + +### Versioning & orchestrator +- Add `mediaCollections: 1` to `PORTOS_SCHEMA_VERSIONS` + (`server/lib/schemaVersions.js:38`, currently commented). The push receiver's + schema-version gate (`peerSync.js:1094-1124`) then protects older peers, and + the sender pauses pushes to peers that don't advertise the version. +- `categoriesCoveredByPeerSync` (`syncOrchestrator.js:356`) maps a + `mediaCollection` subscription → skip the `mediaCollections` snapshot category + for that peer (now push-driven), exactly like universe→universe and + series→pipeline. + +--- + +## Piece 2 — Image gen-params (sidecar) sync + +- `pullOneAsset` / `doPullOneAsset` (`peerSync.js:1404-1495`): after writing an + `image`, also pull its sidecar — `imageSidecarName(filename)` + (`server/services/sharing/buckets.js:107` → `foo.png` → `foo.metadata.json`) + — from the peer's `/data/images/` static mount (which serves the whole + directory, `server/index.js:540`) and `atomicWrite` it into `PATHS.images` + alongside the image. Best-effort — a missing sidecar (404) is normal for + assets that never had gen-params. Apply the same `sanitizeAssetFilename` + posture to the sidecar name before any FS/network op. +- Asset manifest `image` entries carry an optional `sidecarSha256` (when a + sidecar exists on the sender) so a *metadata-only* edit re-pulls. The + `diffAssetManifestAgainstLocal` comparison treats a present-but-mismatched (or + absent-locally) sidecar as a reason to re-pull. +- **Auto-backfill:** when an image is pulled (or detected sidecar-less) during a + sync cycle, automatically attempt the sidecar pull from the source peer. +- **Manual backfill:** a "Pull missing prompts" action (see Piece 4) repairs + images already sitting bare in Unsorted by fetching sidecars from whichever + online peer has them. + +--- + +## Piece 3 — Per-category sync integrity (badges + detail drawer) + +### Peer-facing manifest endpoint (Tailnet-only) +`GET /api/peer-sync/manifest?kind=` returns a +lightweight `{ id, name, updatedAt, deleted, assetHashes }[]`. Same trust +posture as the existing peer-sync routes (Tailnet-only per the documented +security model). Older peers 404 → the client renders "integrity unavailable +(peer too old)". + +### Integrity API +`GET /api/peer-sync/integrity?peerId=&kind=` fetches the peer manifest, diffs it +against the local manifest, and returns per-record status: + +| status | meaning | +| --- | --- | +| `in-parity` | same id, matching `updatedAt` + asset hashes | +| `local-only` | exists here, not on peer (and not a peer tombstone) | +| `peer-only` | exists on peer, not here | +| `diverged` | record fields differ (`updatedAt` mismatch) | +| `assets-missing` | record matches but one or more asset bytes absent on one side | +| `metadata-missing` | image present but sidecar (prompt) absent | + +The diff is a **pure, unit-tested function** (`computeRecordIntegrity(local, +remote)`), separate from the I/O that fetches the manifests. + +### Client +- A `SyncBadge` component on rows in `/universes`, `/pipeline`, + `/media/collections` showing the worst-case status for that record across + enabled peers, plus a distinct "not syncing — enable?" state when the category + is opt-in-off for a peer. +- Clicking opens a **deep-linkable** detail drawer (`/media/collections/:id/sync`, + `/universes/:id/sync`, `/pipeline/:id/sync`, per the linkable-routes rule) + showing thumbnails/previews, the field/asset diff, and the Piece-4 actions. +- Reads from already-fetched list/dashboard state where possible (no duplicate + fetch), per the reactive-UI conventions. Reuses existing thumbnail components. + +--- + +## Piece 4 — Manual sync trigger + +- `POST /api/peer-sync/sync-record { peerId, recordKind, recordId }` — forces + `pushRecordToPeer` bypassing the `lastPushedHash` unchanged short-circuit + (`peerSync.js:666`). +- `POST /api/peer-sync/sync-now { peerId }` — backfill-subscribe + push all + records of the peer's enabled kinds (initial parity restore; + `autoSubscribePeerToAllRecords` already exists for the subscribe half). +- `POST /api/peer-sync/pull-metadata { peerId?, filenames? }` — fetch sidecars + for bare local images from online peers (the manual Unsorted repair). +- Buttons wired into the badge drawer: "Sync to peers", "Pull metadata", + "Re-pull from peer". + +--- + +## Backward / forward compatibility (distribution model) + +- New push `kind: 'mediaCollection'` → the push Zod schema gains the kind; the + schema-version gate stops us pushing it to peers that don't advertise a + `mediaCollections` version, so no 400 storms on mixed-version federations. +- `deleted` / `deletedAt` / `sidecarSha256` are additive; older peers ignore + unknown fields. **No on-disk migration is required** for the soft-delete field + — absence of `deleted` reads as "live". (If runtime testing reveals existing + collections need normalization, add a `scripts/migrations/NNN-…js`; not + expected.) +- Integrity + manifest endpoints are new; they degrade gracefully (404 → "peer + too old") so they never break an older peer. + +## Validation & module conventions + +- All new routes validated via Zod in `server/lib/validation.js`; the peer-sync + push schema updated to accept `mediaCollection` (POST + any PUT use + `.partial()` per the schema-parity rule). +- **No new page** → no `NAV_COMMANDS` change. Badges live on existing, + already-registered pages; the sync detail drawer is a sub-route. +- Any new pure helper added to `server/lib/` / `client/src/lib/` / hooks / + services gets a barrel re-export + README row per the catalog rule. + +## Testing + +- **Unit:** `computeRecordIntegrity` diff; collection soft-delete + + LWW-with-`deleted` merge; `collectCollectionAssetReferences`; sidecar pull + + backfill; `mediaCollections` tombstone GC. +- **peerSync.test.js / mediaCollections.test.js** additions for the new kind. +- Record-creating suites mock `mockNoPeers()` **and** `mockNoPeerSync()` per the + test rule (`server/lib/mockPathsDataRoot.js`). + +## Logical commit boundaries + +1. Collections as first-class sync records (push/receive/subscribe/tombstone + + schemaVersion + orchestrator skip). +2. Sidecar metadata sync (manifest field + auto-pull + backfill endpoint). +3. Integrity API + peer manifest endpoint + pure diff (server). +4. SyncBadge + detail drawer + manual-sync buttons (client). diff --git a/server/lib/README.md b/server/lib/README.md index 9400e9e98..785b87463 100644 --- a/server/lib/README.md +++ b/server/lib/README.md @@ -88,7 +88,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub | `multipart.js` | Streaming multipart/form-data parser. | | `pdfImageEmbed.js` | PDF image embed helpers for comic / volume PDFs. | | `zipStream.js` | Streaming ZIP parser. | -| `assetHash.js` | Cross-transport SHA-256 cache for `data/images/*` — persists hashes in the asset's `.metadata.json` sidecar so the share-bucket exporter and the federated peer-sync push pipeline reuse the same value. | +| `assetHash.js` | Cross-transport SHA-256 cache for `data/images/*` — persists hashes in the asset's `.metadata.json` sidecar so the share-bucket exporter and the federated peer-sync push pipeline reuse the same value. `sidecarGenParamsHash` canonically hashes a sidecar's gen-params (excludes the machine-local `sha256` cache block) for cross-machine sidecar-convergence comparisons. | ## Process execution @@ -110,6 +110,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub | `peerSelfHost.js` | Tailscale-issued hostname this PortOS sends in federation. | | `peerUrl.js` | Build the base URL for a peer. | | `sharingOrigin.js` | Origin metadata for records imported from share buckets. | +| `syncIntegrity.js` | Pure diff of local vs remote manifest lists. `INTEGRITY_STATUS` constants + `computeRecordIntegrity(localList, remoteList)` — classifies each record as `in-parity`, `local-only`, `peer-only`, `diverged`, or `assets-missing`. No I/O. | | `syncWire.js` | Single source of truth for what fields cross the federated-peer wire (snapshot loop + per-record push agree). | | `tailscale.js` | Locate the Tailscale CLI binary and flag the sandboxed macOS App-bundle build (which can't write `tailscale cert` output outside its container). | | `httpsState.js` | Captures whether PortOS booted with HTTPS active. | @@ -165,7 +166,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub |---|---| | `asyncMutex.js` | Promise-based async mutex. | | `errorHandler.js` | `ServerError` + `asyncHandler` middleware. | -| `objects.js` | Object utilities — `deepMerge` (recursive merge w/ array replacement), `isPlainObject` (non-null, non-array `object` guard for JSON / LLM payloads), `POLLUTING_KEYS` (shared `__proto__`/`constructor`/`prototype` denylist for sanitizers). | +| `objects.js` | Object utilities — `deepMerge` (recursive merge w/ array replacement), `isPlainObject` (non-null, non-array `object` guard for JSON / LLM payloads), `POLLUTING_KEYS` (shared `__proto__`/`constructor`/`prototype` denylist for sanitizers), `canonicalStringify` (recursive sorted-key JSON serialization for cross-machine content hashing). | | `sseUtils.js` | Per-job SSE stream helpers (imageGen + others). | | `uuid.js` | `v4()` thin wrapper over `crypto.randomUUID()`. | diff --git a/server/lib/assetHash.js b/server/lib/assetHash.js index fef7a7475..3dc97a2b9 100644 --- a/server/lib/assetHash.js +++ b/server/lib/assetHash.js @@ -1,6 +1,8 @@ import { stat } from 'fs/promises'; import { basename, join } from 'path'; +import { createHash } from 'crypto'; import { atomicWrite, readJSONFile, sha256File, PATHS } from './fileUtils.js'; +import { isPlainObject, canonicalStringify, POLLUTING_KEYS } from './objects.js'; // Each image at `data/images/{uuid}.png` has a sibling `.metadata.json` // sidecar carrying its generation provenance (model, prompt, dimensions, etc). @@ -85,3 +87,40 @@ export async function getOrComputeImageSha256(imagePath) { }); return { hash, sidecar: next }; } + +/** + * Content hash of a sidecar's GEN-PARAMS, computed identically on every + * machine so a sender and receiver can compare "do we carry the same + * prompt/model metadata?" without the comparison drifting on machine-local + * cache state. + * + * CRITICAL: the `sha256` key is the per-image hash cache + * (`{ value, mtimeMs, size }`) that `getOrComputeImageSha256` re-stamps with + * the LOCAL image file's mtime+size on every read. Including it in the hash + * would make two byte-identical-gen-params sidecars on different machines + * produce DIFFERENT hashes that never converge — the receiver re-stamps its + * local mtime after every pull and re-diverges, re-flagging the image + * "missing" every sync cycle. So we strip `sha256` and hash a CANONICAL + * (sorted-key) serialization of the remaining gen-params. + * + * Returns a hex sha256 string, or null when the sidecar carries no gen-params + * beyond the cache key (nothing worth syncing — callers must NOT advertise a + * sidecarSha256 in that case). + */ +export function sidecarGenParamsHash(sidecar) { + if (!isPlainObject(sidecar)) return null; + // Object.create(null) so a peer-supplied `__proto__` key can't mutate the + // temp object's prototype via `genParams[key] = …` (prototype-pollution + // footgun). POLLUTING_KEYS (__proto__/constructor/prototype) are skipped + // outright — legitimate gen-params never use them, so the hash is identical + // for real sidecars while a hostile one stays inert and deterministic. + const genParams = Object.create(null); + let count = 0; + for (const key of Object.keys(sidecar)) { + if (key === 'sha256' || POLLUTING_KEYS.has(key)) continue; + genParams[key] = sidecar[key]; + count += 1; + } + if (count === 0) return null; + return createHash('sha256').update(canonicalStringify(genParams)).digest('hex'); +} diff --git a/server/lib/assetHash.test.js b/server/lib/assetHash.test.js index 98c244d65..8718c2580 100644 --- a/server/lib/assetHash.test.js +++ b/server/lib/assetHash.test.js @@ -2,7 +2,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { mkdir, rm, writeFile, readFile } from 'fs/promises'; import { join, dirname } from 'path'; import { tmpdir } from 'os'; -import { sidecarPathForImage, getOrComputeImageSha256 } from './assetHash.js'; +import { sidecarPathForImage, getOrComputeImageSha256, sidecarGenParamsHash } from './assetHash.js'; // PATHS.images is resolved at module-load from fileUtils and we deliberately // do NOT monkey-patch it — the absent-file tests use a tmpdir-rooted dir for @@ -138,4 +138,65 @@ describe('assetHash', () => { } }); }); + + describe('sidecarGenParamsHash', () => { + it('returns null when the sidecar has no gen-params (only the sha256 cache)', () => { + expect(sidecarGenParamsHash({ sha256: { value: 'a'.repeat(64), mtimeMs: 1, size: 2 } })).toBeNull(); + expect(sidecarGenParamsHash({})).toBeNull(); + }); + + it('returns null for non-object inputs', () => { + expect(sidecarGenParamsHash(null)).toBeNull(); + expect(sidecarGenParamsHash(undefined)).toBeNull(); + expect(sidecarGenParamsHash('x')).toBeNull(); + expect(sidecarGenParamsHash([1, 2])).toBeNull(); + }); + + it('returns a hex64 hash when gen-params exist', () => { + const h = sidecarGenParamsHash({ prompt: 'a cat', model: 'flux' }); + expect(h).toMatch(/^[a-f0-9]{64}$/); + }); + + it('CONVERGENCE: identical gen-params hash regardless of the sha256 cache block', () => { + // The core fix: two machines with byte-identical gen-params but DIFFERENT + // per-machine sha256 cache blocks (mtimeMs/size) must produce the SAME hash. + const a = sidecarGenParamsHash({ + prompt: 'a wizard', model: 'flux', steps: 30, + sha256: { value: 'a'.repeat(64), mtimeMs: 111, size: 222 }, + }); + const b = sidecarGenParamsHash({ + prompt: 'a wizard', model: 'flux', steps: 30, + sha256: { value: 'b'.repeat(64), mtimeMs: 999, size: 888 }, + }); + expect(a).toBe(b); + }); + + it('CONVERGENCE: identical hash regardless of key order', () => { + const a = sidecarGenParamsHash({ prompt: 'x', model: 'flux', steps: 30 }); + const b = sidecarGenParamsHash({ steps: 30, model: 'flux', prompt: 'x' }); + expect(a).toBe(b); + }); + + it('different gen-params produce different hashes', () => { + const a = sidecarGenParamsHash({ prompt: 'a cat' }); + const b = sidecarGenParamsHash({ prompt: 'a dog' }); + expect(a).not.toBe(b); + }); + + it('SECURITY: a hostile __proto__/constructor/prototype key does not pollute Object.prototype', () => { + // JSON.parse creates these as OWN keys; sidecarGenParamsHash must skip them + // and never mutate any prototype. Build via JSON.parse so __proto__ is a + // real own property (an object literal would invoke the proto setter). + const hostile = JSON.parse('{"prompt":"x","__proto__":{"polluted":true},"constructor":{"y":1},"prototype":{"z":2}}'); + sidecarGenParamsHash(hostile); + expect({}.polluted).toBeUndefined(); + expect(Object.prototype.polluted).toBeUndefined(); + }); + + it('SECURITY: the hash ignores polluting keys (identical to a sidecar without them)', () => { + const withPolluting = JSON.parse('{"prompt":"a wizard","model":"flux","__proto__":{"a":1},"constructor":{"b":2},"prototype":{"c":3}}'); + const clean = sidecarGenParamsHash({ prompt: 'a wizard', model: 'flux' }); + expect(sidecarGenParamsHash(withPolluting)).toBe(clean); + }); + }); }); diff --git a/server/lib/index.js b/server/lib/index.js index 6fd6a668d..e736538d3 100644 --- a/server/lib/index.js +++ b/server/lib/index.js @@ -94,6 +94,7 @@ export * from './peerHttpClient.js'; export * from './peerSelfHost.js'; export * from './peerUrl.js'; export * from './sharingOrigin.js'; +export * from './syncIntegrity.js'; export * from './syncWire.js'; export * from './tailscale.js'; diff --git a/server/lib/objects.js b/server/lib/objects.js index 78d4b95ba..469cc0b4b 100644 --- a/server/lib/objects.js +++ b/server/lib/objects.js @@ -54,3 +54,50 @@ export const deepMerge = (base, patch) => { } return out; }; + +/** + * Stable, canonical JSON serialization: recursively sorts object keys so two + * structurally-equal values produce byte-identical strings regardless of the + * key-insertion order they happened to be built with. Use this when a string + * (or hash of a string) must be COMPARABLE ACROSS MACHINES — e.g. content- + * hashing a sidecar's gen-params on a sender and re-deriving the same hash on + * a receiver where the object was rebuilt in a different key order. + * + * Arrays preserve order (order is semantic for arrays); object keys are sorted + * lexicographically. Only TRUE plain objects (prototype `Object.prototype` or + * `null`) get key-sorted; everything else — primitives, null, and exotic + * objects like Date / Map / class instances — serializes via native + * `JSON.stringify` rules (so e.g. a Date round-trips through `toJSON` to its + * ISO string instead of collapsing to `{}`). `undefined` and functions are + * dropped exactly as `JSON.stringify` drops them. + */ +// NB: isPlainObject() is intentionally loose (true for Date/Map/class +// instances), so it's WRONG here — it would key-sort a Date's (empty) own keys +// into `{}`. Require the prototype to be Object.prototype or null instead. +const isCanonicalSortable = (v) => { + if (v === null || typeof v !== 'object' || Array.isArray(v)) return false; + const proto = Object.getPrototypeOf(v); + return proto === Object.prototype || proto === null; +}; + +export const canonicalStringify = (value) => { + if (Array.isArray(value)) { + // Array.from (not .map) so SPARSE-array holes are visited and serialized as + // `null` — matching JSON.stringify ([1,,2] → "[1,null,2]"). `.map` preserves + // holes, and `.join` would then emit invalid JSON ("[1,,2]"), diverging the + // cross-machine hash for any value containing a sparse array. + return `[${Array.from(value, (v) => canonicalStringify(v) ?? 'null').join(',')}]`; + } + if (isCanonicalSortable(value)) { + const parts = []; + for (const key of Object.keys(value).sort()) { + const serialized = canonicalStringify(value[key]); + // Skip keys whose value serializes to undefined (functions / undefined) + // — matches JSON.stringify dropping them from objects. + if (serialized === undefined) continue; + parts.push(`${JSON.stringify(key)}:${serialized}`); + } + return `{${parts.join(',')}}`; + } + return JSON.stringify(value); +}; diff --git a/server/lib/objects.test.js b/server/lib/objects.test.js index 5d365bf3a..0b01201cf 100644 --- a/server/lib/objects.test.js +++ b/server/lib/objects.test.js @@ -1,5 +1,5 @@ import { describe, it, expect } from 'vitest'; -import { deepMerge, isPlainObject } from './objects.js'; +import { deepMerge, isPlainObject, canonicalStringify } from './objects.js'; describe('isPlainObject', () => { it('returns true for plain `{}`-shaped values', () => { @@ -94,3 +94,73 @@ describe('deepMerge', () => { expect(deepMerge([1, 2], { a: 1 })).toEqual({ a: 1 }); }); }); + +describe('canonicalStringify', () => { + it('produces identical output regardless of key insertion order', () => { + const a = canonicalStringify({ b: 1, a: 2, c: 3 }); + const b = canonicalStringify({ c: 3, a: 2, b: 1 }); + expect(a).toBe(b); + }); + + it('sorts keys recursively in nested objects', () => { + const a = canonicalStringify({ outer: { z: 1, a: 2 }, first: true }); + const b = canonicalStringify({ first: true, outer: { a: 2, z: 1 } }); + expect(a).toBe(b); + }); + + it('preserves array order (order is semantic for arrays)', () => { + expect(canonicalStringify([3, 1, 2])).toBe('[3,1,2]'); + expect(canonicalStringify([1, 2, 3])).not.toBe(canonicalStringify([3, 2, 1])); + }); + + it('serializes primitives via native JSON rules', () => { + expect(canonicalStringify('hi')).toBe('"hi"'); + expect(canonicalStringify(42)).toBe('42'); + expect(canonicalStringify(true)).toBe('true'); + expect(canonicalStringify(null)).toBe('null'); + }); + + it('drops undefined / function values inside objects (matches JSON.stringify)', () => { + expect(canonicalStringify({ a: 1, b: undefined, c: () => {} })).toBe('{"a":1}'); + }); + + it('serializes undefined array elements as null (matches JSON.stringify)', () => { + expect(canonicalStringify([1, undefined, 2])).toBe('[1,null,2]'); + }); + + it('serializes SPARSE-array holes as null, not invalid JSON (matches JSON.stringify)', () => { + // eslint-disable-next-line no-sparse-arrays + const sparse = [1, , 2]; + expect(canonicalStringify(sparse)).toBe('[1,null,2]'); + expect(canonicalStringify(sparse)).toBe(JSON.stringify(sparse)); + }); + + it('handles nested arrays of objects with sorted keys', () => { + const a = canonicalStringify({ items: [{ y: 1, x: 2 }] }); + const b = canonicalStringify({ items: [{ x: 2, y: 1 }] }); + expect(a).toBe(b); + }); + + it('serializes a Date via native JSON rules (toJSON → ISO), not {} (matches JSON.stringify)', () => { + const d = new Date('2026-01-02T03:04:05.000Z'); + expect(canonicalStringify(d)).toBe(JSON.stringify(d)); + expect(canonicalStringify(d)).toBe('"2026-01-02T03:04:05.000Z"'); + }); + + it('serializes a nested Date value through native rules', () => { + const d = new Date('2026-01-02T03:04:05.000Z'); + expect(canonicalStringify({ when: d })).toBe('{"when":"2026-01-02T03:04:05.000Z"}'); + }); + + it('still key-sorts an Object.create(null) value (prototype null is canonical-sortable)', () => { + const o = Object.create(null); + o.b = 1; o.a = 2; + expect(canonicalStringify(o)).toBe('{"a":2,"b":1}'); + }); + + it('serializes a class instance with toJSON via native rules (not key-sorted own props)', () => { + class Box { constructor() { this.z = 1; this.a = 2; } toJSON() { return { tag: 'box' }; } } + expect(canonicalStringify(new Box())).toBe(JSON.stringify(new Box())); + expect(canonicalStringify(new Box())).toBe('{"tag":"box"}'); + }); +}); diff --git a/server/lib/schemaVersions.js b/server/lib/schemaVersions.js index 59781e50f..c66f5eb84 100644 --- a/server/lib/schemaVersions.js +++ b/server/lib/schemaVersions.js @@ -35,7 +35,7 @@ export const PORTOS_SCHEMA_VERSIONS = Object.freeze({ // layout for issues and series. pipelineIssues: 1, pipelineSeries: 1, - // future: mediaCollections: N, ... + mediaCollections: 1, }); /** diff --git a/server/lib/schemaVersions.test.js b/server/lib/schemaVersions.test.js index eb16d8b63..95399ecbd 100644 --- a/server/lib/schemaVersions.test.js +++ b/server/lib/schemaVersions.test.js @@ -23,6 +23,10 @@ describe('PORTOS_SCHEMA_VERSIONS', () => { expect(PORTOS_SCHEMA_VERSIONS.pipelineIssues).toBe(1); expect(PORTOS_SCHEMA_VERSIONS.pipelineSeries).toBe(1); }); + + it('declares mediaCollections layout version', () => { + expect(PORTOS_SCHEMA_VERSIONS.mediaCollections).toBe(1); + }); }); describe('buildPortosMeta', () => { diff --git a/server/lib/syncIntegrity.js b/server/lib/syncIntegrity.js new file mode 100644 index 000000000..945b73441 --- /dev/null +++ b/server/lib/syncIntegrity.js @@ -0,0 +1,80 @@ +/** + * Pure integrity diff for federated record lists. + * + * Compares local vs remote manifest lists and classifies each record by + * synchronisation status. No I/O — all logic is pure so it can be called + * from any service without side effects. + */ + +export const INTEGRITY_STATUS = Object.freeze({ + IN_PARITY: 'in-parity', + LOCAL_ONLY: 'local-only', + PEER_ONLY: 'peer-only', + DIVERGED: 'diverged', + ASSETS_MISSING: 'assets-missing', +}); + +const sortedHashes = (a) => [...(Array.isArray(a) ? a : [])].sort(); + +const hashesEqual = (a, b) => { + const sa = sortedHashes(a); + const sb = sortedHashes(b); + return sa.length === sb.length && sa.every((h, i) => h === sb[i]); +}; + +/** + * Pure diff of two manifest lists. + * + * Each entry shape: `{ id, name?, updatedAt, deleted?, assetHashes }`. + * + * Tombstone handling: when BOTH sides are tombstoned (deleted === true) the + * pair is omitted from the output entirely — both agree the record is gone, so + * there's nothing to reconcile. A live-vs-tombstoned mismatch IS surfaced as + * LOCAL_ONLY / PEER_ONLY (the live side still holds a record the other side + * deleted), so the user can decide whether to propagate the delete or re-push. + * + * @param {Array} localList - Local manifest rows. + * @param {Array} remoteList - Remote manifest rows. + * @returns {Array<{id:string, name:string, status:string}>} + */ +export function computeRecordIntegrity(localList, remoteList) { + const byId = new Map(); + + for (const l of localList || []) { + byId.set(l.id, { id: l.id, name: l.name, local: l, remote: null }); + } + + for (const r of remoteList || []) { + const cur = byId.get(r.id) || { id: r.id, name: r.name, local: null, remote: null }; + cur.remote = r; + if (!cur.name) cur.name = r.name; + byId.set(r.id, cur); + } + + const out = []; + + for (const { id, name, local, remote } of byId.values()) { + const localLive = local && local.deleted !== true; + const remoteLive = remote && remote.deleted !== true; + + let status; + if (localLive && !remoteLive) { + status = INTEGRITY_STATUS.LOCAL_ONLY; + } else if (!localLive && remoteLive) { + status = INTEGRITY_STATUS.PEER_ONLY; + } else if (!localLive && !remoteLive) { + // Both tombstoned — agree on deletion, omit from output. + continue; + } else if (local.updatedAt !== remote.updatedAt) { + status = INTEGRITY_STATUS.DIVERGED; + } else if (!hashesEqual(local.assetHashes, remote.assetHashes)) { + status = INTEGRITY_STATUS.ASSETS_MISSING; + } else { + status = INTEGRITY_STATUS.IN_PARITY; + } + + out.push({ id, name: name || id, status }); + } + + return out; +} diff --git a/server/lib/syncIntegrity.test.js b/server/lib/syncIntegrity.test.js new file mode 100644 index 000000000..4f5b3981a --- /dev/null +++ b/server/lib/syncIntegrity.test.js @@ -0,0 +1,121 @@ +import { describe, it, expect } from 'vitest'; +import { computeRecordIntegrity, INTEGRITY_STATUS } from './syncIntegrity.js'; + +const makeRow = (overrides) => ({ + id: 'r1', + name: 'Record One', + updatedAt: '2026-05-23T00:00:00.000Z', + deleted: false, + assetHashes: [], + ...overrides, +}); + +describe('computeRecordIntegrity', () => { + it('returns IN_PARITY when both sides match (same updatedAt, same hashes)', () => { + const row = makeRow({ assetHashes: ['aaaa', 'bbbb'] }); + const result = computeRecordIntegrity([row], [row]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.IN_PARITY }); + }); + + it('returns LOCAL_ONLY when record exists locally but not on peer', () => { + const local = makeRow(); + const result = computeRecordIntegrity([local], []); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.LOCAL_ONLY }); + }); + + it('returns PEER_ONLY when record exists on peer but not locally', () => { + const remote = makeRow(); + const result = computeRecordIntegrity([], [remote]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.PEER_ONLY }); + }); + + it('returns DIVERGED when updatedAt differs', () => { + const local = makeRow({ updatedAt: '2026-05-23T00:00:00.000Z' }); + const remote = makeRow({ updatedAt: '2026-05-24T00:00:00.000Z' }); + const result = computeRecordIntegrity([local], [remote]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.DIVERGED }); + }); + + it('returns ASSETS_MISSING when updatedAt is the same but hashes differ', () => { + const ts = '2026-05-23T00:00:00.000Z'; + const local = makeRow({ updatedAt: ts, assetHashes: ['aaaa'] }); + const remote = makeRow({ updatedAt: ts, assetHashes: ['bbbb'] }); + const result = computeRecordIntegrity([local], [remote]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.ASSETS_MISSING }); + }); + + it('treats hash order as irrelevant for IN_PARITY (sorts before compare)', () => { + const ts = '2026-05-23T00:00:00.000Z'; + const local = makeRow({ updatedAt: ts, assetHashes: ['bbbb', 'aaaa'] }); + const remote = makeRow({ updatedAt: ts, assetHashes: ['aaaa', 'bbbb'] }); + const result = computeRecordIntegrity([local], [remote]); + expect(result[0].status).toBe(INTEGRITY_STATUS.IN_PARITY); + }); + + it('omits pairs where both sides are tombstoned (deleted)', () => { + const local = makeRow({ deleted: true }); + const remote = makeRow({ deleted: true }); + const result = computeRecordIntegrity([local], [remote]); + expect(result).toHaveLength(0); + }); + + it('treats deleted:true locally + live remotely as PEER_ONLY', () => { + const local = makeRow({ deleted: true }); + const remote = makeRow({ deleted: false }); + const result = computeRecordIntegrity([local], [remote]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.PEER_ONLY }); + }); + + it('treats live locally + deleted:true remotely as LOCAL_ONLY', () => { + const local = makeRow({ deleted: false }); + const remote = makeRow({ deleted: true }); + const result = computeRecordIntegrity([local], [remote]); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ id: 'r1', status: INTEGRITY_STATUS.LOCAL_ONLY }); + }); + + it('handles null/undefined localList and remoteList gracefully', () => { + expect(computeRecordIntegrity(null, null)).toEqual([]); + expect(computeRecordIntegrity(undefined, undefined)).toEqual([]); + const row = makeRow(); + expect(computeRecordIntegrity([row], null)).toHaveLength(1); + expect(computeRecordIntegrity(null, [row])).toHaveLength(1); + }); + + it('uses name from peer when local is absent', () => { + const remote = makeRow({ name: 'Peer Name' }); + const result = computeRecordIntegrity([], [remote]); + expect(result[0].name).toBe('Peer Name'); + }); + + it('falls back to id when name is absent on both sides', () => { + const local = makeRow({ name: undefined }); + const remote = makeRow({ name: undefined }); + const result = computeRecordIntegrity([local], [remote]); + expect(result[0].name).toBe('r1'); + }); + + it('handles multiple records correctly', () => { + const ts = '2026-05-23T00:00:00.000Z'; + const localList = [ + makeRow({ id: 'a', name: 'A', updatedAt: ts, assetHashes: [] }), + makeRow({ id: 'b', name: 'B', updatedAt: ts, assetHashes: [] }), + ]; + const remoteList = [ + makeRow({ id: 'a', name: 'A', updatedAt: ts, assetHashes: [] }), + makeRow({ id: 'c', name: 'C', updatedAt: ts, assetHashes: [] }), + ]; + const result = computeRecordIntegrity(localList, remoteList); + expect(result).toHaveLength(3); + const byId = Object.fromEntries(result.map((r) => [r.id, r.status])); + expect(byId.a).toBe(INTEGRITY_STATUS.IN_PARITY); + expect(byId.b).toBe(INTEGRITY_STATUS.LOCAL_ONLY); + expect(byId.c).toBe(INTEGRITY_STATUS.PEER_ONLY); + }); +}); diff --git a/server/lib/syncWire.js b/server/lib/syncWire.js index c14c7c407..474e3fbe0 100644 --- a/server/lib/syncWire.js +++ b/server/lib/syncWire.js @@ -126,12 +126,13 @@ export function sanitizeRecordForWire(kind, record) { return { ...rest, ...sanitizeSoftDeleteFields(record) }; } case 'mediaCollection': { - // Collections have no `ephemeral` / `deleted` semantics today — the - // record shape is already wire-safe as produced by sanitizeCollection. - // The case exists so future per-record push fan-out for collections has - // a single sanitizer to extend (e.g. stripping a future local-only - // field like a UI sort preference) without forking the wire contract. - return record; + // Strip then re-add soft-delete fields at tail so the byte-stable + // checksum invariant holds regardless of the on-disk key position. + // Collections have no `ephemeral` field — unlike universe/series, + // they are always wire-syncable when non-ephemeral (there is no + // collection-level ephemeral flag), so no ephemeral minimization path. + const { deleted: _d, deletedAt: _da, ...rest } = record; + return { ...rest, ...sanitizeSoftDeleteFields(record) }; } default: return null; diff --git a/server/lib/validation.js b/server/lib/validation.js index 8da83deb1..9fb36d333 100644 --- a/server/lib/validation.js +++ b/server/lib/validation.js @@ -1255,7 +1255,7 @@ export const subscriptionCreateSchema = z.object({ // subscriptions target another PortOS instance over Tailnet. export const peerSubscribeSchema = z.object({ peerId: z.string().trim().min(1).max(120), - recordKind: z.enum(['universe', 'series']), + recordKind: z.enum(['universe', 'series', 'mediaCollection']), recordId: z.string().trim().min(1).max(120), }).strict(); @@ -1263,11 +1263,25 @@ export const peerSubscribeSchema = z.object({ // second-pass scrub against path separators inside the service layer; this // schema just constrains shape + caps so a malformed manifest doesn't bypass // validation entirely. SHA-256 is hex-64 when present. -const peerAssetManifestEntrySchema = z.object({ - filename: z.string().trim().min(1).max(255), - kind: z.enum(['image', 'image-ref', 'video']), - sha256: z.string().regex(/^[a-f0-9]{64}$/i).optional(), -}).strict(); +// +// Discriminated on `kind` because `sidecarSha256` (the gen-params sidecar hash) +// is ONLY meaningful for images — image-ref/video entries carry no sidecar, so +// `.strict()` on the non-image branch rejects a stray `sidecarSha256` instead +// of silently accepting a malformed sender payload. +const hex64 = z.string().regex(/^[a-f0-9]{64}$/i); +const peerAssetManifestEntrySchema = z.discriminatedUnion('kind', [ + z.object({ + filename: z.string().trim().min(1).max(255), + kind: z.literal('image'), + sha256: hex64.optional(), + sidecarSha256: hex64.optional(), + }).strict(), + z.object({ + filename: z.string().trim().min(1).max(255), + kind: z.enum(['image-ref', 'video']), + sha256: hex64.optional(), + }).strict(), +]); // One sanitized record on the wire. Mirrors sanitizeRecordForWire's output: // id is required, soft-delete fields are tail-canonical, and the receiver's @@ -1312,29 +1326,60 @@ const peerSyncPushBase = { assetManifest: z.array(peerAssetManifestEntrySchema).max(2000), sourceInstanceId: z.string().trim().min(1).max(120), portosMeta: portosMetaSchema, - // Optional bundled media collection — Stage 5 media-collections sync - // attaches the universe / series's linked collection so collection-only - // edits propagate via the per-record push pipeline. Same shape as a - // record on the wire (id required, sanitizer handles the rest); without - // this field on both push branches the strict() rejection drops every - // production push from a universe / series with images. See - // peerSync.js buildPushPayload and applyIncomingPush. - linkedCollection: peerWireRecordSchema.optional(), }; +// Optional bundled media collection — Stage 5 media-collections sync attaches +// the universe / series's linked collection so collection-only edits propagate +// via the per-record push pipeline. Same shape as a record on the wire (id +// required, sanitizer handles the rest). ONLY valid on universe/series pushes: +// a mediaCollection push IS the collection, so accepting linkedCollection there +// would let a sender smuggle an arbitrary EXTRA collection that the receiver's +// applyIncomingPush merges — a side-channel to overwrite collections outside the +// explicit per-record subscription. The mediaCollection branch's .strict() +// therefore rejects it. See peerSync.js buildPushPayload (never sets it for the +// mediaCollection kind) and applyIncomingPush. +const linkedCollectionField = { linkedCollection: peerWireRecordSchema.optional() }; const universePushSchema = z.object({ kind: z.literal('universe'), ...peerSyncPushBase, + ...linkedCollectionField, }).strict(); const seriesPushSchema = z.object({ kind: z.literal('series'), ...peerSyncPushBase, + ...linkedCollectionField, issues: z.array(peerWireRecordSchema).max(1000).optional(), }).strict(); +const mediaCollectionPushSchema = z.object({ + kind: z.literal('mediaCollection'), + ...peerSyncPushBase, +}).strict(); export const peerSyncPushSchema = z.discriminatedUnion('kind', [ universePushSchema, seriesPushSchema, + mediaCollectionPushSchema, ]); +// Manual sync action schemas — used by POST /sync-record, /sync-now, /pull-metadata. + +export const peerSyncRecordSchema = z.object({ + peerId: z.string().trim().min(1).max(120), + recordKind: z.enum(['universe', 'series', 'mediaCollection']), + recordId: z.string().trim().min(1).max(200), +}).strict(); + +export const peerSyncNowSchema = z.object({ + peerId: z.string().trim().min(1).max(120), +}).strict(); + +export const peerPullMetadataSchema = z.object({ + // Backfill tries every online peer; no per-peer scoping field today. + // .trim() so a stray-whitespace filename (' a.png ') normalizes to the real + // name instead of passing validation and then failing sanitization/disk + // lookup (a confusing 200 with attempted>0, recovered=0). Matches the + // manifest-entry filename handling. + filenames: z.array(z.string().trim().min(1).max(300)).max(5000), +}).strict(); + // ============================================================================= // CREATIVE DIRECTOR SCHEMAS // ============================================================================= diff --git a/server/routes/peerSync.js b/server/routes/peerSync.js index 74e344eff..1a19a52b8 100644 --- a/server/routes/peerSync.js +++ b/server/routes/peerSync.js @@ -28,16 +28,23 @@ import { validateRequest, peerSubscribeSchema, peerSyncPushSchema, + peerSyncRecordSchema, + peerSyncNowSchema, + peerPullMetadataSchema, } from '../lib/validation.js'; import { listPeerSubscriptions, subscribePeer, unsubscribePeer, applyIncomingPush, + forcePushRecord, + syncNowForPeer, ERR_NOT_FOUND, ERR_VALIDATION, ERR_SCHEMA_VERSION_AHEAD, + PEER_SUBSCRIBABLE_KINDS, } from '../services/sharing/peerSync.js'; +import { buildLocalManifest, getPeerIntegrity } from '../services/sharing/integrity.js'; const router = Router(); @@ -118,4 +125,71 @@ router.delete('/subscriptions/:id', asyncHandler(async (req, res) => { res.json(result); })); +// Guard: only accept record kinds that the peer-sync pipeline actually handles. +const validKind = (k) => typeof k === 'string' && PEER_SUBSCRIBABLE_KINDS.includes(k); + +// --- GET /manifest --- advertise this instance's record manifest for a kind. +// +// Called by peers running `getPeerIntegrity` to compare their local state +// against ours. The response is a flat list of rows — one per record — +// including tombstones so deletes diff correctly. Asset hashes are sorted +// sha256 strings so the diff is order-independent. +router.get('/manifest', asyncHandler(async (req, res) => { + // Trim once and validate/use the trimmed value — a padded `?kind= universe ` + // should resolve like `universe`, not 400 on whitespace. + const kind = typeof req.query.kind === 'string' ? req.query.kind.trim() : ''; + if (!validKind(kind)) { + throw new ServerError('invalid kind', { status: 400, code: 'VALIDATION_ERROR' }); + } + res.json({ records: await buildLocalManifest(kind) }); +})); + +// --- GET /integrity --- compare this instance's records against a peer's. +// +// Fetches the peer's /manifest, runs the pure diff, and returns +// `{ available, reason?, records: [{ id, name, status }] }`. +router.get('/integrity', asyncHandler(async (req, res) => { + // Trim ONCE and use the trimmed values for both validation AND the service + // call. Validating the trimmed value but passing the raw one let + // `?peerId=%20peer-a%20` pass the emptiness check yet fail to match the peer + // registry, returning a confusing `peer-not-found`. + const peerId = typeof req.query.peerId === 'string' ? req.query.peerId.trim() : ''; + const kind = typeof req.query.kind === 'string' ? req.query.kind.trim() : ''; + if (!peerId) { + throw new ServerError('peerId required', { status: 400, code: 'VALIDATION_ERROR' }); + } + if (!validKind(kind)) { + throw new ServerError('invalid kind', { status: 400, code: 'VALIDATION_ERROR' }); + } + res.json(await getPeerIntegrity({ peerId, kind })); +})); + +// --- POST /sync-record --- force a push for a specific record to a specific peer. +// +// Bypasses the unchanged-hash short-circuit so the receiver always gets the +// latest state. Creates the subscription if it doesn't exist yet. +router.post('/sync-record', asyncHandler(async (req, res) => { + const { peerId, recordKind, recordId } = validateRequest(peerSyncRecordSchema, req.body || {}); + res.json(await forcePushRecord(peerId, recordKind, recordId).catch(mapAndRethrow)); +})); + +// --- POST /sync-now --- trigger an immediate full-sync for a peer. +// +// Backfills subscriptions for every enabled category then retries all pending +// pushes. Best-effort — per-kind failures are swallowed server-side. +router.post('/sync-now', asyncHandler(async (req, res) => { + const { peerId } = validateRequest(peerSyncNowSchema, req.body || {}); + res.json(await syncNowForPeer(peerId).catch(mapAndRethrow)); +})); + +// --- POST /pull-metadata --- backfill missing sidecar metadata for images. +// +// Accepts a list of image filenames and attempts to pull their .metadata.json +// sidecar from any peer that has a copy. Delegates to sidecarSync.js. +router.post('/pull-metadata', asyncHandler(async (req, res) => { + const { filenames } = validateRequest(peerPullMetadataSchema, req.body || {}); + const { backfillMissingSidecars } = await import('../services/sharing/sidecarSync.js'); + res.json(await backfillMissingSidecars({ filenames })); +})); + export default router; diff --git a/server/routes/peerSync.test.js b/server/routes/peerSync.test.js index 60666580f..eadc07177 100644 --- a/server/routes/peerSync.test.js +++ b/server/routes/peerSync.test.js @@ -8,12 +8,26 @@ vi.mock('../services/sharing/peerSync.js', () => ({ subscribePeer: vi.fn(), unsubscribePeer: vi.fn(), applyIncomingPush: vi.fn(), + forcePushRecord: vi.fn(), + syncNowForPeer: vi.fn(), ERR_NOT_FOUND: 'PEER_SYNC_SUBSCRIPTION_NOT_FOUND', ERR_VALIDATION: 'PEER_SYNC_SUBSCRIPTION_VALIDATION', ERR_SCHEMA_VERSION_AHEAD: 'PEER_SYNC_SCHEMA_VERSION_AHEAD', + PEER_SUBSCRIBABLE_KINDS: Object.freeze(['universe', 'series', 'mediaCollection']), +})); + +vi.mock('../services/sharing/integrity.js', () => ({ + buildLocalManifest: vi.fn(), + getPeerIntegrity: vi.fn(), +})); + +vi.mock('../services/sharing/sidecarSync.js', () => ({ + backfillMissingSidecars: vi.fn(), })); import * as svc from '../services/sharing/peerSync.js'; +import * as integritySvc from '../services/sharing/integrity.js'; +import * as sidecarSvc from '../services/sharing/sidecarSync.js'; import peerSyncRoutes from './peerSync.js'; const buildApp = () => { @@ -29,6 +43,8 @@ const serviceError = (msg, code) => Object.assign(new Error(msg), { code }); describe('peer-sync routes', () => { beforeEach(() => { vi.clearAllMocks(); + integritySvc.buildLocalManifest.mockResolvedValue([]); + integritySvc.getPeerIntegrity.mockResolvedValue({ available: false, reason: 'peer-not-found', records: [] }); }); describe('POST /api/peer-sync/push', () => { @@ -105,6 +121,78 @@ describe('peer-sync routes', () => { expect(svc.applyIncomingPush).not.toHaveBeenCalled(); }); + it('accepts an image manifest entry carrying sidecarSha256', async () => { + svc.applyIncomingPush.mockResolvedValue({ missingAssets: [], reverseSubscriptionCreated: false, ackedDeletesUpTo: 0 }); + const res = await request(buildApp()) + .post('/api/peer-sync/push') + .send({ + kind: 'universe', + record: { id: 'u1' }, + assetManifest: [{ filename: 'a.png', kind: 'image', sha256: 'a'.repeat(64), sidecarSha256: 'b'.repeat(64) }], + sourceInstanceId: 'peer-a', + }); + expect(res.status).toBe(200); + }); + + it('accepts a video/image-ref manifest entry without a sidecar hash', async () => { + svc.applyIncomingPush.mockResolvedValue({ missingAssets: [], reverseSubscriptionCreated: false, ackedDeletesUpTo: 0 }); + const res = await request(buildApp()) + .post('/api/peer-sync/push') + .send({ + kind: 'mediaCollection', + record: { id: 'c1' }, + assetManifest: [ + { filename: 'v.mp4', kind: 'video', sha256: 'c'.repeat(64) }, + { filename: 'r.png', kind: 'image-ref', sha256: 'd'.repeat(64) }, + ], + sourceInstanceId: 'peer-a', + }); + expect(res.status).toBe(200); + }); + + it('accepts linkedCollection on a universe push (parent-bundled collection)', async () => { + svc.applyIncomingPush.mockResolvedValue({ missingAssets: [], reverseSubscriptionCreated: false, ackedDeletesUpTo: 0 }); + const res = await request(buildApp()) + .post('/api/peer-sync/push') + .send({ + kind: 'universe', + record: { id: 'u1' }, + assetManifest: [], + linkedCollection: { id: 'col-bundled' }, + sourceInstanceId: 'peer-a', + }); + expect(res.status).toBe(200); + }); + + it('400s when a mediaCollection push carries linkedCollection (no smuggled extra collection)', async () => { + // A mediaCollection push IS the collection — accepting linkedCollection + // would be a side-channel to overwrite an unrelated collection. + const res = await request(buildApp()) + .post('/api/peer-sync/push') + .send({ + kind: 'mediaCollection', + record: { id: 'c1' }, + assetManifest: [], + linkedCollection: { id: 'col-smuggled' }, + sourceInstanceId: 'peer-a', + }); + expect(res.status).toBe(400); + expect(svc.applyIncomingPush).not.toHaveBeenCalled(); + }); + + it('400s when a non-image manifest entry carries sidecarSha256 (discriminated union)', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/push') + .send({ + kind: 'mediaCollection', + record: { id: 'c1' }, + assetManifest: [{ filename: 'v.mp4', kind: 'video', sha256: 'c'.repeat(64), sidecarSha256: 'e'.repeat(64) }], + sourceInstanceId: 'peer-a', + }); + expect(res.status).toBe(400); + expect(svc.applyIncomingPush).not.toHaveBeenCalled(); + }); + it('400s when the record is missing an id', async () => { // Stage 1's schema-parity rule: validation must catch the malformed // record at the route boundary, not let the service throw. @@ -313,4 +401,237 @@ describe('peer-sync routes', () => { expect(res.status).toBe(404); }); }); + + describe('GET /api/peer-sync/manifest', () => { + it('200 with records for a valid kind', async () => { + const records = [ + { id: 'col-1', name: 'My Collection', updatedAt: '2026-05-23T00:00:00.000Z', deleted: false, assetHashes: [] }, + ]; + integritySvc.buildLocalManifest.mockResolvedValue(records); + + const res = await request(buildApp()) + .get('/api/peer-sync/manifest?kind=mediaCollection'); + expect(res.status).toBe(200); + expect(res.body).toEqual({ records }); + expect(integritySvc.buildLocalManifest).toHaveBeenCalledWith('mediaCollection'); + }); + + it('400 when kind is missing', async () => { + const res = await request(buildApp()) + .get('/api/peer-sync/manifest'); + expect(res.status).toBe(400); + expect(integritySvc.buildLocalManifest).not.toHaveBeenCalled(); + }); + + it('400 when kind is invalid', async () => { + const res = await request(buildApp()) + .get('/api/peer-sync/manifest?kind=unknown'); + expect(res.status).toBe(400); + expect(integritySvc.buildLocalManifest).not.toHaveBeenCalled(); + }); + + it('accepts all valid subscribable kinds', async () => { + for (const kind of ['universe', 'series', 'mediaCollection']) { + integritySvc.buildLocalManifest.mockResolvedValue([]); + const res = await request(buildApp()) + .get(`/api/peer-sync/manifest?kind=${kind}`); + expect(res.status).toBe(200); + } + }); + + it('trims surrounding whitespace from kind before validation + the service call', async () => { + integritySvc.buildLocalManifest.mockResolvedValue([]); + const res = await request(buildApp()) + .get('/api/peer-sync/manifest?kind=%20universe%20'); + expect(res.status).toBe(200); + expect(integritySvc.buildLocalManifest).toHaveBeenCalledWith('universe'); + }); + }); + + describe('GET /api/peer-sync/integrity', () => { + it('200 with available:false when peer is not found', async () => { + integritySvc.getPeerIntegrity.mockResolvedValue({ + available: false, + reason: 'peer-not-found', + records: [], + }); + + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?peerId=no-such-peer&kind=mediaCollection'); + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ available: false, reason: 'peer-not-found', records: [] }); + expect(integritySvc.getPeerIntegrity).toHaveBeenCalledWith({ + peerId: 'no-such-peer', + kind: 'mediaCollection', + }); + }); + + it('200 with available:true and records when peer responds', async () => { + integritySvc.getPeerIntegrity.mockResolvedValue({ + available: true, + records: [{ id: 'col-1', name: 'My Collection', status: 'in-parity' }], + }); + + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?peerId=peer-x&kind=mediaCollection'); + expect(res.status).toBe(200); + expect(res.body.available).toBe(true); + expect(res.body.records).toHaveLength(1); + }); + + it('trims surrounding whitespace from peerId and kind before the service call', async () => { + integritySvc.getPeerIntegrity.mockResolvedValue({ available: true, records: [] }); + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?peerId=%20peer-x%20&kind=%20mediaCollection%20'); + expect(res.status).toBe(200); + // Service receives the trimmed values — otherwise ' peer-x ' silently + // fails to match the peer registry and returns peer-not-found. + expect(integritySvc.getPeerIntegrity).toHaveBeenCalledWith({ + peerId: 'peer-x', + kind: 'mediaCollection', + }); + }); + + it('400 when peerId is missing', async () => { + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?kind=mediaCollection'); + expect(res.status).toBe(400); + expect(integritySvc.getPeerIntegrity).not.toHaveBeenCalled(); + }); + + it('400 when peerId is an empty / whitespace string', async () => { + for (const peerId of ['', '%20%20']) { + const res = await request(buildApp()) + .get(`/api/peer-sync/integrity?peerId=${peerId}&kind=mediaCollection`); + expect(res.status).toBe(400); + } + expect(integritySvc.getPeerIntegrity).not.toHaveBeenCalled(); + }); + + it('400 when kind is missing', async () => { + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?peerId=peer-x'); + expect(res.status).toBe(400); + expect(integritySvc.getPeerIntegrity).not.toHaveBeenCalled(); + }); + + it('400 when kind is invalid', async () => { + const res = await request(buildApp()) + .get('/api/peer-sync/integrity?peerId=peer-x&kind=issue'); + expect(res.status).toBe(400); + expect(integritySvc.getPeerIntegrity).not.toHaveBeenCalled(); + }); + }); + + describe('POST /api/peer-sync/sync-record', () => { + it('200 with the result when body is valid', async () => { + svc.forcePushRecord.mockResolvedValue({ pushed: true, hash: 'abc' }); + const res = await request(buildApp()) + .post('/api/peer-sync/sync-record') + .send({ peerId: 'peer-a', recordKind: 'universe', recordId: 'u1' }); + expect(res.status).toBe(200); + expect(res.body.pushed).toBe(true); + expect(svc.forcePushRecord).toHaveBeenCalledWith('peer-a', 'universe', 'u1'); + }); + + it('400 when recordId is missing', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/sync-record') + .send({ peerId: 'peer-a', recordKind: 'universe' }); + expect(res.status).toBe(400); + expect(svc.forcePushRecord).not.toHaveBeenCalled(); + }); + + it('400 when recordKind is invalid', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/sync-record') + .send({ peerId: 'peer-a', recordKind: 'issue', recordId: 'i1' }); + expect(res.status).toBe(400); + expect(svc.forcePushRecord).not.toHaveBeenCalled(); + }); + + it('400 when peerId is missing', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/sync-record') + .send({ recordKind: 'universe', recordId: 'u1' }); + expect(res.status).toBe(400); + expect(svc.forcePushRecord).not.toHaveBeenCalled(); + }); + }); + + describe('POST /api/peer-sync/sync-now', () => { + it('200 with {ok:true} for a valid peerId', async () => { + svc.syncNowForPeer.mockResolvedValue({ ok: true }); + const res = await request(buildApp()) + .post('/api/peer-sync/sync-now') + .send({ peerId: 'peer-a' }); + expect(res.status).toBe(200); + expect(res.body.ok).toBe(true); + expect(svc.syncNowForPeer).toHaveBeenCalledWith('peer-a'); + }); + + it('200 with {ok:false} when peer has no instanceId', async () => { + svc.syncNowForPeer.mockResolvedValue({ ok: false }); + const res = await request(buildApp()) + .post('/api/peer-sync/sync-now') + .send({ peerId: 'ghost-peer' }); + expect(res.status).toBe(200); + expect(res.body.ok).toBe(false); + }); + + it('400 when peerId is missing', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/sync-now') + .send({}); + expect(res.status).toBe(400); + expect(svc.syncNowForPeer).not.toHaveBeenCalled(); + }); + }); + + describe('POST /api/peer-sync/pull-metadata', () => { + it('200 with backfill result for valid body', async () => { + sidecarSvc.backfillMissingSidecars.mockResolvedValue({ attempted: 3, recovered: 2 }); + const res = await request(buildApp()) + .post('/api/peer-sync/pull-metadata') + .send({ filenames: ['a.png', 'b.png', 'c.png'] }); + expect(res.status).toBe(200); + expect(res.body).toEqual({ attempted: 3, recovered: 2 }); + expect(sidecarSvc.backfillMissingSidecars).toHaveBeenCalledWith({ filenames: ['a.png', 'b.png', 'c.png'] }); + }); + + it('trims surrounding whitespace from filenames before the service call', async () => { + sidecarSvc.backfillMissingSidecars.mockResolvedValue({ attempted: 1, recovered: 1 }); + const res = await request(buildApp()) + .post('/api/peer-sync/pull-metadata') + .send({ filenames: [' a.png ', 'b.png'] }); + expect(res.status).toBe(200); + // Whitespace would otherwise yield a real-but-different name that fails + // disk lookup (confusing attempted>0, recovered=0). + expect(sidecarSvc.backfillMissingSidecars).toHaveBeenCalledWith({ filenames: ['a.png', 'b.png'] }); + }); + + it('400 when filenames is not an array', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/pull-metadata') + .send({ filenames: 'not-an-array' }); + expect(res.status).toBe(400); + expect(sidecarSvc.backfillMissingSidecars).not.toHaveBeenCalled(); + }); + + it('400 when filenames is missing', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/pull-metadata') + .send({}); + expect(res.status).toBe(400); + expect(sidecarSvc.backfillMissingSidecars).not.toHaveBeenCalled(); + }); + + it('400 when filenames exceeds 5000 entries', async () => { + const res = await request(buildApp()) + .post('/api/peer-sync/pull-metadata') + .send({ filenames: Array.from({ length: 5001 }, (_, i) => `f${i}.png`) }); + expect(res.status).toBe(400); + expect(sidecarSvc.backfillMissingSidecars).not.toHaveBeenCalled(); + }); + }); }); diff --git a/server/services/mediaCollections.js b/server/services/mediaCollections.js index f32406dd8..4f39f5237 100644 --- a/server/services/mediaCollections.js +++ b/server/services/mediaCollections.js @@ -25,7 +25,7 @@ import { PATHS, atomicWrite, readJSONFile, ensureDir } from '../lib/fileUtils.js import { createFileWriteQueue } from '../lib/fileWriteQueue.js'; import { ITEM_KIND, REF_MAX_LENGTH, itemKey } from '../lib/mediaItemKey.js'; import { sanitizeOrigin } from '../lib/sharingOrigin.js'; -import { emitRecordUpdated } from './sharing/recordEvents.js'; +import { emitRecordUpdated, emitRecordDeleted } from './sharing/recordEvents.js'; // Lazy resolution — PATHS.data may not be available at module-load time // (e.g. tests that swap it through a Proxy mock so different cases get @@ -125,10 +125,16 @@ const sanitizeCollection = (raw) => { : null; const createdAt = typeof raw.createdAt === 'string' ? raw.createdAt : new Date().toISOString(); const updatedAt = typeof raw.updatedAt === 'string' ? raw.updatedAt : createdAt; - return { id: raw.id, name, description, coverKey, universeId, seriesId, items, createdAt, updatedAt }; + const deleted = raw.deleted === true; + // When a tombstone is missing its explicit deletedAt, fall back to updatedAt + // (the most-recent timestamp we have) rather than createdAt — the deletion + // happened at or after the last edit, so createdAt would make the tombstone + // look far older than it is and skew LWW merges + the GC cutoff window. + const deletedAt = deleted && typeof raw.deletedAt === 'string' ? raw.deletedAt : (deleted ? updatedAt : null); + return { id: raw.id, name, description, coverKey, universeId, seriesId, items, createdAt, updatedAt, deleted, deletedAt }; }; -export async function listCollections() { +export async function listCollections({ includeDeleted = false } = {}) { await ensureDir(PATHS.data); const raw = await readJSONFile(statePath(), DEFAULT_STATE, { logError: false }); if (!Array.isArray(raw.collections)) return []; @@ -138,13 +144,14 @@ export async function listCollections() { const s = sanitizeCollection(c); if (!s || seen.has(s.id)) continue; seen.add(s.id); + if (!includeDeleted && s.deleted === true) continue; out.push(s); } return out; } -export async function getCollection(id) { - const all = await listCollections(); +export async function getCollection(id, { includeDeleted = false } = {}) { + const all = await listCollections({ includeDeleted }); const c = all.find((x) => x.id === id); if (!c) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); return c; @@ -160,6 +167,21 @@ const writeAll = async (collections) => { return collections; }; +// Announce a newly-created collection to the per-record peer-sync pipeline: +// emit the 'updated' event so any existing subscription pushes it, AND +// auto-subscribe every mediaCollections-enabled peer so brand-new collections +// (and their later tombstones) propagate even when that peer has universe/series +// sync disabled. Dynamic import avoids a module cycle — peerSync imports +// mergeMediaCollectionsFromSync from here, so a static import would close one. +// Call ONLY when a brand-new record was persisted — never on a find-existing +// hit, or every render would re-announce and churn the pipeline. +const announceNewCollection = (id) => { + emitRecordUpdated('mediaCollection', id); + import('./sharing/peerSync.js') + .then(({ autoSubscribeRecordToAllPeers }) => autoSubscribeRecordToAllPeers('mediaCollection', id)) + .catch(() => {}); +}; + export async function createCollection({ name, description = '' }) { // Service-layer guards mirror sanitizeCollection so a direct caller // (tests, future internal usage) can't persist a record that the next @@ -171,8 +193,8 @@ export async function createCollection({ name, description = '' }) { const trimmedDescription = typeof description === 'string' ? description.trim().slice(0, DESCRIPTION_MAX_LENGTH) : ''; - return serializeFileWrite(async () => { - const all = await listCollections(); + const created = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); const now = new Date().toISOString(); const next = { id: randomUUID(), @@ -186,6 +208,8 @@ export async function createCollection({ name, description = '' }) { await writeAll([...all, next]); return next; }); + announceNewCollection(created.id); + return created; } // Find an existing collection by case-insensitive trimmed name, else create @@ -208,10 +232,13 @@ export async function findOrCreateCollectionByName({ name, description = '', uni const trimmedDescription = typeof description === 'string' ? description.trim().slice(0, DESCRIPTION_MAX_LENGTH) : ''; - return serializeFileWrite(async () => { - const all = await listCollections(); + let createdId = null; + const result = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); const needle = trimmed.toLowerCase(); - const existing = all.find((c) => c.name.toLowerCase() === needle); + // Do not match (or reuse) tombstoned records — a deleted collection + // should not be resurrected by name. + const existing = all.find((c) => !c.deleted && c.name.toLowerCase() === needle); if (existing) { if (universeId && !existing.universeId) { // Lazy backfill so legacy "Universe: " collections gain the @@ -234,9 +261,12 @@ export async function findOrCreateCollectionByName({ name, description = '', uni createdAt: now, updatedAt: now, }; + createdId = next.id; await writeAll([...all, next]); return next; }); + if (createdId) announceNewCollection(createdId); + return result; } // Naming convention for the auto-managed universe collection. Single source @@ -302,9 +332,11 @@ export async function findOrCreateUniverseCollection({ universeId, universeName, const trimmedDescription = typeof description === 'string' ? description.trim().slice(0, DESCRIPTION_MAX_LENGTH) : ''; - return serializeFileWrite(async () => { - const all = await listCollections(); - const linked = all.find((c) => c.universeId === normalizedUniverseId); + let createdId = null; + const result = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + // Do not adopt a tombstoned universe-linked collection — deleted means gone. + const linked = all.find((c) => !c.deleted && c.universeId === normalizedUniverseId); if (linked) return linked; // No universeId match — always create fresh. The runtime intentionally // does NOT adopt a same-named unlinked collection here: it can't tell @@ -328,9 +360,15 @@ export async function findOrCreateUniverseCollection({ universeId, universeName, createdAt: now, updatedAt: now, }; + createdId = next.id; await writeAll([...all, next]); return next; }); + // Announce only when a NEW record was persisted (not a find-existing hit), so + // a mediaCollections-enabled peer receives universe-linked collections even + // with universe sync off. + if (createdId) announceNewCollection(createdId); + return result; } // Series-side mirror of the universe collection helpers above @@ -360,9 +398,11 @@ export async function findOrCreateSeriesCollection({ seriesId, seriesName, descr const trimmedDescription = typeof description === 'string' ? description.trim().slice(0, DESCRIPTION_MAX_LENGTH) : ''; - return serializeFileWrite(async () => { - const all = await listCollections(); - const linked = all.find((c) => c.seriesId === normalizedSeriesId); + let createdId = null; + const result = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + // Do not adopt a tombstoned series-linked collection — deleted means gone. + const linked = all.find((c) => !c.deleted && c.seriesId === normalizedSeriesId); if (linked) return linked; const now = new Date().toISOString(); const next = { @@ -375,16 +415,19 @@ export async function findOrCreateSeriesCollection({ seriesId, seriesName, descr createdAt: now, updatedAt: now, }; + createdId = next.id; await writeAll([...all, next]); return next; }); + if (createdId) announceNewCollection(createdId); + return result; } export async function unlinkCollectionsForSeries(seriesId) { if (typeof seriesId !== 'string' || !seriesId) return []; const needle = seriesId.slice(0, SERIES_ID_MAX); return serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const matches = all .map((c, i) => (c.seriesId === needle ? i : -1)) .filter((i) => i >= 0); @@ -405,7 +448,7 @@ export async function renameCollectionForSeries(seriesId, newSeriesName) { if (typeof seriesId !== 'string' || !seriesId) return null; const needle = seriesId.slice(0, SERIES_ID_MAX); return serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const matchIdxs = all .map((c, i) => (c.seriesId === needle ? i : -1)) .filter((i) => i >= 0); @@ -435,7 +478,7 @@ export async function unlinkCollectionsForUniverse(universeId) { if (typeof universeId !== 'string' || !universeId) return []; const needle = universeId.slice(0, UNIVERSE_ID_MAX); return serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const matches = all .map((c, i) => (c.universeId === needle ? i : -1)) .filter((i) => i >= 0); @@ -465,7 +508,7 @@ export async function renameCollectionForUniverse(universeId, newUniverseName) { if (typeof universeId !== 'string' || !universeId) return null; const needle = universeId.slice(0, UNIVERSE_ID_MAX); return serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const matchIdxs = all .map((c, i) => (c.universeId === needle ? i : -1)) .filter((i) => i >= 0); @@ -487,11 +530,12 @@ export async function renameCollectionForUniverse(universeId, newUniverseName) { } export async function updateCollection(id, patch) { - return serializeFileWrite(async () => { - const all = await listCollections(); + const merged = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); const idx = all.findIndex((c) => c.id === id); if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const cur = all[idx]; + if (cur.deleted === true) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); // Product/UI constraint: universe-linked collections own their visible // name. The name is the user-facing identity of a universe's bucket, so // renaming it independent of the universe is confusing — the supported @@ -529,16 +573,37 @@ export async function updateCollection(id, patch) { await writeAll(next); return merged; }); + // A standalone collection (no universe/series link) reaches peers ONLY via a + // direct per-record mediaCollection subscription — without this emit a + // rename/description/cover edit never propagates. For linked collections the + // universe/series emit nudges the share-bucket re-export. Emit outside the + // serialized critical section so subscribers' own reads don't deadlock the tail. + emitRecordUpdated('mediaCollection', merged.id); + if (merged.universeId) emitRecordUpdated('universe', merged.universeId); + if (merged.seriesId) emitRecordUpdated('series', merged.seriesId); + return merged; } export async function deleteCollection(id) { - const { universeId: deletedUniverseId, seriesId: deletedSeriesId } = await serializeFileWrite(async () => { - const all = await listCollections(); - const target = all.find((c) => c.id === id); - if (!target) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); - await writeAll(all.filter((c) => c.id !== id)); - return { universeId: target.universeId || null, seriesId: target.seriesId || null }; + const { universeId: deletedUniverseId, seriesId: deletedSeriesId, alreadyDeleted } = await serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + const idx = all.findIndex((c) => c.id === id); + if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); + const target = all[idx]; + // Idempotent on an already-tombstoned record: re-stamping deletedAt/ + // updatedAt would make an old tombstone look "newer" to a peer's LWW (and + // re-emitting churns the sync pipeline). Return without rewriting or + // re-emitting. + if (target.deleted === true) return { universeId: null, seriesId: null, alreadyDeleted: true }; + const now = new Date().toISOString(); + const next = [...all]; + // Clear coverKey too — with items emptied it would otherwise dangle + // (point at a non-existent item) and leak into the tombstone's wire payload. + next[idx] = { ...target, deleted: true, deletedAt: now, updatedAt: now, items: [], coverKey: null, universeId: null, seriesId: null }; + await writeAll(next); + return { universeId: target.universeId || null, seriesId: target.seriesId || null, alreadyDeleted: false }; }); + if (alreadyDeleted) return { id }; // Mirror addItem/removeItem/bulkUpdateCollectionItems — universe-linked // shares need to know membership changed so the subscriber doesn't keep // publishing the deleted collection's contents until an unrelated edit @@ -546,6 +611,7 @@ export async function deleteCollection(id) { // own reads don't deadlock the tail. if (deletedUniverseId) emitRecordUpdated('universe', deletedUniverseId); if (deletedSeriesId) emitRecordUpdated('series', deletedSeriesId); + emitRecordDeleted('mediaCollection', id); return { id }; } @@ -579,10 +645,13 @@ const validateItemInput = (item) => { export async function addItem(id, item) { const { kind, ref } = validateItemInput(item); const merged = await serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const idx = all.findIndex((c) => c.id === id); if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const cur = all[idx]; + // A soft-deleted collection behaves as not-found for mutations (matches + // updateCollection) — never resurrect a tombstone or churn its timestamps. + if (cur.deleted === true) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const key = `${kind}:${ref}`; if (cur.items.find((it) => itemKey(it) === key)) { throw makeErr(`Item already in collection: ${key}`, ERR_DUPLICATE); @@ -602,6 +671,9 @@ export async function addItem(id, item) { }); // Emit outside the serialized critical section — subscribers may issue // their own collection reads and we don't want to deadlock the tail. + // mediaCollection covers standalone direct subscriptions; universe/series + // nudge the bundled share-bucket re-export for linked collections. + emitRecordUpdated('mediaCollection', merged.id); if (merged.universeId) emitRecordUpdated('universe', merged.universeId); if (merged.seriesId) emitRecordUpdated('series', merged.seriesId); return merged; @@ -639,10 +711,13 @@ export async function bulkUpdateCollectionItems(id, { add = [], remove = [] } = } const result = await serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const idx = all.findIndex((c) => c.id === id); if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const cur = all[idx]; + // A soft-deleted collection behaves as not-found for mutations (matches + // updateCollection) — never resurrect a tombstone or churn its timestamps. + if (cur.deleted === true) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const removeSet = new Set(remove); const remainingItems = cur.items.filter((it) => !removeSet.has(itemKey(it))); @@ -683,6 +758,7 @@ export async function bulkUpdateCollectionItems(id, { add = [], remove = [] } = return { collection: merged, added: additions.length, removed }; }); if (result.added || result.removed) { + emitRecordUpdated('mediaCollection', result.collection.id); if (result.collection.universeId) emitRecordUpdated('universe', result.collection.universeId); if (result.collection.seriesId) emitRecordUpdated('series', result.collection.seriesId); } @@ -691,10 +767,13 @@ export async function bulkUpdateCollectionItems(id, { add = [], remove = [] } = export async function removeItem(id, key) { const merged = await serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const idx = all.findIndex((c) => c.id === id); if (idx < 0) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const cur = all[idx]; + // A soft-deleted collection behaves as not-found for mutations (matches + // updateCollection) — never resurrect a tombstone or churn its timestamps. + if (cur.deleted === true) throw makeErr(`Collection not found: ${id}`, ERR_NOT_FOUND); const before = cur.items.length; const items = cur.items.filter((it) => itemKey(it) !== key); if (items.length === before) throw makeErr(`Item not in collection: ${key}`, ERR_NOT_FOUND); @@ -712,11 +791,29 @@ export async function removeItem(id, key) { await writeAll(next); return updated; }); + emitRecordUpdated('mediaCollection', merged.id); if (merged.universeId) emitRecordUpdated('universe', merged.universeId); if (merged.seriesId) emitRecordUpdated('series', merged.seriesId); return merged; } +// Hard-remove tombstoned collections whose deletedAt is older than the cutoff. +// Called by tombstoneGc once every subscribed peer has acked the deletion. +export async function pruneTombstonedCollections(olderThanMs) { + if (!Number.isFinite(olderThanMs)) return { pruned: 0 }; + return serializeFileWrite(async () => { + const all = await listCollections({ includeDeleted: true }); + const keep = all.filter((c) => { + if (c.deleted !== true) return true; + const ms = Date.parse(c.deletedAt || ''); + return !(Number.isFinite(ms) && ms < olderThanMs); + }); + if (keep.length === all.length) return { pruned: 0 }; + await writeAll(keep); + return { pruned: all.length - keep.length }; + }); +} + /** * Merge an incoming list of collections from a peer (snapshot sync OR the * per-record push payload's `linkedCollection` field). Per-collection @@ -738,7 +835,7 @@ export async function removeItem(id, key) { export async function mergeMediaCollectionsFromSync(remoteCollections) { if (!Array.isArray(remoteCollections)) return { applied: false, count: 0 }; return serializeFileWrite(async () => { - const all = await listCollections(); + const all = await listCollections({ includeDeleted: true }); const localById = new Map(all.map((c) => [c.id, c])); let changed = 0; for (const remote of remoteCollections) { @@ -767,19 +864,22 @@ export async function mergeMediaCollectionsFromSync(remoteCollections) { // Cover key: only adopt the scalar source's coverKey if it points at an // item that survives the union — otherwise sanitizeCollection on next // read would drop it back to null and we'd churn updatedAt forever. + const scalarDeleted = scalarSource.deleted === true; const presentKeys = new Set(mergedItems.map(itemKey)); - const coverKey = scalarSource.coverKey && presentKeys.has(scalarSource.coverKey) + const coverKey = !scalarDeleted && scalarSource.coverKey && presentKeys.has(scalarSource.coverKey) ? scalarSource.coverKey : null; const next = { ...local, name: scalarSource.name, description: scalarSource.description, - coverKey, - universeId: scalarSource.universeId, - seriesId: scalarSource.seriesId, - items: mergedItems, + coverKey: scalarDeleted ? null : coverKey, + universeId: scalarDeleted ? null : scalarSource.universeId, + seriesId: scalarDeleted ? null : scalarSource.seriesId, + items: scalarDeleted ? [] : mergedItems, updatedAt: remoteWins ? remoteTs : localTs, + deleted: scalarDeleted, + deletedAt: scalarDeleted ? (scalarSource.deletedAt || (remoteWins ? remoteTs : localTs)) : null, }; if (collectionsEqual(local, next)) continue; localById.set(sanitized.id, next); diff --git a/server/services/mediaCollections.test.js b/server/services/mediaCollections.test.js index dd1abe491..8fe3730e1 100644 --- a/server/services/mediaCollections.test.js +++ b/server/services/mediaCollections.test.js @@ -1,4 +1,5 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { mockNoPeerSync } from '../lib/mockPathsDataRoot.js'; const fileStore = new Map(); @@ -10,6 +11,11 @@ tryReadFile: vi.fn().mockResolvedValue(null), readJSONFile: vi.fn(async (path, fallback) => fileStore.has(path) ? fileStore.get(path) : fallback), })); +// Suppress the fire-and-forget dynamic import so tests don't load the real +// peerSync module graph (which reads the live peer registry and imports +// universe/series services). +vi.mock('./sharing/peerSync.js', () => mockNoPeerSync()); + let uuidCounter = 0; vi.mock('crypto', async () => { const actual = await vi.importActual('crypto'); @@ -92,10 +98,81 @@ describe('mediaCollections service', () => { .rejects.toMatchObject({ code: svc.ERR_VALIDATION }); }); - it('deleteCollection removes the entry', async () => { + it('deleteCollection soft-deletes: absent from live list, present with deleted===true in includeDeleted list', async () => { const c = await svc.createCollection({ name: 'A' }); await svc.deleteCollection(c.id); expect(await svc.listCollections()).toEqual([]); + const all = await svc.listCollections({ includeDeleted: true }); + expect(all).toHaveLength(1); + expect(all[0].deleted).toBe(true); + expect(all[0].id).toBe(c.id); + }); + + it('deleteCollection clears coverKey on the persisted tombstone (no dangling cover in the wire record)', async () => { + const c = await svc.createCollection({ name: 'WithCover' }); + await svc.addItem(c.id, { kind: 'image', ref: 'cover.png' }); + await svc.updateCollection(c.id, { coverKey: 'image:cover.png' }); + await svc.deleteCollection(c.id); + // Assert on the PERSISTED record — listCollections' sanitizer would null a + // dangling coverKey on read regardless, so inspect storage directly. + const stored = fileStore.get('/mock/data/media-collections.json').collections.find((x) => x.id === c.id); + expect(stored.deleted).toBe(true); + expect(stored.items).toEqual([]); + expect(stored.coverKey).toBeNull(); + }); + + it('deleteCollection emits recordDeleted for mediaCollection and receivable via recordEvents', async () => { + const { recordEvents } = await import('./sharing/recordEvents.js'); + const deletedEvts = []; + const handler = (evt) => deletedEvts.push(evt); + recordEvents.on('deleted', handler); + try { + const c = await svc.createCollection({ name: 'B' }); + await svc.deleteCollection(c.id); + expect(deletedEvts).toContainEqual(expect.objectContaining({ recordKind: 'mediaCollection', recordId: c.id })); + } finally { + recordEvents.off('deleted', handler); + } + }); + + it('updateCollection throws ERR_NOT_FOUND on a soft-deleted collection', async () => { + const c = await svc.createCollection({ name: 'Live' }); + await svc.deleteCollection(c.id); + await expect(svc.updateCollection(c.id, { name: 'Revived' })) + .rejects.toMatchObject({ code: svc.ERR_NOT_FOUND }); + }); + + it('item mutators (addItem/removeItem/bulkUpdateCollectionItems) throw ERR_NOT_FOUND on a tombstone', async () => { + const c = await svc.createCollection({ name: 'Live' }); + await svc.addItem(c.id, { kind: 'image', ref: 'keep.png' }); + await svc.deleteCollection(c.id); + // All three behave as not-found after soft-delete (no tombstone resurrection). + await expect(svc.addItem(c.id, { kind: 'image', ref: 'new.png' })) + .rejects.toMatchObject({ code: svc.ERR_NOT_FOUND }); + await expect(svc.removeItem(c.id, 'image:keep.png')) + .rejects.toMatchObject({ code: svc.ERR_NOT_FOUND }); + await expect(svc.bulkUpdateCollectionItems(c.id, { add: [{ kind: 'image', ref: 'b.png' }] })) + .rejects.toMatchObject({ code: svc.ERR_NOT_FOUND }); + }); + + it('deleteCollection is idempotent on an already-tombstoned record (no re-stamp, no re-emit)', async () => { + const { recordEvents } = await import('./sharing/recordEvents.js'); + const c = await svc.createCollection({ name: 'DoubleDelete' }); + await svc.deleteCollection(c.id); + const firstDeletedAt = (await svc.listCollections({ includeDeleted: true })).find((x) => x.id === c.id).deletedAt; + await new Promise((r) => setTimeout(r, 5)); // ensure a later timestamp WOULD differ if re-stamped + const deletedEvts = []; + const handler = (evt) => deletedEvts.push(evt); + recordEvents.on('deleted', handler); + try { + const res = await svc.deleteCollection(c.id); + expect(res).toEqual({ id: c.id }); + const afterSecond = (await svc.listCollections({ includeDeleted: true })).find((x) => x.id === c.id); + expect(afterSecond.deletedAt).toBe(firstDeletedAt); // not re-stamped + expect(deletedEvts).toEqual([]); // not re-emitted + } finally { + recordEvents.off('deleted', handler); + } }); it('deleteCollection emits recordUpdated on the universe when the deleted collection was linked', async () => { @@ -946,6 +1023,104 @@ describe('mergeMediaCollectionsFromSync', () => { expect(merged.description).toBe('local'); }); + it('preserves deleted + deletedAt through sync merge', async () => { + await svc.mergeMediaCollectionsFromSync([{ + id: 'c1', name: 'C1', items: [], + deleted: true, deletedAt: '2026-05-23T00:00:00.000Z', + updatedAt: '2026-05-23T00:00:00.000Z', + }]); + const live = await svc.listCollections(); + expect(live.find((c) => c.id === 'c1')).toBeUndefined(); + const all = await svc.listCollections({ includeDeleted: true }); + const c = all.find((x) => x.id === 'c1'); + expect(c?.deleted).toBe(true); + expect(c?.deletedAt).toBe('2026-05-23T00:00:00.000Z'); + }); + + it('a newer remote tombstone deletes a local live collection', async () => { + const c = await svc.createCollection({ name: 'WillDie' }); + const tombstone = { + id: c.id, + name: 'WillDie', + description: '', + coverKey: null, + universeId: null, + seriesId: null, + items: [], + createdAt: c.createdAt, + updatedAt: '2099-01-01T00:00:00.000Z', + deleted: true, + deletedAt: '2099-01-01T00:00:00.000Z', + }; + await svc.mergeMediaCollectionsFromSync([tombstone]); + expect(await svc.listCollections()).toEqual([]); + const all = await svc.listCollections({ includeDeleted: true }); + const found = all.find((x) => x.id === c.id); + expect(found?.deleted).toBe(true); + }); + + it('an older remote tombstone does NOT delete a newer local collection', async () => { + const c = await svc.createCollection({ name: 'Survivor' }); + const tombstone = { + id: c.id, + name: 'Survivor', + description: '', + coverKey: null, + universeId: null, + seriesId: null, + items: [], + createdAt: c.createdAt, + updatedAt: '2000-01-01T00:00:00.000Z', + deleted: true, + deletedAt: '2000-01-01T00:00:00.000Z', + }; + await svc.mergeMediaCollectionsFromSync([tombstone]); + const live = await svc.listCollections(); + expect(live.find((x) => x.id === c.id)).toBeTruthy(); + expect(live.find((x) => x.id === c.id)?.deleted).toBeFalsy(); + }); + + it('a remote tombstone for an unknown id is recorded as a tombstone (no resurrection)', async () => { + const tombstone = { + id: 'ghost-id', + name: 'Ghost', + description: '', + coverKey: null, + universeId: null, + seriesId: null, + items: [], + createdAt: '2026-05-23T00:00:00.000Z', + updatedAt: '2026-05-23T00:00:00.000Z', + deleted: true, + deletedAt: '2026-05-23T00:00:00.000Z', + }; + await svc.mergeMediaCollectionsFromSync([tombstone]); + expect(await svc.listCollections()).toEqual([]); + const all = await svc.listCollections({ includeDeleted: true }); + const found = all.find((x) => x.id === 'ghost-id'); + expect(found?.deleted).toBe(true); + }); + + it('a tombstone missing deletedAt falls back to updatedAt (not the older createdAt)', async () => { + // A hand-edited / legacy tombstone may carry deleted:true with no explicit + // deletedAt. The effective deletion time should align with the most-recent + // timestamp (updatedAt), not createdAt — otherwise LWW + GC see it as far + // older than it really is. + fileStore.set('/mock/data/media-collections.json', { + collections: [{ + id: 'c1', name: 'Gone', description: '', coverKey: null, + universeId: null, seriesId: null, items: [], + createdAt: '2026-01-01T00:00:00.000Z', + updatedAt: '2026-05-22T03:00:00.000Z', + deleted: true, + // deletedAt intentionally omitted + }], + }); + const [c] = await svc.listCollections({ includeDeleted: true }); + expect(c.deleted).toBe(true); + expect(c.deletedAt).toBe('2026-05-22T03:00:00.000Z'); + }); + it('compares addedAt as parsed milliseconds (not lexicographic) when picking the earlier item', async () => { // sanitizeItem accepts any Date.parse-able string, not strictly ISO-8601. // Lexicographic compare would order "05/22/2026 ..." AFTER "2026-..." (the @@ -972,3 +1147,172 @@ describe('mergeMediaCollectionsFromSync', () => { expect(Date.parse(merged.items[0].addedAt)).toBe(Date.parse('05/22/2026 08:00:00 UTC')); }); }); + +describe('createCollection — event emission', () => { + it('emits a mediaCollection updated event with the created record id', async () => { + const { recordEvents } = await import('./sharing/recordEvents.js'); + const updatedEvts = []; + const handler = (evt) => updatedEvts.push(evt); + recordEvents.on('updated', handler); + try { + const c = await svc.createCollection({ name: 'New' }); + expect(updatedEvts).toContainEqual( + expect.objectContaining({ recordKind: 'mediaCollection', recordId: c.id }), + ); + } finally { + recordEvents.off('updated', handler); + } + }); +}); + +describe('findOrCreate* — announces new collections to the sync pipeline', () => { + // A newly-created universe/series-linked (or named) collection must emit a + // mediaCollection 'updated' event so a peer with mediaCollections syncing + // enabled (but universe/series sync off) still receives it via per-record sync. + // It must NOT re-announce on a find-existing hit (would churn every render). + const collectMediaUpdateIds = async (fn) => { + const { recordEvents } = await import('./sharing/recordEvents.js'); + const ids = []; + const handler = (evt) => { if (evt.recordKind === 'mediaCollection') ids.push(evt.recordId); }; + recordEvents.on('updated', handler); + try { await fn(); } finally { recordEvents.off('updated', handler); } + return ids; + }; + + it('findOrCreateUniverseCollection announces on create, stays quiet on find-existing', async () => { + const created = await collectMediaUpdateIds(() => + svc.findOrCreateUniverseCollection({ universeId: 'u1', universeName: 'Iron Veil' })); + const c = await svc.findCollectionByUniverseId('u1'); + expect(created).toEqual([c.id]); + const again = await collectMediaUpdateIds(() => + svc.findOrCreateUniverseCollection({ universeId: 'u1', universeName: 'Iron Veil' })); + expect(again).toEqual([]); + }); + + it('findOrCreateSeriesCollection announces on create, stays quiet on find-existing', async () => { + const created = await collectMediaUpdateIds(() => + svc.findOrCreateSeriesCollection({ seriesId: 's1', seriesName: 'Salt Run' })); + expect(created).toHaveLength(1); + const again = await collectMediaUpdateIds(() => + svc.findOrCreateSeriesCollection({ seriesId: 's1', seriesName: 'Salt Run' })); + expect(again).toEqual([]); + }); + + it('findOrCreateCollectionByName announces on create, stays quiet on find-existing', async () => { + const created = await collectMediaUpdateIds(() => + svc.findOrCreateCollectionByName({ name: 'Loose Bucket' })); + expect(created).toHaveLength(1); + const again = await collectMediaUpdateIds(() => + svc.findOrCreateCollectionByName({ name: 'Loose Bucket' })); + expect(again).toEqual([]); + }); +}); + +describe('mutators — mediaCollection updated emission (standalone per-record sync)', () => { + // A standalone collection (no universe/series link) reaches a directly- + // subscribed peer ONLY through the per-record mediaCollection push pipeline, + // so every content mutator must emit a mediaCollection 'updated' event or the + // edit silently never propagates. + const collectMediaUpdates = async (fn) => { + const { recordEvents } = await import('./sharing/recordEvents.js'); + const ids = []; + const handler = (evt) => { if (evt.recordKind === 'mediaCollection') ids.push(evt.recordId); }; + recordEvents.on('updated', handler); + try { + await fn(); + } finally { + recordEvents.off('updated', handler); + } + return ids; + }; + + it('updateCollection emits a mediaCollection updated event', async () => { + const c = await svc.createCollection({ name: 'Standalone' }); + const ids = await collectMediaUpdates(() => svc.updateCollection(c.id, { description: 'changed' })); + expect(ids).toContain(c.id); + }); + + it('addItem emits a mediaCollection updated event', async () => { + const c = await svc.createCollection({ name: 'Standalone' }); + const ids = await collectMediaUpdates(() => svc.addItem(c.id, { kind: 'image', ref: 'x.png' })); + expect(ids).toContain(c.id); + }); + + it('removeItem emits a mediaCollection updated event', async () => { + const c = await svc.createCollection({ name: 'Standalone' }); + await svc.addItem(c.id, { kind: 'image', ref: 'x.png' }); + const ids = await collectMediaUpdates(() => svc.removeItem(c.id, 'image:x.png')); + expect(ids).toContain(c.id); + }); + + it('bulkUpdateCollectionItems emits a mediaCollection updated event when items change', async () => { + const c = await svc.createCollection({ name: 'Standalone' }); + const ids = await collectMediaUpdates(() => + svc.bulkUpdateCollectionItems(c.id, { add: [{ kind: 'image', ref: 'y.png' }] }), + ); + expect(ids).toContain(c.id); + }); +}); + +describe('pruneTombstonedCollections', () => { + beforeEach(() => { + fileStore.clear(); + uuidCounter = 0; + }); + + it('prunes tombstoned collections older than the cutoff and returns the count', async () => { + // Create a live collection and soft-delete it with an old timestamp by + // injecting the tombstone directly into the file store. + const c = await svc.createCollection({ name: 'ToDelete' }); + const oldTs = new Date(Date.now() - 48 * 60 * 60 * 1000).toISOString(); + // Overwrite the file store with a tombstone carrying an old deletedAt. + const { atomicWrite, readJSONFile } = await import('../lib/fileUtils.js'); + const path = '/mock/data/media-collections.json'; + const current = await readJSONFile(path, { collections: [] }); + const withTombstone = current.collections.map((col) => + col.id === c.id + ? { ...col, deleted: true, deletedAt: oldTs, updatedAt: oldTs, items: [] } + : col, + ); + await atomicWrite(path, { collections: withTombstone }); + + const result = await svc.pruneTombstonedCollections(Date.now()); + expect(result).toEqual({ pruned: 1 }); + expect(await svc.listCollections({ includeDeleted: true })).toHaveLength(0); + }); + + it('does NOT prune a live collection', async () => { + await svc.createCollection({ name: 'Live' }); + const result = await svc.pruneTombstonedCollections(Date.now()); + expect(result).toEqual({ pruned: 0 }); + expect(await svc.listCollections()).toHaveLength(1); + }); + + it('does NOT prune a tombstone newer than the cutoff', async () => { + const c = await svc.createCollection({ name: 'RecentDelete' }); + // Soft-delete with a future timestamp (simulating a delete that just happened). + const { atomicWrite, readJSONFile } = await import('../lib/fileUtils.js'); + const path = '/mock/data/media-collections.json'; + const current = await readJSONFile(path, { collections: [] }); + const futureTs = new Date(Date.now() + 60 * 1000).toISOString(); + const withTombstone = current.collections.map((col) => + col.id === c.id + ? { ...col, deleted: true, deletedAt: futureTs, updatedAt: futureTs, items: [] } + : col, + ); + await atomicWrite(path, { collections: withTombstone }); + + // Cut-off is now; the tombstone's deletedAt is in the future → not pruned. + const result = await svc.pruneTombstonedCollections(Date.now()); + expect(result).toEqual({ pruned: 0 }); + const all = await svc.listCollections({ includeDeleted: true }); + expect(all).toHaveLength(1); + expect(all[0].deleted).toBe(true); + }); + + it('returns { pruned: 0 } without touching the file when cutoff is not a finite number', async () => { + await svc.createCollection({ name: 'A' }); + expect(await svc.pruneTombstonedCollections(NaN)).toEqual({ pruned: 0 }); + expect(await svc.pruneTombstonedCollections(Infinity)).toEqual({ pruned: 0 }); + }); +}); diff --git a/server/services/sharing/buckets.js b/server/services/sharing/buckets.js index fb244c036..f7d3fe0b5 100644 --- a/server/services/sharing/buckets.js +++ b/server/services/sharing/buckets.js @@ -11,7 +11,7 @@ */ import { randomUUID } from 'crypto'; -import { join } from 'path'; +import { join, basename } from 'path'; import { access, constants, stat } from 'fs/promises'; import { PATHS, atomicWrite, readJSONFile, ensureDir } from '../../lib/fileUtils.js'; import { isStr, trimTo } from '../../lib/storyBible.js'; @@ -106,6 +106,26 @@ export function bucketBlobSidecarPath(bucketPath, hash) { export function bucketBlobIndexPath(bucketPath) { return join(bucketBlobsDir(bucketPath), '.index.json'); } export function imageSidecarName(filename) { return filename.replace(IMAGE_EXT_RE, '') + '.metadata.json'; } +/** + * Returns the filename if it's safe to use as a path segment under an asset + * directory, otherwise null. Rejects path separators, parent-directory + * tokens, and any value that doesn't match its own basename — the canonical + * traversal guard for inbound/peer-supplied asset filenames before they hit a + * `join(dir, name)` FS op. Shared by the peer-sync diff/pull paths + * (`peerSync.js`, `sidecarSync.js`) so every callsite scrubs identically. + */ +export function sanitizeAssetFilename(name) { + if (typeof name !== 'string' || !name) return null; + // Reject separators and exact parent-directory segments (`.` / `..` + // as the whole basename). A basename like `my..render.png` is + // legitimate (the gallery filename validator permits `..` inside a + // basename) — only the path-segment forms are traversal. + if (name.includes('/') || name.includes('\\')) return null; + if (name === '.' || name === '..') return null; + if (basename(name) !== name) return null; + return name; +} + /** Lay out the canonical bucket structure (idempotent). */ export async function ensureBucketLayout(bucket) { const base = bucket.path; diff --git a/server/services/sharing/buckets.test.js b/server/services/sharing/buckets.test.js index 5095de118..cf053939e 100644 --- a/server/services/sharing/buckets.test.js +++ b/server/services/sharing/buckets.test.js @@ -97,4 +97,28 @@ describe('sharing/buckets', () => { rmSync(b, { recursive: true, force: true }); } }); + + describe('sanitizeAssetFilename', () => { + it('returns the name for a safe bare basename', () => { + expect(buckets.sanitizeAssetFilename('abc-123.png')).toBe('abc-123.png'); + // `..` inside a basename is legitimate (gallery validator permits it). + expect(buckets.sanitizeAssetFilename('my..render.png')).toBe('my..render.png'); + }); + + it('rejects path separators, parent-dir tokens, and non-basename values', () => { + expect(buckets.sanitizeAssetFilename('../../etc/passwd')).toBeNull(); + expect(buckets.sanitizeAssetFilename('..\\windows\\system32')).toBeNull(); + expect(buckets.sanitizeAssetFilename('sub/dir/asset.png')).toBeNull(); + expect(buckets.sanitizeAssetFilename('/etc/hosts')).toBeNull(); + expect(buckets.sanitizeAssetFilename('.')).toBeNull(); + expect(buckets.sanitizeAssetFilename('..')).toBeNull(); + }); + + it('rejects non-string and empty inputs', () => { + expect(buckets.sanitizeAssetFilename('')).toBeNull(); + expect(buckets.sanitizeAssetFilename(null)).toBeNull(); + expect(buckets.sanitizeAssetFilename(undefined)).toBeNull(); + expect(buckets.sanitizeAssetFilename(42)).toBeNull(); + }); + }); }); diff --git a/server/services/sharing/index.js b/server/services/sharing/index.js index 779636e66..cad4e7a2c 100644 --- a/server/services/sharing/index.js +++ b/server/services/sharing/index.js @@ -17,6 +17,7 @@ import { initAnnotationsSync } from './annotationsSync.js'; export { sharingEvents } from './importer.js'; export { attachWatcher, detachWatcher, listAttachedWatchers }; +export { pullSidecarForImage, backfillMissingSidecars } from './sidecarSync.js'; let initialized = false; let io = null; diff --git a/server/services/sharing/integrity.js b/server/services/sharing/integrity.js new file mode 100644 index 000000000..332636d28 --- /dev/null +++ b/server/services/sharing/integrity.js @@ -0,0 +1,98 @@ +/** + * Per-category sync integrity: build a local manifest and compare it against + * a remote peer's manifest to surface records that are out-of-parity. + * + * `buildLocalManifest(kind)` — returns one row per record with id, name, + * updatedAt, deleted, and sorted sha256 asset hashes. Tombstones are + * included so that deletes diff correctly against the peer. + * + * `getPeerIntegrity({ peerId, kind })` — fetches the peer's manifest via + * GET /api/peer-sync/manifest, then runs the pure diff. Returns + * `{ available: bool, reason?, records: [...] }`. + */ + +import { computeRecordIntegrity } from '../../lib/syncIntegrity.js'; +import { listCollections } from '../mediaCollections.js'; +import { listUniverses } from '../universeBuilder.js'; +import { listSeries } from '../pipeline/series.js'; +import { getPeers } from '../instances.js'; +import { assetShaListForRecord } from './peerSync.js'; +import { peerBaseUrl } from '../../lib/peerUrl.js'; +import { peerFetch } from '../../lib/peerHttpClient.js'; + +async function recordsForKind(kind) { + if (kind === 'mediaCollection') return listCollections({ includeDeleted: true }); + if (kind === 'universe') return listUniverses({ includeDeleted: true }); + if (kind === 'series') return listSeries({ includeDeleted: true }); + return []; +} + +/** + * Build a local manifest for the given kind. + * One row per record: `{ id, name, updatedAt, deleted, assetHashes }`. + * Includes tombstoned records so deletes surface correctly in the diff. + * + * @param {'universe'|'series'|'mediaCollection'} kind + * @returns {Promise} + */ +export async function buildLocalManifest(kind) { + const records = await recordsForKind(kind); + // Hash records SEQUENTIALLY (for...of, not Promise.all(map)) so a large + // library can't fan out an unbounded number of concurrent file-hash reads and + // spike CPU/disk. Each assetShaListForRecord already reads many files; doing + // every record's pass at once would multiply that. + const out = []; + for (const r of records) { + const deleted = r.deleted === true; + out.push({ + id: r.id, + name: r.name, + updatedAt: r.updatedAt, + deleted, + // Tombstones never need asset hashes: computeRecordIntegrity only + // compares assetHashes when BOTH sides are live (and drops + // deleted-vs-deleted pairs entirely). Hashing a deleted record's + // still-on-disk assets is pure wasted file I/O. + assetHashes: deleted ? [] : await assetShaListForRecord(kind, r), + }); + } + return out; +} + +/** + * Fetch the peer's manifest for `kind`, run the local-vs-remote diff, and + * return the classified record list. + * + * @param {{ peerId: string, kind: string }} opts + * @returns {Promise<{ available: boolean, reason?: string, records: Array }>} + */ +export async function getPeerIntegrity({ peerId, kind }) { + const peers = await getPeers().catch(() => []); + const peer = peers.find((p) => p.instanceId === peerId) || null; + + if (!peer) return { available: false, reason: 'peer-not-found', records: [] }; + + const res = await peerFetch( + `${peerBaseUrl(peer)}/api/peer-sync/manifest?kind=${encodeURIComponent(kind)}`, + ).catch(() => null); + + // Distinguish a network failure (peerFetch threw / returned null) from a 404. + // A null result means the peer is offline/unreachable — NOT that it's running + // an older PortOS without the /manifest route. Lumping them together would + // tell the user "peer too old, upgrade it" when the peer is simply down. + if (!res) return { available: false, reason: 'peer-unreachable', records: [] }; + if (res.status === 404) return { available: false, reason: 'peer-too-old', records: [] }; + if (!res.ok) return { available: false, reason: 'fetch-failed', records: [] }; + + const body = await res.json().catch(() => null); + // The peer response is untrusted — computeRecordIntegrity assumes every entry + // is a non-null object with a string `id` (it reads `r.id` and keys a Map on + // it). A hostile/malformed manifest (nulls, scalars, id-less objects) would + // otherwise throw and 500 this endpoint, so filter to well-formed rows first. + const remote = (Array.isArray(body?.records) ? body.records : []).filter( + (r) => r && typeof r === 'object' && typeof r.id === 'string' && r.id, + ); + + const local = await buildLocalManifest(kind); + return { available: true, records: computeRecordIntegrity(local, remote) }; +} diff --git a/server/services/sharing/integrity.test.js b/server/services/sharing/integrity.test.js new file mode 100644 index 000000000..2075ec844 --- /dev/null +++ b/server/services/sharing/integrity.test.js @@ -0,0 +1,267 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +// Mock peerFetch so no network calls go out. +vi.mock('../../lib/peerHttpClient.js', async () => ({ + peerFetch: vi.fn(), + peerSocketOptions: {}, +})); + +// Mock peerUrl for deterministic base URL generation. +vi.mock('../../lib/peerUrl.js', async () => ({ + peerBaseUrl: vi.fn((peer) => `http://${peer.instanceId}.test:5555`), +})); + +// Mock instances.js — peer list is controlled per test. +vi.mock('../instances.js', async () => ({ + UNKNOWN_INSTANCE_ID: 'unknown', + getInstanceId: vi.fn().mockResolvedValue('local-instance'), + getPeers: vi.fn(), +})); + +// Mock peerSync.js — we test integrity.js logic, not asset hashing. +vi.mock('./peerSync.js', async () => ({ + assetShaListForRecord: vi.fn().mockResolvedValue([]), + PEER_SUBSCRIBABLE_KINDS: ['universe', 'series', 'mediaCollection'], +})); + +// Mock mediaCollections, universeBuilder, series so tests control the record set. +vi.mock('../mediaCollections.js', async () => ({ + listCollections: vi.fn(), +})); +vi.mock('../universeBuilder.js', async () => ({ + listUniverses: vi.fn(), +})); +vi.mock('../pipeline/series.js', async () => ({ + listSeries: vi.fn(), +})); + +import { peerFetch } from '../../lib/peerHttpClient.js'; +import { getPeers } from '../instances.js'; +import { assetShaListForRecord } from './peerSync.js'; +import { listCollections } from '../mediaCollections.js'; +import { listUniverses } from '../universeBuilder.js'; +import { listSeries } from '../pipeline/series.js'; +import { buildLocalManifest, getPeerIntegrity } from './integrity.js'; +import { INTEGRITY_STATUS } from '../../lib/syncIntegrity.js'; + +const makeCollection = (overrides = {}) => ({ + id: 'col-1', + name: 'My Collection', + updatedAt: '2026-05-23T00:00:00.000Z', + deleted: false, + items: [], + ...overrides, +}); + +const makePeer = (id = 'peer-x') => ({ instanceId: id, name: `Peer ${id}` }); + +beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(getPeers).mockResolvedValue([]); + vi.mocked(listCollections).mockResolvedValue([]); + vi.mocked(listUniverses).mockResolvedValue([]); + vi.mocked(listSeries).mockResolvedValue([]); +}); + +describe('buildLocalManifest', () => { + it('returns an empty array when no records exist', async () => { + vi.mocked(listCollections).mockResolvedValue([]); + const result = await buildLocalManifest('mediaCollection'); + expect(result).toEqual([]); + }); + + it('returns one row per collection with the correct shape', async () => { + const col = makeCollection(); + vi.mocked(listCollections).mockResolvedValue([col]); + vi.mocked(assetShaListForRecord).mockResolvedValue(['aabb', 'ccdd']); + + const result = await buildLocalManifest('mediaCollection'); + expect(result).toHaveLength(1); + expect(result[0]).toMatchObject({ + id: 'col-1', + name: 'My Collection', + updatedAt: '2026-05-23T00:00:00.000Z', + deleted: false, + assetHashes: ['aabb', 'ccdd'], + }); + expect(assetShaListForRecord).toHaveBeenCalledWith('mediaCollection', col); + }); + + it('marks deleted collections as deleted:true and skips asset hashing for them', async () => { + const col = makeCollection({ deleted: true }); + vi.mocked(listCollections).mockResolvedValue([col]); + + const result = await buildLocalManifest('mediaCollection'); + expect(result[0].deleted).toBe(true); + // Tombstones are never hashed — computeRecordIntegrity ignores assets for + // deleted records, so the file I/O would be wasted. + expect(result[0].assetHashes).toEqual([]); + expect(assetShaListForRecord).not.toHaveBeenCalled(); + }); + + it('passes includeDeleted:true so listCollections returns tombstones', async () => { + await buildLocalManifest('mediaCollection'); + expect(listCollections).toHaveBeenCalledWith({ includeDeleted: true }); + }); + + it('passes includeDeleted:true to listUniverses', async () => { + await buildLocalManifest('universe'); + expect(listUniverses).toHaveBeenCalledWith({ includeDeleted: true }); + }); + + it('passes includeDeleted:true to listSeries', async () => { + await buildLocalManifest('series'); + expect(listSeries).toHaveBeenCalledWith({ includeDeleted: true }); + }); + + it('returns empty array for an unrecognised kind', async () => { + const result = await buildLocalManifest('unknown-kind'); + expect(result).toEqual([]); + }); + + it('handles multiple collections and calls assetShaListForRecord for each', async () => { + const col1 = makeCollection({ id: 'c1', name: 'C1' }); + const col2 = makeCollection({ id: 'c2', name: 'C2' }); + vi.mocked(listCollections).mockResolvedValue([col1, col2]); + vi.mocked(assetShaListForRecord) + .mockResolvedValueOnce(['hash-a']) + .mockResolvedValueOnce(['hash-b']); + + const result = await buildLocalManifest('mediaCollection'); + expect(result).toHaveLength(2); + expect(result.find((r) => r.id === 'c1').assetHashes).toEqual(['hash-a']); + expect(result.find((r) => r.id === 'c2').assetHashes).toEqual(['hash-b']); + }); +}); + +describe('getPeerIntegrity', () => { + it('returns available:false with reason peer-not-found when peer is unknown', async () => { + vi.mocked(getPeers).mockResolvedValue([]); + const result = await getPeerIntegrity({ peerId: 'no-such-peer', kind: 'mediaCollection' }); + expect(result).toEqual({ available: false, reason: 'peer-not-found', records: [] }); + }); + + it('returns available:false with reason peer-too-old on 404 from peer', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + vi.mocked(peerFetch).mockResolvedValue({ ok: false, status: 404 }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result).toEqual({ available: false, reason: 'peer-too-old', records: [] }); + }); + + it('returns available:false with reason peer-unreachable when peerFetch throws', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + vi.mocked(peerFetch).mockRejectedValue(new Error('ECONNREFUSED')); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result).toEqual({ available: false, reason: 'peer-unreachable', records: [] }); + }); + + it('returns available:false with reason fetch-failed on non-404 error status', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + vi.mocked(peerFetch).mockResolvedValue({ ok: false, status: 500 }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result).toEqual({ available: false, reason: 'fetch-failed', records: [] }); + }); + + it('returns available:true with classified records when peer responds ok', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + + const ts = '2026-05-23T00:00:00.000Z'; + const localCol = makeCollection({ id: 'col-1', updatedAt: ts }); + vi.mocked(listCollections).mockResolvedValue([localCol]); + vi.mocked(assetShaListForRecord).mockResolvedValue([]); + + const remoteRecords = [ + { id: 'col-1', name: 'My Collection', updatedAt: ts, deleted: false, assetHashes: [] }, + ]; + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + status: 200, + json: async () => ({ records: remoteRecords }), + }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result.available).toBe(true); + expect(result.records).toHaveLength(1); + expect(result.records[0]).toMatchObject({ + id: 'col-1', + status: INTEGRITY_STATUS.IN_PARITY, + }); + }); + + it('does not throw on a hostile/malformed peer manifest (nulls, scalars, id-less objects)', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + const ts = '2026-05-23T00:00:00.000Z'; + vi.mocked(listCollections).mockResolvedValue([makeCollection({ id: 'col-1', updatedAt: ts })]); + vi.mocked(assetShaListForRecord).mockResolvedValue([]); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + status: 200, + json: async () => ({ records: [ + null, + 'not-an-object', + 42, + { name: 'no id here' }, // missing id + { id: 42 }, // non-string id + { id: 'col-1', name: 'My Collection', updatedAt: ts, deleted: false, assetHashes: [] }, // the only valid one + ] }), + }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result.available).toBe(true); + // Only the one well-formed remote row is diffed; the junk is filtered out. + expect(result.records).toHaveLength(1); + expect(result.records[0]).toMatchObject({ id: 'col-1', status: INTEGRITY_STATUS.IN_PARITY }); + }); + + it('surfaces PEER_ONLY records from peer that are absent locally', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + vi.mocked(listCollections).mockResolvedValue([]); + + const remoteRecords = [ + { + id: 'col-remote', + name: 'Remote Only', + updatedAt: '2026-05-23T00:00:00.000Z', + deleted: false, + assetHashes: [], + }, + ]; + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + status: 200, + json: async () => ({ records: remoteRecords }), + }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result.available).toBe(true); + expect(result.records[0]).toMatchObject({ + id: 'col-remote', + status: INTEGRITY_STATUS.PEER_ONLY, + }); + }); + + it('treats malformed peer body (no records array) as empty remote list', async () => { + const peer = makePeer('peer-x'); + vi.mocked(getPeers).mockResolvedValue([peer]); + vi.mocked(listCollections).mockResolvedValue([]); + + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + status: 200, + json: async () => ({ unexpected: 'shape' }), + }); + + const result = await getPeerIntegrity({ peerId: 'peer-x', kind: 'mediaCollection' }); + expect(result.available).toBe(true); + expect(result.records).toEqual([]); + }); +}); diff --git a/server/services/sharing/peerSync.js b/server/services/sharing/peerSync.js index 621b27833..055e567e5 100644 --- a/server/services/sharing/peerSync.js +++ b/server/services/sharing/peerSync.js @@ -35,7 +35,7 @@ * and tombstone GC (Stage 5; `sharing/tombstoneGc.js`). */ -import { join, basename } from 'path'; +import { join } from 'path'; import { existsSync } from 'fs'; import { EventEmitter } from 'events'; import { PATHS, atomicWrite, readJSONFile, ensureDir, sha256File } from '../../lib/fileUtils.js'; @@ -43,9 +43,11 @@ import { isStr } from '../../lib/storyBible.js'; import { isPlainObject } from '../../lib/objects.js'; import { peerBaseUrl } from '../../lib/peerUrl.js'; import { peerFetch } from '../../lib/peerHttpClient.js'; -import { getOrComputeImageSha256 } from '../../lib/assetHash.js'; +import { getOrComputeImageSha256, sidecarGenParamsHash } from '../../lib/assetHash.js'; import { sanitizeRecordForWire } from '../../lib/syncWire.js'; import { collectAssetReferences } from './exporter.js'; +import { imageSidecarName, sanitizeAssetFilename } from './buckets.js'; +import { pullSidecarForImage } from './sidecarSync.js'; import { recordEvents } from './recordEvents.js'; import { PORTOS_SCHEMA_VERSIONS, @@ -60,6 +62,8 @@ import { getUniverse, mergeUniversesFromSync } from '../universeBuilder.js'; import { getSeries, mergeSeriesFromSync } from '../pipeline/series.js'; import { listIssues, mergeIssuesFromSync } from '../pipeline/issues.js'; import { + getCollection, + listCollections, findCollectionByUniverseId, findCollectionBySeriesId, mergeMediaCollectionsFromSync, @@ -70,7 +74,7 @@ import { removeCursor as removeTombstoneCursor, } from './peerTombstoneCursors.js'; -export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series']); +export const PEER_SUBSCRIBABLE_KINDS = Object.freeze(['universe', 'series', 'mediaCollection']); /** * Cross-cutting event bus for the peer-sync receiver. The asset-pull worker @@ -264,6 +268,7 @@ export async function unsubscribePeer(id) { const KIND_TO_CATEGORY = Object.freeze({ universe: 'universe', series: 'pipeline', + mediaCollection: 'mediaCollections', }); function peerAllowsOutbound(peer) { @@ -342,6 +347,8 @@ export async function autoSubscribePeerToAllRecords(peerId, recordKind) { } else if (recordKind === 'series') { const { listSeries } = await import('../pipeline/series.js'); records = await listSeries({ includeDeleted: false }).catch(() => []); + } else if (recordKind === 'mediaCollection') { + records = await listCollections({ includeDeleted: false }).catch(() => []); } // Drop ephemeral records before the set-difference / sub creation. The wire // sanitizer would short-circuit any push anyway, but creating a sub that @@ -478,6 +485,67 @@ export async function buildAssetManifest(record) { return out; } +/** + * Map a collection's items array to the `{ directImageFilenames, + * directImageRefFilenames, directVideoFilenames }` shape consumed by the + * per-item manifest hashers. Collections store items as + * `{ kind:'image'|'video', ref, addedAt }` and carry no image-ref kind. + */ +export function collectCollectionAssetReferences(collection) { + const items = Array.isArray(collection?.items) ? collection.items : []; + const directImageFilenames = []; + const directVideoFilenames = []; + for (const it of items) { + if (it?.kind === 'image' && typeof it.ref === 'string') directImageFilenames.push(it.ref); + else if (it?.kind === 'video' && typeof it.ref === 'string') directVideoFilenames.push(it.ref); + } + return { directImageFilenames, directImageRefFilenames: [], directVideoFilenames }; +} + +// Video collection items store the BARE video id (e.g. a UUID), while the +// on-disk file is `.mp4` (today every PortOS-managed video is mp4 — +// confirmed by inspecting video-history.json). The image side stores refs +// WITH the extension already. Append `.mp4` unless the ref already carries an +// extension (defensive — older state may have stamped a filename instead of an +// id, and a future video format would land as `.webm` etc.). Shared by BOTH +// collection manifest builders (`buildCollectionAssetManifest` for standalone +// mediaCollection pushes, `buildAssetManifestForCollection` for the +// linkedCollection bundle) so the two can't diverge on the extension rule. +function collectionVideoRefToFilename(ref) { + return /\.[a-z0-9]+$/i.test(ref) ? ref : `${ref}.mp4`; +} + +async function buildCollectionAssetManifest(collection) { + const refs = collectCollectionAssetReferences(collection); + const out = []; + for (const filename of refs.directImageFilenames) { + const entry = await hashImageForManifest(filename); + if (entry) out.push(entry); + } + for (const ref of refs.directVideoFilenames) { + const entry = await hashSimpleAsset(collectionVideoRefToFilename(ref), 'video', PATHS.videos); + if (entry) out.push(entry); + } + return out; +} + +/** + * Returns sorted sha256 hashes for a record's own assets (used by the + * integrity manifest builder). For `series` this captures the series' OWN + * asset refs only — child-issue assets are not yet included (v1 limitation; + * full issue-level integrity is deferred to a future pass). + * + * @param {'universe'|'series'|'mediaCollection'} kind + * @param {object} record + * @returns {Promise} sorted sha256 strings (falsy hashes omitted) + */ +export async function assetShaListForRecord(kind, record) { + const manifest = kind === 'mediaCollection' + ? await buildCollectionAssetManifest(record) + : await buildAssetManifest(record); + return manifest.map((e) => e.sha256).filter(Boolean).sort(); +} + async function hashImageForManifest(filename) { // Sanitize before join — a record with `imageRefs` containing a path- // traversal filename (peer-pushed via linkedCollection, hand-edited, @@ -489,7 +557,17 @@ async function hashImageForManifest(filename) { const fullPath = join(PATHS.images, safeName); const result = await getOrComputeImageSha256(fullPath); if (!result) return null; - return { filename: safeName, kind: 'image', sha256: result.hash }; + // Advertise a sidecarSha256 only when the sidecar carries gen-params beyond + // the `sha256` cache block. CRITICAL: we hash the GEN-PARAMS ONLY (sorted-key + // canonical form, `sha256` cache key stripped) via `sidecarGenParamsHash` — + // NOT the raw sidecar file. The `sha256` block embeds the LOCAL image's + // mtime+size, so hashing the whole file would never converge across machines + // (the receiver re-stamps its own mtime after every pull and re-diverges, + // re-pulling the sidecar every sync cycle). `sidecarGenParamsHash` returns + // null when there are no gen-params, so we never advertise a hash for a + // pure cache-only sidecar. + const sidecarSha256 = sidecarGenParamsHash(result.sidecar); + return { filename: safeName, kind: 'image', sha256: result.hash, ...(sidecarSha256 ? { sidecarSha256 } : {}) }; } async function hashSimpleAsset(filename, kind, sourceDir) { @@ -526,8 +604,8 @@ export async function diffAssetManifestAgainstLocal(manifest) { // entry. Reject anything that isn't a bare basename before any FS op. const safeName = sanitizeAssetFilename(entry.filename); if (!safeName) continue; - // Build a sanitized projection: only the three fields the sender needs - // back to pull. Echoing the raw peer-supplied entry would amplify any + // Build a sanitized projection: only the known fields the receiver needs + // to pull. Echoing the raw peer-supplied entry would amplify any // junk fields it shipped (large strings, extra kinds, prototype-pollution // attempts) into the response — wire-symmetry should not let untrusted // input round-trip through our process untouched. @@ -535,12 +613,22 @@ export async function diffAssetManifestAgainstLocal(manifest) { filename: safeName, kind: entry.kind, ...(isStr(entry.sha256) ? { sha256: entry.sha256 } : {}), + ...(isStr(entry.sidecarSha256) ? { sidecarSha256: entry.sidecarSha256 } : {}), }; const fullPath = join(dir, safeName); if (!existsSync(fullPath)) { missing.push(sanitizedEntry); continue; } + // For images, compute the hash result once up front: it carries both the + // sha256 AND the parsed sidecar JSON, so the sidecarSha256 comparison below + // reuses it instead of re-reading the same file (one sidecar read per image + // instead of two). Only touch the cache machinery when a comparison will + // actually use it (sha256 or sidecarSha256 advertised by the peer). + let imageHashResult = null; + if (entry.kind === 'image' && (isStr(entry.sha256) || isStr(entry.sidecarSha256))) { + imageHashResult = await getOrComputeImageSha256(fullPath); + } // Compare SHA when the manifest carries one — for ALL kinds, not just // images. The image path uses the sidecar cache (fast for the common // ~200-asset universe case); image-ref/video stream-hash on demand. @@ -549,9 +637,30 @@ export async function diffAssetManifestAgainstLocal(manifest) { // ONLY thing that would catch it 60s later — better to detect on push. if (isStr(entry.sha256)) { const localHash = entry.kind === 'image' - ? (await getOrComputeImageSha256(fullPath))?.hash ?? null + ? imageHashResult?.hash ?? null : await sha256File(fullPath).catch(() => null); - if (localHash !== entry.sha256) missing.push(sanitizedEntry); + if (localHash !== entry.sha256) { + missing.push(sanitizedEntry); + continue; + } + } + // Sidecar-only divergence: image bytes are already present and hash-match, + // but the peer has a gen-params sidecar we're missing or have stale. + // Pull the entry so the worker can fetch ONLY the sidecar (it checks the + // image hash before deciding whether to re-pull the image bytes). + // + // We MUST recompute the local sidecar hash the SAME way the sender did + // (`sidecarGenParamsHash` — gen-params only, sorted-key canonical, `sha256` + // cache block stripped). Hashing the raw sidecar file would never match the + // sender's gen-params-only hash and would re-flag the image every cycle. + if (entry.kind === 'image' && isStr(entry.sidecarSha256)) { + // Reuse the sidecar already loaded by getOrComputeImageSha256; only fall + // back to a direct read if that result was unavailable (e.g. the image + // became unreadable between the existsSync check and the stat). + const localSidecar = imageHashResult?.sidecar + ?? await readJSONFile(join(PATHS.images, imageSidecarName(safeName)), null, { logError: false }); + const localSidecarHash = sidecarGenParamsHash(localSidecar); + if (localSidecarHash !== entry.sidecarSha256) missing.push(sanitizedEntry); } } return missing; @@ -564,25 +673,6 @@ function directoryForAssetKind(kind) { return null; } -/** - * Returns the filename if it's safe to use as a path segment under the asset - * directory, otherwise null. Rejects path separators, parent-directory - * tokens, and any value that doesn't match its own basename — same posture - * as `jobFromSidecar` in services/sharing/exporter.js for symmetry with - * how the share-bucket importer validates inbound asset filenames. - */ -function sanitizeAssetFilename(name) { - if (typeof name !== 'string' || !name) return null; - // Reject separators and exact parent-directory segments (`.` / `..` - // as the whole basename). A basename like `my..render.png` is - // legitimate (the gallery filename validator permits `..` inside a - // basename) — only the path-segment forms are traversal. - if (name.includes('/') || name.includes('\\')) return null; - if (name === '.' || name === '..') return null; - if (basename(name) !== name) return null; - return name; -} - // --- Push pipeline (sender side) ---------------------------------------- /** @@ -937,6 +1027,14 @@ async function buildPushPayload(sub, sourceInstanceId) { ...(linkedCollection ? { linkedCollection } : {}), }; } + if (sub.recordKind === 'mediaCollection') { + const record = await getCollection(sub.recordId, { includeDeleted: true }).catch(() => null); + if (!record) return null; + const sanitized = sanitizeRecordForWire('mediaCollection', record); + if (!sanitized) return null; + const assetManifest = record.deleted === true ? [] : await buildCollectionAssetManifest(record); + return { kind: 'mediaCollection', record: sanitized, assetManifest, sourceInstanceId, portosMeta }; + } return null; } @@ -1003,15 +1101,10 @@ async function buildAssetManifestForCollection(collection) { const safeName = sanitizeAssetFilename(it.ref); if (!safeName) continue; if (it.kind === 'video') { - // Video collection items store the bare video id (e.g. a UUID), while - // the on-disk file is `.mp4` (today every PortOS-managed video is - // mp4 — confirmed by inspecting video-history.json). The image side - // stores refs WITH the extension already, so it works as-is. Append - // `.mp4` here unless the ref already carries an extension (defensive - // — older state may have stamped a filename instead of an id, and a - // future video format would land as `.webm` etc.). - const filename = /\.[a-z0-9]+$/i.test(safeName) ? safeName : `${safeName}.mp4`; - const entry = await hashSimpleAsset(filename, 'video', PATHS.videos); + // Bare videoId → `.mp4` via the shared helper (see + // collectionVideoRefToFilename). `sanitizeAssetFilename` already ran on + // `it.ref` above; the extension append is purely the on-disk naming rule. + const entry = await hashSimpleAsset(collectionVideoRefToFilename(safeName), 'video', PATHS.videos); if (entry) out.push(entry); } else { // Treat 'image' (and any unknown kind that isn't 'video') as a gallery @@ -1156,6 +1249,8 @@ export async function applyIncomingPush(payload) { if (!localEphemeral && Array.isArray(issues) && issues.length > 0) { await mergeIssuesFromSync(issues); } + } else if (kind === 'mediaCollection') { + await mergeMediaCollectionsFromSync([record]); } // Apply the bundled collection (if any) — same LWW + union-of-items @@ -1329,6 +1424,16 @@ async function classifyLocalRecord(recordKind, recordId) { if (!s) return 'missing'; return s.ephemeral === true ? 'ephemeral' : 'syncable'; } + if (recordKind === 'mediaCollection') { + // Collections have no `ephemeral` concept, so a found record is always + // 'syncable'. Without this branch, maybeCreateReverseSubscription's + // `localState !== 'syncable'` guard would never bootstrap bidirectional + // collection sync from an inbound push. No ping-pong risk — the + // lastPushedHash short-circuit + LWW same-`updatedAt` no-op merge prevent + // it, same as universe/series. + const c = await getCollection(recordId, { includeDeleted: true }).catch(() => null); + return c ? 'syncable' : 'missing'; + } return 'missing'; } @@ -1425,6 +1530,23 @@ async function pullOneAsset(peer, base, entry) { } async function doPullOneAsset(peer, base, entry, urlPrefix, localDir, safeName) { + // Sidecar-only divergence: image bytes are already present and hash-match + // the sender's manifest (diffAssetManifestAgainstLocal still returned this + // entry because the local sidecar is absent or stale). Skip the image + // re-pull and go straight to the sidecar fetch — avoids re-downloading a + // potentially large PNG for a metadata-only update. + if (entry.kind === 'image' && isStr(entry.sha256)) { + const localFullPath = join(localDir, safeName); + if (existsSync(localFullPath)) { + const localHash = (await getOrComputeImageSha256(localFullPath))?.hash ?? null; + if (localHash === entry.sha256) { + // Image bytes already up-to-date — pull sidecar only. + await pullSidecarForImage(peer, base, safeName).catch(() => {}); + return; + } + } + } + const url = `${base}${urlPrefix}/${encodeURIComponent(safeName)}`; const controller = new AbortController(); const timeoutId = setTimeout(() => controller.abort(), ASSET_PULL_TIMEOUT_MS); @@ -1492,6 +1614,12 @@ async function doPullOneAsset(peer, base, entry, urlPrefix, localDir, safeName) peerId: peer.instanceId, }); console.log(`📥 peerSync: pulled ${entry.kind}/${safeName} from ${peer.name || peer.instanceId} (${buffer.length} bytes)`); + // After a successful image pull, also fetch the gen-params sidecar if present + // on the sender. Best-effort: the image is already safely written above; + // a missing sidecar just means the image lands in Unsorted without a prompt. + if (entry.kind === 'image') { + await pullSidecarForImage(peer, base, safeName).catch(() => {}); + } } // --- Listener install + debounced trigger ------------------------------- @@ -1546,8 +1674,14 @@ async function pushFromFreshSubscription(subId) { * - For issue updates, the subscription on the parent series (resolved * via `getIssueSeriesId` — see below). */ -async function collectSubscriptionsForUpdate(recordKind, recordId) { - if (recordKind === 'universe' || recordKind === 'series') { +export async function collectSubscriptionsForUpdate(recordKind, recordId) { + // Direct-subscription kinds: a peer subscribes to the record itself, so an + // edit/delete fires a push to exactly those subs. mediaCollection belongs + // here (standalone collections sync per-record) — omitting it would make + // mediaCollections.js's emitRecordUpdated('mediaCollection', …) inert, so + // collection edits would only reach peers via initial subscribe / manual + // force-push, never on subsequent edits. + if (recordKind === 'universe' || recordKind === 'series' || recordKind === 'mediaCollection') { return listPeerSubscriptions({ recordKind, recordId }); } if (recordKind === 'issue') { @@ -1616,6 +1750,45 @@ export async function retryPendingPushesForPeer(peerId) { return { walked: subs.length, pushed }; } +/** + * Force a push for a specific (peer, kind, record) regardless of the + * unchanged-hash short-circuit. Resolves or creates the subscription first, + * then pushes with lastPushedHash nulled so pushRecordToPeer always fires a + * network call (idempotent LWW on the receiver). + * + * `subscribePeer` fires its own initial push on first insert; `forcePushRecord` + * then force-pushes again. The double-push is acceptable — the receiver's + * merge*FromSync paths are LWW and the second push is a no-op content-wise. + */ +export async function forcePushRecord(peerId, recordKind, recordId) { + const existing = await findPeerSubscription(peerId, recordKind, recordId); + const sub = existing || await subscribePeer({ peerId, recordKind, recordId }); + // Null the lastPushedHash to bypass the unchanged short-circuit in pushRecordToPeer. + console.log(`🔄 peerSync: force-push ${recordKind}/${recordId} → ${peerId}`); + return pushRecordToPeer({ ...sub, lastPushedHash: null }, { bypassSchemaCooldown: true }); +} + +/** + * Trigger an immediate full-sync for a single peer: backfill subscriptions for + * every enabled category and then retry all pending/stale pushes. Best-effort + * — per-kind failures are swallowed so one bad kind doesn't block the rest. + */ +export async function syncNowForPeer(peerId) { + const peer = await findPeerById(peerId); + if (!peer?.instanceId) return { ok: false }; + for (const kind of PEER_SUBSCRIBABLE_KINDS) { + if (peerHasCategory(peer, kind)) { + await autoSubscribePeerToAllRecords(peer.instanceId, kind).catch((err) => { + console.log(`⚠️ peerSync: syncNow backfill ${kind} → ${peerId} failed: ${err.message}`); + }); + } + } + await retryPendingPushesForPeer(peer.instanceId).catch((err) => { + console.log(`⚠️ peerSync: syncNow retry pushes → ${peerId} failed: ${err.message}`); + }); + return { ok: true }; +} + let onUpdated = null; let onDeleted = null; let onPeerOnline = null; diff --git a/server/services/sharing/peerSync.test.js b/server/services/sharing/peerSync.test.js index b25edfd09..7b9e1c2bf 100644 --- a/server/services/sharing/peerSync.test.js +++ b/server/services/sharing/peerSync.test.js @@ -39,6 +39,8 @@ vi.mock('../pipeline/issues.js', async () => ({ })); vi.mock('../mediaCollections.js', async () => ({ + getCollection: vi.fn(), + listCollections: vi.fn(), findCollectionByUniverseId: vi.fn(), findCollectionBySeriesId: vi.fn(), mergeMediaCollectionsFromSync: vi.fn(), @@ -61,9 +63,13 @@ import { applyIncomingPush, diffAssetManifestAgainstLocal, buildAssetManifest, + collectCollectionAssetReferences, autoSubscribeRecordToAllPeers, autoSubscribePeerToAllRecords, retryPendingPushesForPeer, + forcePushRecord, + syncNowForPeer, + collectSubscriptionsForUpdate, __resetForTests, __drainForTests, } from './peerSync.js'; @@ -73,6 +79,8 @@ import { getUniverse, mergeUniversesFromSync, listUniverses } from '../universeB import { getSeries, mergeSeriesFromSync, listSeries } from '../pipeline/series.js'; import { listIssues, mergeIssuesFromSync } from '../pipeline/issues.js'; import { + getCollection, + listCollections, findCollectionByUniverseId, findCollectionBySeriesId, mergeMediaCollectionsFromSync, @@ -143,6 +151,8 @@ beforeEach(async () => { vi.mocked(listIssues).mockReset().mockResolvedValue([]); // Default: no linked collection for any record. Tests that exercise the // bundle path override these per-call. + vi.mocked(getCollection).mockReset().mockRejectedValue(Object.assign(new Error('Collection not found'), { code: 'NOT_FOUND' })); + vi.mocked(listCollections).mockReset().mockResolvedValue([]); vi.mocked(findCollectionByUniverseId).mockReset().mockResolvedValue(null); vi.mocked(findCollectionBySeriesId).mockReset().mockResolvedValue(null); vi.mocked(mergeMediaCollectionsFromSync).mockReset().mockResolvedValue({ applied: false, count: 0 }); @@ -173,10 +183,12 @@ afterEach(async () => { describe('peerSync', () => { describe('PEER_SUBSCRIBABLE_KINDS', () => { - it('exposes the canonical kinds (universe + series only)', () => { - // Issues piggyback on series subscriptions — direct issue subs are - // intentionally rejected per the Stage 2 design. - expect(PEER_SUBSCRIBABLE_KINDS).toEqual(['universe', 'series']); + it('is exactly [universe, series, mediaCollection]', () => { + // Exact equality (not toContain) so an accidental add/remove/reorder is + // caught — this list is canonical and its order can affect iteration + // elsewhere (e.g. syncNow's per-kind backfill). Issues piggyback on series + // subscriptions; direct issue subs are intentionally rejected (Stage 2). + expect(PEER_SUBSCRIBABLE_KINDS).toEqual(['universe', 'series', 'mediaCollection']); }); }); @@ -651,6 +663,86 @@ describe('peerSync', () => { }); }); + describe('forcePushRecord', () => { + beforeEach(() => { + vi.mocked(getUniverse).mockResolvedValue({ id: 'u1', name: 'Forced', updatedAt: '2026-01-01T00:00:00Z' }); + vi.mocked(listIssues).mockResolvedValue([]); + vi.mocked(findCollectionByUniverseId).mockResolvedValue(null); + }); + + it('pushes even when existing sub lastPushedHash matches (bypasses unchanged short-circuit)', async () => { + // First subscribe + initial push — record gets a lastPushedHash. + vi.mocked(peerFetch).mockResolvedValue({ ok: true, json: async () => ({ missingAssets: [] }) }); + await subscribePeer({ peerId: 'peer-a', recordKind: 'universe', recordId: 'u1' }); + // Poll for the fire-and-forget initial push to persist its hash. A fixed + // sleep OR a single writeTail drain (__drainForTests) is racy in slower CI + // because the push's peerFetch + persistPushSuccess chain may not have even + // started when we read — vi.waitFor retries the real condition deterministically. + let sub; + await vi.waitFor(async () => { + sub = await findPeerSubscription('peer-a', 'universe', 'u1'); + expect(sub?.lastPushedHash).toBeTruthy(); + }); + expect(sub.lastPushedHash).toBeTruthy(); // hash was recorded + + // Clear the mock call count — we care only about calls from forcePushRecord. + vi.mocked(peerFetch).mockClear(); + vi.mocked(peerFetch).mockResolvedValue({ ok: true, json: async () => ({ missingAssets: [] }) }); + + // forcePushRecord MUST hit the network despite the hash being unchanged. + const result = await forcePushRecord('peer-a', 'universe', 'u1'); + expect(result.pushed).toBe(true); + expect(vi.mocked(peerFetch)).toHaveBeenCalledTimes(1); + expect(vi.mocked(peerFetch)).toHaveBeenCalledWith( + expect.stringContaining('/api/peer-sync/push'), + expect.objectContaining({ method: 'POST' }), + ); + }); + + it('creates a subscription and pushes when no sub existed yet', async () => { + vi.mocked(peerFetch).mockResolvedValue({ ok: true, json: async () => ({ missingAssets: [] }) }); + + const result = await forcePushRecord('peer-a', 'universe', 'u1'); + expect(result.pushed).toBe(true); + + // Subscription should now exist. + const sub = await findPeerSubscription('peer-a', 'universe', 'u1'); + expect(sub).not.toBeNull(); + }); + }); + + describe('syncNowForPeer', () => { + it('returns {ok:false} when the peer has no instanceId', async () => { + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: null, name: 'No ID Peer', enabled: true, syncEnabled: true }, + ]); + const result = await syncNowForPeer('no-such-peer'); + expect(result).toEqual({ ok: false }); + }); + + it('returns {ok:false} when the peer is not found', async () => { + const result = await syncNowForPeer('ghost-peer'); + expect(result).toEqual({ ok: false }); + }); + + it('calls backfill+retry for a peer with enabled categories and returns {ok:true}', async () => { + vi.mocked(getPeers).mockResolvedValue([ + { + instanceId: 'peer-a', name: 'Peer A', host: null, address: '10.0.0.2', port: 5555, + enabled: true, syncEnabled: true, + directions: ['outbound', 'inbound'], + syncCategories: { universe: true, pipeline: false, mediaCollections: false }, + }, + ]); + vi.mocked(listUniverses).mockResolvedValue([{ id: 'u1', name: 'Universe 1' }]); + vi.mocked(getUniverse).mockResolvedValue({ id: 'u1', name: 'Universe 1', updatedAt: '2026-01-01T00:00:00Z' }); + vi.mocked(peerFetch).mockResolvedValue({ ok: true, json: async () => ({ missingAssets: [] }) }); + + const result = await syncNowForPeer('peer-a'); + expect(result).toEqual({ ok: true }); + }); + }); + describe('buildAssetManifest', () => { it('hashes direct image filenames via the sidecar cache', async () => { await writeFile(join(PATHS.images, 'asset-1.png'), Buffer.from('image bytes')); @@ -678,6 +770,23 @@ describe('peerSync', () => { const manifest = await buildAssetManifest({ id: 'u1', name: 'Bare' }); expect(manifest).toEqual([]); }); + + it('includes sidecarSha256 in the manifest entry when sidecar is present', async () => { + await writeFile(join(PATHS.images, 'with-sidecar.png'), Buffer.from('image bytes')); + await writeFile(join(PATHS.images, 'with-sidecar.metadata.json'), Buffer.from(JSON.stringify({ prompt: 'a cat' }))); + const record = { id: 'u1', characters: [{ imageRefs: ['with-sidecar.png'] }] }; + const manifest = await buildAssetManifest(record); + expect(manifest).toHaveLength(1); + expect(manifest[0].sidecarSha256).toMatch(/^[a-f0-9]{64}$/); + }); + + it('omits sidecarSha256 from the manifest entry when no sidecar exists', async () => { + await writeFile(join(PATHS.images, 'no-sidecar.png'), Buffer.from('image bytes')); + const record = { id: 'u1', characters: [{ imageRefs: ['no-sidecar.png'] }] }; + const manifest = await buildAssetManifest(record); + expect(manifest).toHaveLength(1); + expect(manifest[0]).not.toHaveProperty('sidecarSha256'); + }); }); describe('diffAssetManifestAgainstLocal', () => { @@ -774,6 +883,104 @@ describe('peerSync', () => { expect(missing).toHaveLength(2); expect(missing.map((m) => m.filename).sort()).toEqual(['clip.mp4', 'ref.png']); }); + + it('returns image entry as missing when image hash matches but sidecar is absent', async () => { + // The image file is already present and hash-matches; BUT the sender + // carries a sidecarSha256 and we have no local sidecar. The diff must + // still include the entry so the worker pulls the sidecar. + const imageBytes = Buffer.from('hello world'); + await writeFile(join(PATHS.images, 'nosidecar.png'), imageBytes); + // Actual sha256 of "hello world" + const imageHash = 'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'; + const missing = await diffAssetManifestAgainstLocal([ + { filename: 'nosidecar.png', kind: 'image', sha256: imageHash, sidecarSha256: 'a'.repeat(64) }, + ]); + expect(missing).toHaveLength(1); + expect(missing[0].filename).toBe('nosidecar.png'); + expect(missing[0].sidecarSha256).toBe('a'.repeat(64)); + }); + + it('does NOT return an image as missing when image hash matches and sidecar gen-params match', async () => { + // Both image and sidecar are present; the gen-params are identical — no pull needed. + const imageBytes = Buffer.from('hello world'); + await writeFile(join(PATHS.images, 'fullmatch.png'), imageBytes); + await writeFile(join(PATHS.images, 'fullmatch.metadata.json'), Buffer.from(JSON.stringify({ prompt: 'cat' }))); + const imageHash = 'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'; + const { sidecarGenParamsHash } = await import('../../lib/assetHash.js'); + // The sender advertises the gen-params hash (NOT the raw-file hash). + const senderSidecarHash = sidecarGenParamsHash({ prompt: 'cat' }); + const missing = await diffAssetManifestAgainstLocal([ + { filename: 'fullmatch.png', kind: 'image', sha256: imageHash, sidecarSha256: senderSidecarHash }, + ]); + expect(missing).toEqual([]); + }); + + it('CONVERGENCE: image NOT re-flagged when gen-params match but the sha256 cache block differs', async () => { + // CRITICAL regression for the churn bug. Two machines with byte-identical + // gen-params but different per-machine `sha256` cache blocks (mtimeMs/size + // differ) must NOT perpetually re-pull. The sidecar hash is computed over + // gen-params ONLY (sha256 cache stripped), so it converges regardless of + // the local cache block. + const imageBytes = Buffer.from('hello world'); + await writeFile(join(PATHS.images, 'converge.png'), imageBytes); + // Local sidecar: SAME gen-params, but a DIFFERENT sha256 cache block than + // whatever the sender stamped (simulates the receiver re-stamping its own + // local image mtime+size after a prior pull). + await writeFile(join(PATHS.images, 'converge.metadata.json'), Buffer.from(JSON.stringify({ + prompt: 'a wizard', model: 'flux', steps: 30, + sha256: { value: 'c'.repeat(64), mtimeMs: 111111, size: 222 }, + }))); + const imageHash = 'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'; + const { sidecarGenParamsHash } = await import('../../lib/assetHash.js'); + // The SENDER computes the gen-params hash over its OWN sidecar, which has + // the SAME gen-params but a DIFFERENT sha256 cache block (different mtime/size). + const senderSidecarHash = sidecarGenParamsHash({ + prompt: 'a wizard', model: 'flux', steps: 30, + sha256: { value: 'd'.repeat(64), mtimeMs: 999999, size: 888 }, + }); + // The receiver computes its local gen-params hash the same way. + const localSidecarHash = sidecarGenParamsHash({ + prompt: 'a wizard', model: 'flux', steps: 30, + sha256: { value: 'c'.repeat(64), mtimeMs: 111111, size: 222 }, + }); + // The two MUST be equal despite differing cache blocks — proves convergence. + expect(senderSidecarHash).toBe(localSidecarHash); + // And the diff must NOT flag the image as missing (no re-pull churn). + const missing = await diffAssetManifestAgainstLocal([ + { filename: 'converge.png', kind: 'image', sha256: imageHash, sidecarSha256: senderSidecarHash }, + ]); + expect(missing).toEqual([]); + }); + + it('CONVERGENCE: key-order differences across machines do not break the gen-params hash', async () => { + const { sidecarGenParamsHash } = await import('../../lib/assetHash.js'); + const a = sidecarGenParamsHash({ prompt: 'x', model: 'flux', steps: 30 }); + const b = sidecarGenParamsHash({ steps: 30, prompt: 'x', model: 'flux' }); + expect(a).toBe(b); + }); + + it('returns image entry as missing when sidecar hash differs (peer has updated metadata)', async () => { + const imageBytes = Buffer.from('hello world'); + const sidecarBytes = Buffer.from(JSON.stringify({ prompt: 'old prompt' })); + await writeFile(join(PATHS.images, 'staleside.png'), imageBytes); + await writeFile(join(PATHS.images, 'staleside.metadata.json'), sidecarBytes); + const imageHash = 'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'; + const missing = await diffAssetManifestAgainstLocal([ + { filename: 'staleside.png', kind: 'image', sha256: imageHash, sidecarSha256: 'b'.repeat(64) }, + ]); + expect(missing).toHaveLength(1); + expect(missing[0].filename).toBe('staleside.png'); + }); + + it('preserves sidecarSha256 in the sanitized missing entry (no untrusted round-trip loss)', async () => { + const missing = await diffAssetManifestAgainstLocal([ + { filename: 'absent.png', kind: 'image', sha256: 'a'.repeat(64), sidecarSha256: 'b'.repeat(64) }, + ]); + expect(missing).toHaveLength(1); + expect(missing[0].sidecarSha256).toBe('b'.repeat(64)); + // Junk fields still stripped. + expect(missing[0]).not.toHaveProperty('gigantic'); + }); }); describe('pushRecordToPeer', () => { @@ -1184,6 +1391,122 @@ describe('peerSync', () => { const manifestFilenames = (captured.assetManifest || []).map(a => a.filename); expect(manifestFilenames).toEqual([]); }); + + // --- Standalone mediaCollection push payload -------------------------- + // A peer subscribed directly to a mediaCollection record (NOT via a + // universe/series's linkedCollection bundle). The peer must have the + // `mediaCollections` syncCategory enabled or the push gate short-circuits. + const enableCollectionPeer = () => { + vi.mocked(getPeers).mockResolvedValue([ + { + instanceId: 'peer-a', name: 'Peer A', host: null, address: '10.0.0.2', port: 5555, + enabled: true, syncEnabled: true, + directions: ['outbound', 'inbound'], + syncCategories: { universe: true, pipeline: true, mediaCollections: true }, + }, + ]); + }; + + it('emits BOTH image and video manifest entries for a live mediaCollection push', async () => { + // Regression (Bug 2): video collection items store the bare videoId; the + // on-disk file is `.mp4`. The standalone push manifest builder must + // append `.mp4` (same as the linkedCollection bundle path) or every + // collection video is silently dropped and receivers never pull bytes. + enableCollectionPeer(); + PATHS.videos = join(tmp, 'videos'); + await mkdir(PATHS.videos, { recursive: true }); + await writeFile(join(PATHS.images, 'pic.png'), Buffer.from('image bytes')); + await writeFile(join(PATHS.videos, 'vid-xyz.mp4'), Buffer.from('mp4 bytes')); + + vi.mocked(getCollection).mockResolvedValue({ + id: 'col-9', name: 'Standalone', description: '', coverKey: null, + universeId: null, seriesId: null, + items: [ + { kind: 'image', ref: 'pic.png', addedAt: '2026-05-22T01:00:00Z' }, + { kind: 'video', ref: 'vid-xyz', addedAt: '2026-05-22T02:00:00Z' }, + ], + createdAt: '2026-05-22T00:00:00Z', updatedAt: '2026-05-22T02:00:00Z', + deleted: false, deletedAt: null, + }); + let captured = null; + vi.mocked(peerFetch).mockImplementation(async (_url, opts) => { + captured = JSON.parse(opts.body); + return { ok: true, json: async () => ({}) }; + }); + await pushRecordToPeer({ + id: 'sub-col', peerId: 'peer-a', recordKind: 'mediaCollection', recordId: 'col-9', + }); + expect(captured.kind).toBe('mediaCollection'); + expect(captured.record.id).toBe('col-9'); + const imageEntries = captured.assetManifest.filter(a => a.kind === 'image'); + const videoEntries = captured.assetManifest.filter(a => a.kind === 'video'); + expect(imageEntries.map(a => a.filename)).toEqual(['pic.png']); + // The .mp4 must have been appended to the bare videoId. + expect(videoEntries.map(a => a.filename)).toEqual(['vid-xyz.mp4']); + }); + + it('ships an empty asset manifest for a tombstone mediaCollection push', async () => { + enableCollectionPeer(); + PATHS.videos = join(tmp, 'videos'); + await mkdir(PATHS.videos, { recursive: true }); + // Real files on disk would otherwise hash into the manifest — the + // deleted gate must skip the manifest builder entirely. + await writeFile(join(PATHS.images, 'doomed.png'), Buffer.from('bytes')); + await writeFile(join(PATHS.videos, 'vid-doom.mp4'), Buffer.from('bytes')); + + vi.mocked(getCollection).mockResolvedValue({ + id: 'col-tomb', name: 'Doomed', description: '', coverKey: null, + universeId: null, seriesId: null, + items: [ + { kind: 'image', ref: 'doomed.png', addedAt: '2026-05-22T01:00:00Z' }, + { kind: 'video', ref: 'vid-doom', addedAt: '2026-05-22T02:00:00Z' }, + ], + createdAt: '2026-05-22T00:00:00Z', updatedAt: '2026-05-22T03:00:00Z', + deleted: true, deletedAt: '2026-05-22T03:00:00Z', + }); + let captured = null; + vi.mocked(peerFetch).mockImplementation(async (_url, opts) => { + captured = JSON.parse(opts.body); + return { ok: true, json: async () => ({}) }; + }); + await pushRecordToPeer({ + id: 'sub-tomb', peerId: 'peer-a', recordKind: 'mediaCollection', recordId: 'col-tomb', + }); + expect(captured.record.id).toBe('col-tomb'); + expect(captured.record.deleted).toBe(true); + expect(captured.assetManifest).toEqual([]); + }); + }); + + describe('collectSubscriptionsForUpdate', () => { + // Regression: mediaCollections.js emits emitRecordUpdated('mediaCollection',…) + // on every edit/delete, but the push pipeline only acted on it if + // collectSubscriptionsForUpdate returns the direct subs. Omitting the + // mediaCollection branch made those emits inert (edits never auto-pushed). + it('returns direct mediaCollection subscriptions (so edits/deletes auto-push)', async () => { + vi.mocked(getPeers).mockResolvedValue([{ + instanceId: 'peer-a', name: 'Peer A', host: null, address: '10.0.0.2', port: 5555, + enabled: true, syncEnabled: true, directions: ['outbound', 'inbound'], + syncCategories: { universe: true, pipeline: true, mediaCollections: true }, + }]); + vi.mocked(getCollection).mockResolvedValue({ + id: 'col-7', name: 'Standalone', description: '', coverKey: null, + universeId: null, seriesId: null, items: [], + createdAt: '2026-05-22T00:00:00Z', updatedAt: '2026-05-22T00:00:00Z', + deleted: false, deletedAt: null, + }); + vi.mocked(peerFetch).mockResolvedValue({ ok: true, json: async () => ({ missingAssets: [] }) }); + await subscribePeer({ peerId: 'peer-a', recordKind: 'mediaCollection', recordId: 'col-7' }); + await __drainForTests(); + + const subs = await collectSubscriptionsForUpdate('mediaCollection', 'col-7'); + expect(subs.map((s) => s.recordId)).toContain('col-7'); + expect(subs.every((s) => s.recordKind === 'mediaCollection')).toBe(true); + }); + + it('returns [] for a kind with no direct/parent subscription path', async () => { + expect(await collectSubscriptionsForUpdate('image', 'whatever')).toEqual([]); + }); }); describe('applyIncomingPush', () => { @@ -1703,4 +2026,128 @@ describe('peerSync', () => { }); }); }); + + // ------------------------------------------------------------------------- + // mediaCollection push + receiver + // ------------------------------------------------------------------------- + describe('collectCollectionAssetReferences', () => { + it('maps items to image/video refs with an empty directImageRefFilenames', () => { + const refs = collectCollectionAssetReferences({ items: [ + { kind: 'image', ref: 'a.png' }, + { kind: 'video', ref: 'vid123' }, + { kind: 'image', ref: 'b.png' }, + ] }); + expect(refs.directImageFilenames).toEqual(['a.png', 'b.png']); + expect(refs.directVideoFilenames).toEqual(['vid123']); + expect(refs.directImageRefFilenames).toEqual([]); + }); + + it('returns empty arrays for a collection with no items', () => { + const refs = collectCollectionAssetReferences({ items: [] }); + expect(refs.directImageFilenames).toEqual([]); + expect(refs.directVideoFilenames).toEqual([]); + expect(refs.directImageRefFilenames).toEqual([]); + }); + + it('returns empty arrays for null/undefined input', () => { + expect(collectCollectionAssetReferences(null).directImageFilenames).toEqual([]); + expect(collectCollectionAssetReferences(undefined).directImageFilenames).toEqual([]); + }); + }); + + describe('applyIncomingPush — mediaCollection', () => { + it('applies an incoming mediaCollection push into local collections', async () => { + // Use the real mergeMediaCollectionsFromSync via importActual so the + // write lands in the tmpdir and listCollections can confirm persistence. + const real = await vi.importActual('../mediaCollections.js'); + vi.mocked(mergeMediaCollectionsFromSync).mockImplementationOnce(real.mergeMediaCollectionsFromSync); + vi.mocked(listCollections).mockImplementationOnce(real.listCollections); + + await applyIncomingPush({ + kind: 'mediaCollection', + record: { id: 'col-x', name: 'Synced', items: [], updatedAt: '2026-05-23T00:00:00.000Z' }, + assetManifest: [], + sourceInstanceId: 'peer-abc', + }); + + expect(mergeMediaCollectionsFromSync).toHaveBeenCalledWith([ + expect.objectContaining({ id: 'col-x', name: 'Synced' }), + ]); + + // Confirm the record landed on disk via the real listCollections. + const all = await real.listCollections(); + const found = all.find((c) => c.id === 'col-x'); + expect(found).toBeDefined(); + expect(found.name).toBe('Synced'); + }); + + it('routes a mediaCollection push through mergeMediaCollectionsFromSync (mock assertion)', async () => { + await applyIncomingPush({ + kind: 'mediaCollection', + record: { id: 'col-y', name: 'Test', items: [], updatedAt: '2026-05-23T00:00:00.000Z' }, + assetManifest: [], + sourceInstanceId: 'peer-abc', + }); + expect(mergeMediaCollectionsFromSync).toHaveBeenCalledWith([ + expect.objectContaining({ id: 'col-y' }), + ]); + }); + + it('auto-creates a reverse subscription back to the sender for a syncable local collection', async () => { + // Regression (Bug 1): classifyLocalRecord had no mediaCollection branch, + // so it returned 'missing' and maybeCreateReverseSubscription's + // `localState !== 'syncable'` guard never bootstrapped bidirectional + // collection sync. The merge landed the record locally, so the + // classifyLocalRecord lookup must resolve it as 'syncable'. + vi.mocked(getCollection).mockResolvedValue({ + id: 'col-rev', name: 'Synced', items: [], updatedAt: '2026-05-23T00:00:00.000Z', + deleted: false, deletedAt: null, + }); + const result = await applyIncomingPush({ + kind: 'mediaCollection', + record: { id: 'col-rev', name: 'Synced', items: [] }, + assetManifest: [], + sourceInstanceId: 'peer-a', + }); + expect(result.reverseSubscriptionCreated).toBe(true); + const sub = await findPeerSubscription('peer-a', 'mediaCollection', 'col-rev'); + expect(sub).not.toBeNull(); + expect(sub.adoptedFromReverse).toBe(true); + }); + + it('does NOT create a reverse subscription when the local collection is missing (merge dropped it)', async () => { + // The default getCollection mock rejects (NOT_FOUND) → classifyLocalRecord + // returns 'missing' → no orphan reverse-sub. + const result = await applyIncomingPush({ + kind: 'mediaCollection', + record: { id: 'col-gone', name: 'Gone', items: [] }, + assetManifest: [], + sourceInstanceId: 'peer-a', + }); + expect(result.reverseSubscriptionCreated).toBe(false); + const sub = await findPeerSubscription('peer-a', 'mediaCollection', 'col-gone'); + expect(sub).toBeNull(); + }); + }); + + describe('peerSyncPushSchema — mediaCollection validation', () => { + it('accepts a valid mediaCollection push payload', async () => { + const { peerSyncPushSchema } = await import('../../lib/validation.js'); + expect(() => peerSyncPushSchema.parse({ + kind: 'mediaCollection', + record: { id: 'col-x', name: 'My Collection', items: [] }, + assetManifest: [], + sourceInstanceId: 'peer-abc', + })).not.toThrow(); + }); + + it('rejects a mediaCollection push payload missing sourceInstanceId', async () => { + const { peerSyncPushSchema } = await import('../../lib/validation.js'); + expect(() => peerSyncPushSchema.parse({ + kind: 'mediaCollection', + record: { id: 'col-x' }, + assetManifest: [], + })).toThrow(); + }); + }); }); diff --git a/server/services/sharing/sidecarSync.js b/server/services/sharing/sidecarSync.js new file mode 100644 index 000000000..23148cc5f --- /dev/null +++ b/server/services/sharing/sidecarSync.js @@ -0,0 +1,182 @@ +/** + * Sidecar sync helpers for federated image gen-params metadata. + * + * Each locally-generated image can have a `.metadata.json` sidecar + * stored alongside it in PATHS.images. When a peer pulls an image over + * federated sync they should also receive the gen-params sidecar so the + * image lands in their gallery with its prompt intact (not stuck in Unsorted + * with no prompts). This module provides: + * + * - `pullSidecarForImage` — fetches one sidecar from a peer's /data/images + * static mount and writes it locally. Best-effort; 404 = no sidecar on + * the sender, silently ignored. + * - `backfillMissingSidecars` — walks a list of local image filenames and + * tries each online peer until the sidecar is recovered. Drives the manual + * "Pull missing prompts" action (Unsorted view in MediaCollectionDetail + + * the sync drawer) via POST /api/peer-sync/pull-metadata. + */ + +import { existsSync } from 'fs'; +import { join } from 'path'; +import { atomicWrite, ensureDir, readJSONFile, PATHS } from '../../lib/fileUtils.js'; +import { imageSidecarName, sanitizeAssetFilename } from './buckets.js'; +import { sidecarGenParamsHash } from '../../lib/assetHash.js'; +import { isPlainObject } from '../../lib/objects.js'; +import { peerFetch } from '../../lib/peerHttpClient.js'; +import { getPeers } from '../instances.js'; +import { peerBaseUrl } from '../../lib/peerUrl.js'; + +const SIDECAR_MAX_BYTES = 256 * 1024; +// Mirror the image-pull timeout (peerSync.js ASSET_PULL_TIMEOUT_MS) so a hung +// peer connection aborts instead of holding the pull open indefinitely. +const SIDECAR_PULL_TIMEOUT_MS = 60000; + +/** + * Read a fetch Response body into a Buffer, aborting once `maxBytes` is + * exceeded. peerFetch's HTTPS shim already enforces maxBytes mid-stream, but the + * plain-HTTP path falls back to native fetch, which does NOT — a peer that lies + * about Content-Length (or uses chunked transfer) could otherwise buffer an + * unbounded body via `res.arrayBuffer()` before any post-read size check runs. + * When a ReadableStream reader is available (native fetch) we cap mid-stream; + * otherwise we fall back to arrayBuffer (the shim already bounded it, or it's a + * test mock). Returns null on over-cap or any read error (best-effort, runs + * outside the Express request lifecycle so a throw must not escape). + */ +async function readBodyCapped(res, maxBytes) { + const reader = res.body?.getReader?.(); + if (!reader) { + return res.arrayBuffer().then((ab) => Buffer.from(ab)).catch(() => null); + } + const chunks = []; + let total = 0; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + total += value.length; + if (total > maxBytes) { + await reader.cancel().catch(() => {}); + return null; + } + chunks.push(Buffer.from(value)); + } + } catch { + return null; + } + return Buffer.concat(chunks); +} + +/** + * Pull `.metadata.json` from a peer's /data/images mount and + * write it alongside the image. Best-effort: a 404 (no sidecar on the sender) + * is normal and silently ignored. + * + * Defense-in-depth: the filename is sanitized HERE (not only by callers) so the + * function is safe regardless of entry point — `backfillMissingSidecars` is a + * future client-POST surface and `encodeURIComponent` only protects the URL, + * not the local `join(PATHS.images, …)` path. A `../`-bearing filename is + * rejected before any FS op. + * + * Returns true if the sidecar was successfully fetched, parsed, and written. + */ +export async function pullSidecarForImage(peer, base, imageFilename) { + const safeName = sanitizeAssetFilename(imageFilename); + if (!safeName) return false; + const sidecarName = imageSidecarName(safeName); + const url = `${base}/data/images/${encodeURIComponent(sidecarName)}`; + // Abort a hung connection after the timeout — mirrors the image pull worker. + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), SIDECAR_PULL_TIMEOUT_MS); + const res = await peerFetch(url, { signal: controller.signal, maxBytes: SIDECAR_MAX_BYTES }) + .finally(() => clearTimeout(timeoutId)) + .catch(() => null); + if (!res || !res.ok) return false; + // Require a trustworthy content-length and refuse over-cap BEFORE buffering. + // peerFetch only enforces `maxBytes` on the HTTPS shim; the plain-HTTP path + // falls back to native fetch (no streaming cap), so without this guard an + // HTTP peer could stream an unbounded body into memory. serve-static always + // sets content-length for static files — mirrors doPullOneAsset in peerSync.js. + if (!res.headers.has('content-length')) return false; + const contentLength = Number(res.headers.get('content-length')); + if (!Number.isFinite(contentLength) || contentLength <= 0 || contentLength > SIDECAR_MAX_BYTES) return false; + // Stream with a hard cap so a peer lying about Content-Length can't blow + // memory on the native-fetch (plain-HTTP) path before the size checks below. + const buf = await readBodyCapped(res, SIDECAR_MAX_BYTES); + if (!buf || buf.length === 0 || buf.length > SIDECAR_MAX_BYTES || buf.length !== contentLength) return false; + // JSON-parse gate: a peer (or an intermediary) could serve an HTML error + // page with a 200, which we'd otherwise write as `.metadata.json` and + // corrupt the gallery's gen-params reader. Only write if the body is valid + // JSON. Wrapped in try/catch because JSON.parse throws (this runs outside the + // Express request lifecycle, so an uncaught throw would crash the worker). + let parsed; + try { + parsed = JSON.parse(buf.toString('utf8')); + } catch { + return false; + } + // Must be a JSON OBJECT, not just valid JSON. A scalar/array (`"oops"`, `[]`) + // parses fine but writing it would corrupt the gallery reader: downstream + // getOrComputeImageSha256 does `{ ...(sidecar || {}) }`, and spreading a + // truthy string yields numeric char keys (`{0:'o',1:'o',…}`). + if (!isPlainObject(parsed)) return false; + // Refuse a cache-only sidecar (just the machine-local `sha256` block, no + // gen-params): it carries no prompt worth recovering and writing it could + // clobber a prompt-bearing sidecar already on disk. sidecarGenParamsHash + // returns null exactly when nothing remains beyond the sha256 cache key. + if (sidecarGenParamsHash(parsed) === null) return false; + await ensureDir(PATHS.images); + await atomicWrite(join(PATHS.images, sidecarName), buf); + console.log(`📥 peerSync: pulled sidecar ${sidecarName} from ${peer.name || peer.instanceId}`); + return true; +} + +/** + * For each local image filename lacking a sidecar, try each online peer until + * one yields the sidecar. Returns `{ attempted, recovered }`. + * + * `filenames` should be an array of image filenames (with extension) already + * present in PATHS.images. Only images whose sidecar is absent are attempted — + * images that already have a sidecar are silently skipped. Filenames that fail + * sanitization (path traversal) are skipped entirely. + */ +export async function backfillMissingSidecars({ filenames }) { + // Skip peers the user explicitly turned off (enabled:false) or disabled sync + // for (syncEnabled:false) — the manual backfill must not contact peers the + // user opted out of, matching how syncOrchestrator gates its peer set. Both + // default-on-unless-false (mirrors useSyncIntegrity's eligibility filter). + const peers = (await getPeers().catch(() => [])).filter( + (p) => p?.enabled !== false && p?.syncEnabled !== false && p?.status === 'online' && p.instanceId + ); + let attempted = 0; + let recovered = 0; + for (const filename of Array.isArray(filenames) ? filenames : []) { + const safeName = sanitizeAssetFilename(filename); + if (!safeName) continue; + // Contract: filenames name images already present in PATHS.images. Skip any + // whose image bytes are absent — a stale/invalid filename would otherwise + // write an orphan `.metadata.json` (sidecar with no image) and inflate + // `attempted`. + if (!existsSync(join(PATHS.images, safeName))) continue; + const sidecarPath = join(PATHS.images, imageSidecarName(safeName)); + // A sidecar can exist yet carry NO prompt: getOrComputeImageSha256 writes a + // cache-only `{ sha256: {...} }` sidecar for every image it hashes during + // sync. Skipping on mere file existence would make "Pull missing prompts" a + // no-op for exactly the images this repair targets (synced-in, no gen-params). + // Only skip when the sidecar already holds real gen-params — sidecarGenParamsHash + // strips the sha256 cache key and returns null when nothing else remains. + if (existsSync(sidecarPath)) { + const existing = await readJSONFile(sidecarPath, null, { logError: false }); + if (sidecarGenParamsHash(existing) !== null) continue; + } + attempted++; + for (const peer of peers) { + const ok = await pullSidecarForImage(peer, peerBaseUrl(peer), safeName).catch(() => false); + if (ok) { + recovered++; + break; + } + } + } + console.log(`🔄 sidecar backfill: ${recovered}/${attempted} recovered from ${peers.length} peer(s)`); + return { attempted, recovered }; +} diff --git a/server/services/sharing/sidecarSync.test.js b/server/services/sharing/sidecarSync.test.js new file mode 100644 index 000000000..3f795697d --- /dev/null +++ b/server/services/sharing/sidecarSync.test.js @@ -0,0 +1,365 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { mkdir, rm, writeFile, readFile } from 'fs/promises'; +import { existsSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; + +// Mock peerFetch for network isolation. +vi.mock('../../lib/peerHttpClient.js', async () => ({ + peerFetch: vi.fn(), + peerSocketOptions: {}, +})); + +// Mock instances.js so backfillMissingSidecars can be tested without live peers. +vi.mock('../instances.js', async () => ({ + UNKNOWN_INSTANCE_ID: 'unknown', + getInstanceId: vi.fn().mockResolvedValue('test-instance'), + getPeers: vi.fn(), +})); + +// Mock peerUrl so base URL generation is deterministic in tests. +vi.mock('../../lib/peerUrl.js', async () => ({ + peerBaseUrl: vi.fn((peer) => `http://${peer.instanceId}.test:5555`), +})); + +import { PATHS } from '../../lib/fileUtils.js'; +import { peerFetch } from '../../lib/peerHttpClient.js'; +import { getPeers } from '../instances.js'; +import { pullSidecarForImage, backfillMissingSidecars } from './sidecarSync.js'; + +let tmp; +let originalImagesPath; + +beforeEach(async () => { + originalImagesPath = PATHS.images; + tmp = join(tmpdir(), `portos-sidecar-test-${Date.now()}-${Math.random()}`); + await mkdir(join(tmp, 'images'), { recursive: true }); + PATHS.images = join(tmp, 'images'); + vi.mocked(peerFetch).mockReset(); + vi.mocked(getPeers).mockReset(); +}); + +afterEach(async () => { + await rm(tmp, { recursive: true, force: true }); + PATHS.images = originalImagesPath; +}); + +const fakePeer = { instanceId: 'peer-x', name: 'Peer X' }; +const fakeBase = 'http://peer-x.test:5555'; + +describe('pullSidecarForImage', () => { + it('fetches and writes sidecar when peer returns ok', async () => { + const sidecarBody = JSON.stringify({ prompt: 'a cat', model: 'flux' }); + const buf = Buffer.from(sidecarBody); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(buf.byteLength) }), + arrayBuffer: async () => buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength), + }); + + const result = await pullSidecarForImage(fakePeer, fakeBase, 'test.png'); + expect(result).toBe(true); + + const sidecarPath = join(PATHS.images, 'test.metadata.json'); + expect(existsSync(sidecarPath)).toBe(true); + const written = JSON.parse(await readFile(sidecarPath, 'utf8')); + expect(written.prompt).toBe('a cat'); + }); + + it('returns false and writes nothing when peer returns !ok (404)', async () => { + vi.mocked(peerFetch).mockResolvedValue({ ok: false, status: 404 }); + + const result = await pullSidecarForImage(fakePeer, fakeBase, 'missing.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'missing.metadata.json'))).toBe(false); + }); + + it('returns false and does not throw when peerFetch rejects', async () => { + vi.mocked(peerFetch).mockRejectedValue(new Error('network error')); + + const result = await pullSidecarForImage(fakePeer, fakeBase, 'error.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'error.metadata.json'))).toBe(false); + }); + + it('returns false when peer returns an empty body', async () => { + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': '0' }), + arrayBuffer: async () => new ArrayBuffer(0), + }); + + const result = await pullSidecarForImage(fakePeer, fakeBase, 'empty.png'); + expect(result).toBe(false); + }); + + it('constructs the correct URL including encoded sidecar name + abort signal', async () => { + vi.mocked(peerFetch).mockResolvedValue({ ok: false, status: 404 }); + await pullSidecarForImage(fakePeer, fakeBase, 'my image.png'); + expect(peerFetch).toHaveBeenCalledWith( + `${fakeBase}/data/images/${encodeURIComponent('my image.metadata.json')}`, + expect.objectContaining({ maxBytes: expect.any(Number), signal: expect.any(AbortSignal) }) + ); + }); + + it('returns false and writes nothing when the body is not valid JSON (HTML error page guard)', async () => { + const htmlBody = Buffer.from('500 oops'); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(htmlBody.byteLength) }), + arrayBuffer: async () => htmlBody.buffer.slice(htmlBody.byteOffset, htmlBody.byteOffset + htmlBody.byteLength), + }); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'htmlpage.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'htmlpage.metadata.json'))).toBe(false); + }); + + it('returns false and writes nothing when the JSON body is not an object (scalar/array)', async () => { + // Valid JSON but a non-object — writing it would corrupt the gallery reader + // (getOrComputeImageSha256 spreads `sidecar || {}`; a string spreads to char keys). + for (const body of ['"oops"', '[1,2,3]', '42', 'true', 'null']) { + vi.mocked(peerFetch).mockReset(); + const buf = Buffer.from(body); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(buf.byteLength) }), + arrayBuffer: async () => buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength), + }); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'scalar.png'); + expect(result).toBe(false); + } + expect(existsSync(join(PATHS.images, 'scalar.metadata.json'))).toBe(false); + }); + + // Build a fetch-style Response whose body streams `chunks` via getReader(), + // and whose arrayBuffer() throws (so the test proves the streaming path is used). + const streamingRes = (contentLength, ...chunks) => { + let i = 0; + return { + ok: true, + headers: new Headers({ 'content-length': String(contentLength) }), + body: { + getReader: () => ({ + read: async () => (i < chunks.length + ? { done: false, value: chunks[i++] } + : { done: true, value: undefined }), + cancel: async () => {}, + }), + }, + arrayBuffer: async () => { throw new Error('should stream via getReader, not arrayBuffer'); }, + }; + }; + + it('streams the body via getReader (native-fetch path) and writes a valid sidecar', async () => { + const body = Buffer.from(JSON.stringify({ prompt: 'streamed' })); + vi.mocked(peerFetch).mockResolvedValue(streamingRes(body.byteLength, new Uint8Array(body))); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'streamed.png'); + expect(result).toBe(true); + const written = JSON.parse(await readFile(join(PATHS.images, 'streamed.metadata.json'), 'utf8')); + expect(written.prompt).toBe('streamed'); + }); + + it('aborts the stream and writes nothing when the body exceeds the cap despite a small Content-Length (lying peer)', async () => { + // Content-Length LIES (100 bytes, passes the cap check) but the streamed + // body is 300KB (> 256KB cap). The streaming reader must abort, not buffer. + const oversized = new Uint8Array(300 * 1024); + vi.mocked(peerFetch).mockResolvedValue(streamingRes(100, oversized)); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'liar.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'liar.metadata.json'))).toBe(false); + }); + + it('returns false and writes nothing when the peer sidecar is cache-only (sha256 block, no prompt)', async () => { + // A cache-only sidecar has no prompt to recover and could clobber a + // prompt-bearing local one — must not be written. + const cacheOnly = Buffer.from(JSON.stringify({ sha256: { value: 'a'.repeat(64), mtimeMs: 1, size: 2 } })); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(cacheOnly.byteLength) }), + arrayBuffer: async () => cacheOnly.buffer.slice(cacheOnly.byteOffset, cacheOnly.byteOffset + cacheOnly.byteLength), + }); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'cacheonly.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'cacheonly.metadata.json'))).toBe(false); + }); + + it('rejects path-traversal filenames before any fetch or FS op', async () => { + vi.mocked(peerFetch).mockResolvedValue({ ok: true, arrayBuffer: async () => Buffer.from('{}').buffer }); + const result = await pullSidecarForImage(fakePeer, fakeBase, '../../etc/passwd.png'); + expect(result).toBe(false); + // Never even hit the network for a traversal name. + expect(peerFetch).not.toHaveBeenCalled(); + }); + + it('rejects an over-cap content-length before buffering', async () => { + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(10 * 1024 * 1024) }), // 10MB >> 256KB cap + arrayBuffer: async () => { throw new Error('should not buffer an over-cap body'); }, + }); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'huge.png'); + expect(result).toBe(false); + expect(existsSync(join(PATHS.images, 'huge.metadata.json'))).toBe(false); + }); + + it('refuses when the content-length header is missing (cannot bound the body)', async () => { + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({}), + arrayBuffer: async () => Buffer.from('{}').buffer, + }); + const result = await pullSidecarForImage(fakePeer, fakeBase, 'noheader.png'); + expect(result).toBe(false); + }); +}); + +describe('backfillMissingSidecars', () => { + it('attempts only images without sidecars and skips already-present ones', async () => { + // Create two images: one with a real gen-params sidecar already, one bare. + await writeFile(join(PATHS.images, 'with-sidecar.png'), Buffer.from('img1')); + await writeFile( + join(PATHS.images, 'with-sidecar.metadata.json'), + Buffer.from(JSON.stringify({ prompt: 'already has a prompt' })), + ); + await writeFile(join(PATHS.images, 'bare.png'), Buffer.from('img2')); + + const sidecarBuf = Buffer.from(JSON.stringify({ prompt: 'recovered' })); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(sidecarBuf.byteLength) }), + arrayBuffer: async () => sidecarBuf.buffer.slice( + sidecarBuf.byteOffset, sidecarBuf.byteOffset + sidecarBuf.byteLength + ), + }); + + const result = await backfillMissingSidecars({ + filenames: ['with-sidecar.png', 'bare.png'], + }); + + // Only the bare image was attempted. + expect(result.attempted).toBe(1); + expect(result.recovered).toBe(1); + // The sidecar that was already present was NOT re-fetched. + expect(peerFetch).toHaveBeenCalledTimes(1); + // The recovered sidecar is now on disk. + expect(existsSync(join(PATHS.images, 'bare.metadata.json'))).toBe(true); + }); + + it('returns { attempted: 0, recovered: 0 } when all sidecars already have gen-params', async () => { + await writeFile(join(PATHS.images, 'all.png'), Buffer.from('img')); + await writeFile( + join(PATHS.images, 'all.metadata.json'), + Buffer.from(JSON.stringify({ prompt: 'p', model: 'flux' })), + ); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + + const result = await backfillMissingSidecars({ filenames: ['all.png'] }); + expect(result.attempted).toBe(0); + expect(result.recovered).toBe(0); + expect(peerFetch).not.toHaveBeenCalled(); + }); + + it('treats a cache-only sidecar (just sha256, no prompt) as missing and attempts a pull', async () => { + // getOrComputeImageSha256 writes this exact shape for every image it hashes + // during sync — it has NO prompt, so "Pull missing prompts" must still try. + await writeFile(join(PATHS.images, 'cacheonly.png'), Buffer.from('img')); + await writeFile( + join(PATHS.images, 'cacheonly.metadata.json'), + Buffer.from(JSON.stringify({ sha256: { value: 'a'.repeat(64), mtimeMs: 1, size: 3 } })), + ); + const recoveredBuf = Buffer.from(JSON.stringify({ prompt: 'recovered prompt' })); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + vi.mocked(peerFetch).mockResolvedValue({ + ok: true, + headers: new Headers({ 'content-length': String(recoveredBuf.byteLength) }), + arrayBuffer: async () => recoveredBuf.buffer.slice( + recoveredBuf.byteOffset, recoveredBuf.byteOffset + recoveredBuf.byteLength, + ), + }); + + const result = await backfillMissingSidecars({ filenames: ['cacheonly.png'] }); + expect(result.attempted).toBe(1); + expect(result.recovered).toBe(1); + expect(peerFetch).toHaveBeenCalledTimes(1); + // The cache-only sidecar was overwritten with the real prompt metadata. + const written = JSON.parse(await readFile(join(PATHS.images, 'cacheonly.metadata.json'), 'utf8')); + expect(written.prompt).toBe('recovered prompt'); + }); + + it('returns { attempted: 1, recovered: 0 } when no peer has the sidecar', async () => { + await writeFile(join(PATHS.images, 'bare2.png'), Buffer.from('img')); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + vi.mocked(peerFetch).mockResolvedValue({ ok: false, status: 404 }); + + const result = await backfillMissingSidecars({ filenames: ['bare2.png'] }); + expect(result.attempted).toBe(1); + expect(result.recovered).toBe(0); + }); + + it('skips offline peers', async () => { + await writeFile(join(PATHS.images, 'bare3.png'), Buffer.from('img')); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-offline', name: 'Offline', status: 'offline' }, + ]); + + const result = await backfillMissingSidecars({ filenames: ['bare3.png'] }); + expect(result.attempted).toBe(1); + expect(result.recovered).toBe(0); + expect(peerFetch).not.toHaveBeenCalled(); + }); + + it('skips peers the user turned off (enabled:false or syncEnabled:false)', async () => { + await writeFile(join(PATHS.images, 'bare-off.png'), Buffer.from('img')); + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-disabled', name: 'Disabled', status: 'online', enabled: false }, + { instanceId: 'peer-nosync', name: 'NoSync', status: 'online', syncEnabled: false }, + ]); + + const result = await backfillMissingSidecars({ filenames: ['bare-off.png'] }); + expect(result.attempted).toBe(1); + expect(result.recovered).toBe(0); + // Neither opted-out peer is contacted. + expect(peerFetch).not.toHaveBeenCalled(); + }); + + it('skips filenames whose image bytes are not on disk (no orphan sidecar, no fetch)', async () => { + // Image file intentionally NOT written — only a stale filename is passed. + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + const result = await backfillMissingSidecars({ filenames: ['ghost.png'] }); + expect(result.attempted).toBe(0); + expect(result.recovered).toBe(0); + expect(peerFetch).not.toHaveBeenCalled(); + // No orphan sidecar was written for the absent image. + expect(existsSync(join(PATHS.images, 'ghost.metadata.json'))).toBe(false); + }); + + it('handles non-array filenames gracefully (returns zeros)', async () => { + vi.mocked(getPeers).mockResolvedValue([]); + const result = await backfillMissingSidecars({ filenames: null }); + expect(result.attempted).toBe(0); + expect(result.recovered).toBe(0); + }); + + it('skips path-traversal filenames (never attempts them)', async () => { + vi.mocked(getPeers).mockResolvedValue([ + { instanceId: 'peer-a', name: 'Peer A', status: 'online' }, + ]); + const result = await backfillMissingSidecars({ + filenames: ['../../etc/passwd.png', 'sub/dir/asset.png'], + }); + expect(result.attempted).toBe(0); + expect(result.recovered).toBe(0); + expect(peerFetch).not.toHaveBeenCalled(); + }); +}); diff --git a/server/services/sharing/tombstoneGc.js b/server/services/sharing/tombstoneGc.js index 8ce7297d2..b89c907d3 100644 --- a/server/services/sharing/tombstoneGc.js +++ b/server/services/sharing/tombstoneGc.js @@ -29,6 +29,7 @@ import { pruneTombstonedUniverses } from '../universeBuilder.js'; import { pruneTombstonedSeries } from '../pipeline/series.js'; import { pruneTombstonedIssues } from '../pipeline/issues.js'; +import { pruneTombstonedCollections } from '../mediaCollections.js'; import { listPeerSubscriptions } from './peerSync.js'; import { getMinAckAcrossPeers } from './peerTombstoneCursors.js'; import { getPeers } from '../instances.js'; @@ -61,6 +62,7 @@ function peerIdsSubscribedToKind(subs, peers, recordKind) { function snapshotCategoryForKind(recordKind) { if (recordKind === 'universe') return 'universe'; if (recordKind === 'series' || recordKind === 'issue') return 'pipeline'; + if (recordKind === 'mediaCollection') return 'mediaCollections'; return null; } @@ -113,7 +115,7 @@ async function loadState() { return { peers, subs }; } -function refusedFromCutoffs(universeCutoff, seriesCutoff) { +function refusedFromCutoffs(universeCutoff, seriesCutoff, collectionCutoff) { const refused = []; if (universeCutoff === null) refused.push('universe'); // Issue tombstones ride series pushes — refused exactly when series is. @@ -121,11 +123,12 @@ function refusedFromCutoffs(universeCutoff, seriesCutoff) { refused.push('series'); refused.push('issue'); } + if (collectionCutoff === null) refused.push('mediaCollection'); return refused; } /** - * One sweep cycle. Returns `{ universes, series, issues, refused }`. + * One sweep cycle. Returns `{ universes, series, issues, collections, refused }`. * * `graceMs` defaults to 24h so the orchestrator path is unchanged; the * manual-trigger UI / CLI passes 0 to skip the post-delete buffer. The @@ -134,21 +137,24 @@ function refusedFromCutoffs(universeCutoff, seriesCutoff) { */ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } = {}) { const { peers, subs } = await loadState(); - const [universeCutoff, seriesCutoff] = await Promise.all([ + const [universeCutoff, seriesCutoff, collectionCutoff] = await Promise.all([ cutoffForKind('universe', { peers, subs, now, graceMs }), cutoffForKind('series', { peers, subs, now, graceMs }), + cutoffForKind('mediaCollection', { peers, subs, now, graceMs }), ]); const issueCutoff = seriesCutoff; - const [u, s, i] = await Promise.all([ + const [u, s, i, c] = await Promise.all([ universeCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedUniverses(universeCutoff), seriesCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedSeries(seriesCutoff), issueCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedIssues(issueCutoff), + collectionCutoff === null ? Promise.resolve({ pruned: 0 }) : pruneTombstonedCollections(collectionCutoff), ]); return { universes: u.pruned, series: s.pruned, issues: i.pruned, - refused: refusedFromCutoffs(universeCutoff, seriesCutoff), + collections: c.pruned, + refused: refusedFromCutoffs(universeCutoff, seriesCutoff, collectionCutoff), }; } @@ -157,11 +163,12 @@ export async function sweepTombstones({ now = Date.now(), graceMs = GRACE_MS } = // coverage matters), so this hardcodes graceMs:0 internally. export async function getSweepStatus({ now = Date.now() } = {}) { const { peers, subs } = await loadState(); - const [universeCutoff, seriesCutoff] = await Promise.all([ + const [universeCutoff, seriesCutoff, collectionCutoff] = await Promise.all([ cutoffForKind('universe', { peers, subs, now, graceMs: 0 }), cutoffForKind('series', { peers, subs, now, graceMs: 0 }), + cutoffForKind('mediaCollection', { peers, subs, now, graceMs: 0 }), ]); - return { refused: refusedFromCutoffs(universeCutoff, seriesCutoff) }; + return { refused: refusedFromCutoffs(universeCutoff, seriesCutoff, collectionCutoff) }; } export const TOMBSTONE_GRACE_MS = GRACE_MS; diff --git a/server/services/sharing/tombstoneGc.test.js b/server/services/sharing/tombstoneGc.test.js index bfec07e5d..14febaa2f 100644 --- a/server/services/sharing/tombstoneGc.test.js +++ b/server/services/sharing/tombstoneGc.test.js @@ -13,6 +13,9 @@ vi.mock('../pipeline/series.js', () => ({ vi.mock('../pipeline/issues.js', () => ({ pruneTombstonedIssues: vi.fn().mockResolvedValue({ pruned: 0 }), })); +vi.mock('../mediaCollections.js', () => ({ + pruneTombstonedCollections: vi.fn().mockResolvedValue({ pruned: 0 }), +})); vi.mock('./peerSync.js', () => ({ listPeerSubscriptions: vi.fn(), })); @@ -31,6 +34,7 @@ import { import { pruneTombstonedUniverses } from '../universeBuilder.js'; import { pruneTombstonedSeries } from '../pipeline/series.js'; import { pruneTombstonedIssues } from '../pipeline/issues.js'; +import { pruneTombstonedCollections } from '../mediaCollections.js'; import { listPeerSubscriptions } from './peerSync.js'; import { getMinAckAcrossPeers } from './peerTombstoneCursors.js'; import { getPeers } from '../instances.js'; @@ -63,12 +67,13 @@ describe('TOMBSTONE_GRACE_MS', () => { }); describe('sweepTombstones — no peers subscribed', () => { - it('uses now-GRACE as the cutoff for all three kinds when nobody is subscribed', async () => { + it('uses now-GRACE as the cutoff for all four kinds when nobody is subscribed', async () => { await sweepTombstones({ now: NOW }); const expectedCutoff = NOW - TOMBSTONE_GRACE_MS + 1; expect(pruneTombstonedUniverses).toHaveBeenCalledWith(expectedCutoff); expect(pruneTombstonedSeries).toHaveBeenCalledWith(expectedCutoff); expect(pruneTombstonedIssues).toHaveBeenCalledWith(expectedCutoff); + expect(pruneTombstonedCollections).toHaveBeenCalledWith(expectedCutoff); }); }); @@ -201,20 +206,22 @@ describe('sweepTombstones — return shape', () => { pruneTombstonedUniverses.mockResolvedValueOnce({ pruned: 2 }); pruneTombstonedSeries.mockResolvedValueOnce({ pruned: 0 }); pruneTombstonedIssues.mockResolvedValueOnce({ pruned: 5 }); + pruneTombstonedCollections.mockResolvedValueOnce({ pruned: 3 }); const result = await sweepTombstones({ now: NOW }); - expect(result).toEqual({ universes: 2, series: 0, issues: 5, refused: [] }); + expect(result).toEqual({ universes: 2, series: 0, issues: 5, collections: 3, refused: [] }); }); it('lists kinds whose cutoff was null in `refused` so the manual-trigger UI can explain why nothing pruned', async () => { listPeerSubscriptions.mockResolvedValue([]); getPeers.mockResolvedValue([ - { instanceId: 'peer-a', enabled: true, syncCategories: { universe: true, pipeline: true } }, + { instanceId: 'peer-a', enabled: true, syncCategories: { universe: true, pipeline: true, mediaCollections: true } }, ]); const result = await sweepTombstones({ now: NOW }); - expect(result.refused.sort()).toEqual(['issue', 'series', 'universe']); + expect(result.refused.sort()).toEqual(['issue', 'mediaCollection', 'series', 'universe']); expect(result.universes).toBe(0); expect(result.series).toBe(0); expect(result.issues).toBe(0); + expect(result.collections).toBe(0); }); }); @@ -282,13 +289,14 @@ describe('getSweepStatus — dry-run for UI button gating', () => { it('returns refused kinds without invoking any prune helper', async () => { listPeerSubscriptions.mockResolvedValue([]); getPeers.mockResolvedValue([ - { instanceId: 'peer-a', enabled: true, syncCategories: { universe: true, pipeline: true } }, + { instanceId: 'peer-a', enabled: true, syncCategories: { universe: true, pipeline: true, mediaCollections: true } }, ]); const result = await getSweepStatus({ now: NOW }); - expect(result.refused.sort()).toEqual(['issue', 'series', 'universe']); + expect(result.refused.sort()).toEqual(['issue', 'mediaCollection', 'series', 'universe']); expect(pruneTombstonedUniverses).not.toHaveBeenCalled(); expect(pruneTombstonedSeries).not.toHaveBeenCalled(); expect(pruneTombstonedIssues).not.toHaveBeenCalled(); + expect(pruneTombstonedCollections).not.toHaveBeenCalled(); }); it('returns an empty refused list when every kind has an ack horizon', async () => { @@ -380,3 +388,47 @@ describe('sweepTombstones — resurrection safety against snapshot-mode peers', expect(pruneTombstonedSeries).toHaveBeenCalled(); }); }); + +describe('sweepTombstones — mediaCollection GC (Task 1.10b)', () => { + it('prunes collection tombstones when no peers are subscribed (grace satisfied)', async () => { + // No subs → peerIdsSubscribedToKind returns [] → getMinAckAcrossPeers([]) + // returns Infinity → cutoff = min(Infinity, NOW) - GRACE + 1 = NOW - GRACE + 1. + // pruneTombstonedCollections is called; universe/series/issues also run with + // their own grace-only cutoffs. + await sweepTombstones({ now: NOW }); + expect(pruneTombstonedCollections).toHaveBeenCalledWith(NOW - TOMBSTONE_GRACE_MS + 1); + }); + + it('does NOT prune collection tombstones when a subscribed peer has not acked', async () => { + // Peer-a has a per-record mediaCollection sub but its ack cursor is far + // behind. The cutoff must clamp to the peer's ack water-mark so we can't + // prune a tombstone the peer hasn't received yet. + const minAck = NOW - 48 * 60 * 60 * 1000; // 48h behind now + mockSubs({ mediaCollection: [{ peerId: 'peer-a' }] }); + getPeers.mockResolvedValue([{ instanceId: 'peer-a', enabled: true }]); + getMinAckAcrossPeers.mockImplementation(async (peerIds) => { + if (peerIds.includes('peer-a')) return minAck; + return Infinity; + }); + await sweepTombstones({ now: NOW }); + // Cutoff must be clamped to the laggiest peer's ack, not now-GRACE. + expect(pruneTombstonedCollections).toHaveBeenCalledWith(minAck - TOMBSTONE_GRACE_MS + 1); + // Verify this is older than the grace-only cutoff — i.e. we DID clamp. + const calledWith = pruneTombstonedCollections.mock.calls[0][0]; + expect(calledWith).toBeLessThan(NOW - TOMBSTONE_GRACE_MS + 1); + }); + + it('refuses to prune collection tombstones when a snapshot-mode peer exists for mediaCollections but has no per-record sub', async () => { + // A snapshot-only peer for mediaCollections can resurrect a pruned record + // via its next snapshot push — must refuse the prune until a per-record sub + // gives us an ack horizon. + listPeerSubscriptions.mockResolvedValue([]); + getPeers.mockResolvedValue([ + { instanceId: 'peer-a', enabled: true, syncCategories: { mediaCollections: true } }, + ]); + const result = await sweepTombstones({ now: NOW }); + expect(pruneTombstonedCollections).not.toHaveBeenCalled(); + expect(result.refused).toContain('mediaCollection'); + expect(result.collections).toBe(0); + }); +}); diff --git a/server/services/syncOrchestrator.js b/server/services/syncOrchestrator.js index b19616883..76aeb59b5 100644 --- a/server/services/syncOrchestrator.js +++ b/server/services/syncOrchestrator.js @@ -367,6 +367,7 @@ async function categoriesCoveredByPeerSync(peerId) { for (const sub of subs) { if (sub.recordKind === 'universe') skip.add('universe'); if (sub.recordKind === 'series') skip.add('pipeline'); + if (sub.recordKind === 'mediaCollection') skip.add('mediaCollections'); } return skip; } @@ -574,11 +575,12 @@ async function runTombstoneSweep() { console.error(`❌ Tombstone sweep failed: ${err.message}`); return null; }); - if (result && (result.universes > 0 || result.series > 0 || result.issues > 0)) { + if (result && (result.universes > 0 || result.series > 0 || result.issues > 0 || result.collections > 0)) { // "series" is already its own plural so no s-suffix toggle needed there. const universes = `${result.universes} universe${result.universes === 1 ? '' : 's'}`; const issues = `${result.issues} issue${result.issues === 1 ? '' : 's'}`; - console.log(`🪦 Tombstone GC: pruned ${universes}, ${result.series} series, ${issues}`); + const collections = `${result.collections} collection${result.collections === 1 ? '' : 's'}`; + console.log(`🪦 Tombstone GC: pruned ${universes}, ${result.series} series, ${issues}, ${collections}`); } } diff --git a/server/services/syncOrchestrator.test.js b/server/services/syncOrchestrator.test.js index a1fee46c9..d099e4470 100644 --- a/server/services/syncOrchestrator.test.js +++ b/server/services/syncOrchestrator.test.js @@ -290,6 +290,27 @@ describe('syncOrchestrator', () => { expect(urls.some((u) => u.includes('/api/sync/pipeline/'))).toBe(false); expect(urls.some((u) => u.includes('/api/sync/character/'))).toBe(true); }); + + it('skips mediaCollections snapshot category when peer has a mediaCollection per-record subscription', async () => { + // Task 1.9: a mediaCollection subscription must prevent the 60s snapshot + // loop from also hitting /api/sync/mediaCollections/ — the per-record + // push pipeline owns that category. + const dataSync = await import('./dataSync.js'); + const peerSync = await import('./sharing/peerSync.js'); + dataSync.getSupportedCategories.mockReturnValue(['mediaCollections', 'character']); + peerSync.listPeerSubscriptions.mockResolvedValueOnce([ + { peerId: 'peer-inst-1', recordKind: 'mediaCollection', recordId: 'col-1' }, + ]); + const peerWithCats = { + ...mockPeer, + syncCategories: { mediaCollections: true, character: true }, + }; + mockFetch.mockResolvedValue({ ok: true, json: async () => ({ checksum: 'x', data: null }) }); + await syncWithPeer(peerWithCats); + const urls = mockFetch.mock.calls.map((c) => c[0]); + expect(urls.some((u) => u.includes('/api/sync/mediaCollections/'))).toBe(false); + expect(urls.some((u) => u.includes('/api/sync/character/'))).toBe(true); + }); }); describe('syncAllPeers', () => {