Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
26cd376
docs(sync): federated media collection sync parity + per-category int…
atomantic May 24, 2026
c0ecdef
docs(sync): implementation plan for federated media sync parity + int…
atomantic May 24, 2026
091c115
feat(sync): soft-delete shape + includeDeleted on media collections
atomantic May 24, 2026
ecb58e0
feat(sync): soft-delete deleteCollection + tombstone-aware LWW merge
atomantic May 24, 2026
e4f0afe
feat(sync): register mediaCollection kind + createCollection auto-sub…
atomantic May 24, 2026
67308cc
feat(sync): mediaCollection push payload + receiver apply + subscribe…
atomantic May 24, 2026
68aca03
fix(sync): mediaCollection reverse-subscription + video asset manifest
atomantic May 24, 2026
d2de86e
feat(sync): orchestrator snapshot-skip + tombstone GC for media colle…
atomantic May 24, 2026
344e9ba
fix(sync): count pruned collections in manual tombstone-sweep UI
atomantic May 24, 2026
72d422d
feat(sync): sync image gen-params sidecars + manual backfill
atomantic May 24, 2026
00992f1
fix(sync): hash sidecar gen-params canonically + sanitize sidecar pul…
atomantic May 24, 2026
e582fea
feat(sync): per-category integrity diff + peer manifest + routes
atomantic May 24, 2026
184b71c
feat(sync): manual force-push, sync-now, and pull-metadata routes
atomantic May 24, 2026
4d49d00
feat(sync): client peer-sync API wrappers + useSyncIntegrity hook
atomantic May 24, 2026
1d08a3c
feat(sync): SyncBadge + deep-linkable SyncDetailDrawer + media collec…
atomantic May 24, 2026
73ec68b
fix(sync): drawer scroll-lock + single-fetch collection state (no dou…
atomantic May 24, 2026
62248c4
feat(sync): sync badges + detail drawer on universes and pipeline series
atomantic May 24, 2026
d862b59
feat(sync): Pull missing prompts action on the Unsorted media view
atomantic May 24, 2026
405b853
test(sync): run useSyncIntegrity hook test in jsdom only (.test.jsx)
atomantic May 24, 2026
eea1b34
chore(sync): changelog + PLAN follow-ups + minor cleanups (unused sch…
atomantic May 24, 2026
c0f478e
chore(sync): align client PEER_SUBSCRIBABLE_KINDS with server (add me…
atomantic May 24, 2026
4585ba8
Merge remote-tracking branch 'origin/main' into feat/federated-media-…
atomantic May 24, 2026
91d8c3d
fix(sync): address Copilot review on #468
atomantic May 24, 2026
e70b887
fix(sync): address Copilot re-review on #468
atomantic May 24, 2026
dcdf6ab
fix(sync): address Copilot 3rd re-review on #468
atomantic May 24, 2026
0b165a6
fix(sync): address Copilot 4th re-review on #468
atomantic May 24, 2026
cc90ab6
fix(sync): address Copilot 5th re-review on #468
atomantic May 24, 2026
93078c7
fix(sync): address Copilot 6th re-review on #468
atomantic May 24, 2026
b970430
fix(sync): address Copilot 7th re-review on #468
atomantic May 24, 2026
0a5f8bd
fix(sync): address Copilot 8th re-review on #468
atomantic May 24, 2026
c9cf193
fix(sync): address Copilot 9th re-review on #468
atomantic May 24, 2026
a5c4612
fix(sync): address Copilot 10th re-review on #468
atomantic May 24, 2026
3401f42
fix(sync): address Copilot 11th re-review on #468
atomantic May 24, 2026
9de0239
fix(sync): address Copilot 12th re-review on #468
atomantic May 24, 2026
1116d0c
fix(sync): address Copilot 13th re-review on #468
atomantic May 24, 2026
7e023b6
fix(sync): address Copilot 14th re-review on #468
atomantic May 24, 2026
c6e93fb
fix(sync): address Copilot 15th re-review on #468
atomantic May 24, 2026
5c33d43
fix(sync): address Copilot 16th re-review on #468
atomantic May 24, 2026
68c2abc
fix(sync): address Copilot 17th re-review on #468
atomantic May 24, 2026
9b5493d
fix(sync): address Copilot 18th re-review on #468
atomantic May 24, 2026
7c11e6e
fix(sync): address Copilot 19th re-review on #468
atomantic May 24, 2026
3192537
fix(sync): address Copilot 20th re-review on #468
atomantic May 24, 2026
d73f682
test(sync): fix flaky forcePushRecord test that failed in CI (not local)
atomantic May 24, 2026
f30b10e
fix(sync): address Copilot 21st re-review on #468
atomantic May 24, 2026
c08825a
fix(sync): address Copilot 22nd re-review on #468
atomantic May 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changelog/NEXT.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Unreleased Changes

## Added
- **Federated media collections + per-category sync integrity.** Media collections are now a first-class peer-sync record kind (`mediaCollection` in `PEER_SUBSCRIBABLE_KINDS`, schema-version-gated at `mediaCollections: 1`): they auto-subscribe on create, push per-record on edit, and propagate deletes via tombstones — collections now reach parity between federated machines the way universes/series already did, instead of only riding along as `linkedCollection` cargo or the opt-in 60s snapshot. Required adding soft-delete to collections (`deleted`/`deletedAt`; `deleteCollection` marks instead of splicing; `listCollections({ includeDeleted })`; tombstone-aware LWW merge; tombstone GC), so a delete on one node can't be resurrected by a reverse-subscribed peer. Synced images now carry their **generation prompts**: the peer asset manifest advertises a canonical gen-params-only `sidecarSha256` (machine-local hash-cache stripped so it converges across peers), and the asset-pull worker fetches each image's `.metadata.json` sidecar alongside the bytes — fixing images that previously landed in "Unsorted" with no prompts. A new per-category **sync integrity** surface (`GET /api/peer-sync/manifest`, `GET /api/peer-sync/integrity`, pure `computeRecordIntegrity` diff) drives a `SyncBadge` on every Universe / Series / Media-Collection row (in-sync / diverged / assets-missing / local-only / on-peer-only / "not syncing — enable?") plus a deep-linkable `SyncDetailDrawer` (`/…/:id/sync`) showing the per-peer breakdown, previews, and manual actions: **Sync to peer** (`POST /sync-record`, force-push bypassing the unchanged-hash short-circuit), **Sync now** (`POST /sync-now`, backfill-subscribe + retry a peer's pending pushes), and **Pull missing prompts** (`POST /pull-metadata` + a button on the Unsorted view that backfills sidecars from online peers). `mediaCollections` sync stays **opt-in** per peer (default off); the badge surfaces a clear "enable?" state rather than flipping the default.
- **Manual tombstone GC trigger.** The 60s `syncOrchestrator` sweep buffers tombstones for 24h to give every peer time to ack a deletion before the bytes leave disk — fine for normal use, painful right after a mass-delete. New `POST /api/sync/tombstones/sweep` (Zod-validated optional `{ graceMs }` clamped to `[0, 24h]`) calls `sweepTombstones({ graceMs })` so callers can shrink the buffer; the per-kind null-cutoff refusal (snapshot-mode peer with no per-record subscription) still fires independently of `graceMs` to preserve resurrection safety. Companion `GET /api/sync/tombstones/status` returns the dry-run refusal status without invoking the prune helpers. A new "GC tombstones now" section on the Instances page (between SelfCard/AddPeer and the peers list) POSTs `graceMs: 0` and surfaces per-kind prune counts + any refusals in a toast; the button is disabled only when every kind is refused. `scripts/gc-tombstones-now.js` is the equivalent CLI path (dry-run by default; `--apply` to prune) for headless cleanups — uses the same "stop the server first, single-process write-tail isn't cross-process safe" wording as `cleanup-test-data.js`. `sweepTombstones` now returns `{ universes, series, issues, refused }` so the manual trigger can explain why a kind didn't shrink. The orchestrator path keeps the 24h default behavior unchanged.
- `server/lib/collectionStore.js` — per-type, per-record JSON storage helper with explicit type-level `schemaVersion` stamping. `createCollectionStore({ dir, type, schemaVersion, sanitizeRecord })` returns `loadOne` / `saveOne` / `listIds` / `loadAll` / `deleteOne` / `loadTypeIndex` / `saveTypeIndex` / `verifySchemaVersion`. Per-id write queue means writes to different records run in parallel — the win over a single-file write queue. `verifyCollectionVersions([store, ...])` is called at server boot after `runMigrations()` so a missed migration produces a loud, single-line log per collection (does not crash — PortOS is single-user).
- New layout `data/universes/{id}/index.json` for the universe builder, plus a type-level `data/universes/index.json` carrying `{ schemaVersion: 5, type, updatedAt, config: { runs } }`. Migration 034 (`scripts/migrations/034-split-universe-builder-to-per-uuid.js`) splits the legacy monolithic `data/universe-builder.json` (1.4MB / ~30 universes that previously rewrote in full on every mutation) into per-record files at boot. The legacy file is renamed to `universe-builder.json.bak-034` (not deleted) as a recovery path. Idempotent — partial-completion re-runs finish the split; full-completion re-runs are no-ops.
Expand Down
11 changes: 11 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,17 @@ Items here need a research / design pass, an explicit decision, or a preconditio
- [ ] [cos-on-demand-mark-app-review-started-dedupe] **`evaluateTasks` on-demand loop — dedupe `markAppReviewStarted` calls per app.** Surfaced by gemini review (2026-05-23) at `server/services/cos.js:730`. When multiple on-demand requests for the same app are queued in a single evaluation cycle, `markAppReviewStarted` is called once per request — wasted state writes (single-user / single-instance so not a correctness bug, just churn). Collect unique `appId`s before the loop and call once. Nit-level priority.
- [ ] [subagent-spawner-top-level-side-effects] **`subAgentSpawner.js` top-level side effects — move init into an explicit `initSpawner()`.** Surfaced by gemini review (2026-05-23) at `server/services/subAgentSpawner.js:153`. Module-load wires event listeners + timers, which interferes with test isolation (each test import re-arms them). Wrap the init block in `export function initSpawner()` and call it from `server/index.js` after dependencies are ready. Treat as a refactor — needs care that no consumer of the module relies on the import-time wiring.
- [ ] [ref-watch-phosphene-long-clip-temporal-boost-12fps] **"Long Clip Temporal Boost" (render at 12fps, interpolate to 24fps).** From `reference-watch` review of phosphene (commit `53585fc`, 2026-05-22). Phosphene added a 12fps render path that uses ffmpeg's `minterpolate` / RIFE downstream to fill back to 24fps, doubling the effective clip length per memory budget. PortOS doesn't have an interpolation post-step in `server/services/videoGen/local.js` — adding one is a feature decision, not a bug fix. The math helpers (`_duration_to_8k_frames`, LTX `8k+1` frame-count snapping) are useful regardless; PortOS's `DEFAULT_NUM_FRAMES = 121` is already 8k+1-aligned but UI-side validation could borrow this pattern. **Decision needed:** is this worth a feature surface? No clear user pain motivating it today.

### Federated media sync — deferred follow-ups (2026-05-23)

Surfaced during code review of the federated-media-collection-sync feature and consciously deferred (out of scope for that PR). Each is independently pickable.

- [ ] [media-sync-snapshot-tombstones] **dataSync `mediaCollections` snapshot omits tombstones.** `getMediaCollectionsSnapshot` (`server/services/dataSync.js`) lists live-only collections, so a peer with the `mediaCollections` category enabled but no per-record subscription yet (transient window before auto-subscribe lands) won't receive collection tombstones via the 60s snapshot — only via the push path. Either include tombstones in the snapshot (bounded by a horizon, like the universe/series snapshots) or document that first-class push is the authoritative tombstone path for collections.
- [ ] [media-sync-integrity-unreachable-reason-ui] **Integrity: surface the `peer-unreachable` vs `peer-too-old` reason in the UI.** Server-side is done — `getPeerIntegrity` (`server/services/sharing/integrity.js`) now returns `reason: 'peer-unreachable'` for a network failure, `'peer-too-old'` for a 404, and `'fetch-failed'` for other non-OK statuses. The remaining gap is client-side: `useSyncIntegrity` only checks `available` and silently drops the `reason`, so the badge/drawer can't tell the user "peer is offline" vs "update the peer's PortOS." Thread `reason` through the hook's per-peer payload and render distinct messaging in `SyncBadge`/`SyncDetailDrawer`.
- [ ] [media-sync-series-issue-assets-manifest] **Series integrity manifest omits child-issue assets (v1 limitation).** `assetShaListForRecord('series', …)` (`server/services/sharing/peerSync.js`) uses `buildAssetManifest(series)`, capturing only the series' own asset refs, not its issues' — so a series whose images all live in issues reports assets in-parity even when a peer is missing them. Extend to fold in child-issue asset hashes (mirror `buildAssetManifestForSeries`).
- [ ] [media-sync-metadata-missing-status] **(Enhancement) `metadata-missing` integrity status.** Surface records whose images lack gen-params sidecars as a distinct status so the badge directly flags the "no prompts" condition (today repaired via the Unsorted "Pull missing prompts" action; byte-divergence shows as `assets-missing`). Requires the peer manifest to carry per-record sidecar-presence. `server/lib/syncIntegrity.js` + the manifest builder.
- [ ] [media-sync-drawer-test-act-warning] **Test polish: `SyncDetailDrawer.test.jsx` act() warning.** The universe-kind "shows per-peer breakdown" test renders without awaiting the async record-name fetch, emitting a React `act()` warning (test still passes). Wrap the render/assertions in `waitFor`/`act`. Test-quality only.

## Future Ideas

- **Identity Context Injection** — per-task-type digital twin preamble toggle.
Expand Down
5 changes: 5 additions & 0 deletions client/src/App.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const VideoGen = lazyWithReload(() => import('./pages/VideoGen'));
const MediaHistory = lazyWithReload(() => import('./pages/MediaHistory'));
const MediaCollections = lazyWithReload(() => import('./pages/MediaCollections'));
const MediaCollectionDetail = lazyWithReload(() => import('./pages/MediaCollectionDetail'));
const MediaCollectionSyncView = lazyWithReload(() => import('./pages/MediaCollectionSyncView'));
const SyncView = lazyWithReload(() => import('./pages/SyncView'));
const MediaModels = lazyWithReload(() => import('./pages/MediaModels'));
const Loras = lazyWithReload(() => import('./pages/Loras'));
const UniverseBuilder = lazyWithReload(() => import('./pages/UniverseBuilder'));
Expand Down Expand Up @@ -218,6 +220,7 @@ export default function App() {
<Route path="history" element={<MediaHistory />} />
<Route path="collections" element={<MediaCollections />} />
<Route path="collections/:id" element={<MediaCollectionDetail />} />
<Route path="collections/:id/sync" element={<MediaCollectionSyncView />} />
<Route path="creative-director" element={<CreativeDirector />} />
<Route path="creative-director/:id" element={<Navigate to="overview" replace />} />
<Route path="creative-director/:id/:tab" element={<CreativeDirectorDetail />} />
Expand Down Expand Up @@ -246,6 +249,7 @@ export default function App() {
<Route path="universes" element={<Universes />} />
<Route path="universes/new" element={<UniverseBuilder />} />
<Route path="universes/:universeId" element={<UniverseBuilder />} />
<Route path="universes/:universeId/sync" element={<SyncView kind="universe" param="universeId" backPath="/universes" />} />
<Route path="universes/:universeId/canon" element={<CanonRedirect />} />
{/* Legacy /universe-builder* → /universes* (route renamed when the
index landed). Keeps old bookmarks + in-app deep-links working. */}
Expand All @@ -258,6 +262,7 @@ export default function App() {
<Route path="importer" element={<Importer />} />
<Route path="pipeline" element={<Pipeline />} />
<Route path="pipeline/series/:seriesId" element={<PipelineSeries />} />
<Route path="pipeline/series/:seriesId/sync" element={<SyncView kind="series" param="seriesId" backPath="/pipeline" />} />
<Route path="pipeline/issues/:issueId" element={<Navigate to="idea" replace />} />
<Route path="pipeline/issues/:issueId/:stage" element={<PipelineIssue />} />
<Route path="writers-room/works/:workId" element={<WritersRoom />} />
Expand Down
80 changes: 80 additions & 0 deletions client/src/components/sync/SyncBadge.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/**
* SyncBadge — presentational-only sync status badge.
*
* Props:
* status — 'in-parity' | 'diverged' | 'assets-missing' | 'local-only' |
* 'peer-only' | 'not-syncing' | 'unknown' | null | undefined
* onClick — called when the badge button is clicked (e.g. open detail drawer)
*/

import { CheckCircle2, AlertTriangle, WifiOff, HelpCircle } from 'lucide-react';

const STATUS_CONFIG = {
'in-parity': {
label: 'In sync',
className: 'bg-port-success/15 text-port-success hover:bg-port-success/25',
Icon: CheckCircle2,
title: 'All peers in parity',
},
diverged: {
label: 'Diverged',
className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25',
Icon: AlertTriangle,
title: 'Content differs from at least one peer',
},
'assets-missing': {
label: 'Assets missing',
className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25',
Icon: AlertTriangle,
title: 'Record present but associated files are missing on a peer',
},
'local-only': {
label: 'Local only',
className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25',
Icon: AlertTriangle,
title: 'Not present on at least one peer',
},
'peer-only': {
label: 'On peer only',
className: 'bg-port-warning/15 text-port-warning hover:bg-port-warning/25',
Icon: AlertTriangle,
title: 'Present on peer but not local',
},
'not-syncing': {
label: 'Not syncing',
className: 'bg-gray-600/20 text-gray-400 hover:bg-gray-600/30',
Icon: WifiOff,
// True when no online peer is syncing THIS category — either no peers are
// sync-enabled at all, or they are but have this category turned off.
title: 'No peers syncing this category — enable it for a peer?',
},
unknown: {
label: 'Sync unknown',
className: 'bg-gray-600/20 text-gray-400 hover:bg-gray-600/30',
Icon: HelpCircle,
// Sync IS configured, but every eligible peer was unreachable / too-old /
// errored, so we couldn't compute parity. Distinct from 'not-syncing'.
title: 'Sync status unavailable — peer offline, unreachable, or on an older PortOS',
},
Comment thread
atomantic marked this conversation as resolved.
};

export default function SyncBadge({ status, onClick }) {
if (!status) return null;

const config = STATUS_CONFIG[status];
if (!config) return null;

const { label, className, Icon, title } = config;

return (
<button
type="button"
onClick={onClick}
title={title}
className={`inline-flex items-center gap-1 px-1.5 py-0.5 rounded text-[10px] font-medium transition-colors ${className}`}
>
<Icon className="w-3 h-3 flex-shrink-0" aria-hidden="true" />
<span>{label}</span>
</button>
);
}
79 changes: 79 additions & 0 deletions client/src/components/sync/SyncBadge.test.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import { describe, it, expect, vi } from 'vitest';
import { render, screen, fireEvent } from '@testing-library/react';
import SyncBadge from './SyncBadge';

describe('SyncBadge', () => {
it('renders "In sync" for in-parity status', () => {
render(<SyncBadge status="in-parity" onClick={() => {}} />);
expect(screen.getByRole('button', { name: /in sync/i })).toBeInTheDocument();
expect(screen.getByRole('button').className).toMatch(/port-success/);
});

it('renders "Diverged" for diverged status', () => {
render(<SyncBadge status="diverged" onClick={() => {}} />);
expect(screen.getByRole('button', { name: /diverged/i })).toBeInTheDocument();
expect(screen.getByRole('button').className).toMatch(/port-warning/);
});

it('renders "Assets missing" for assets-missing status', () => {
render(<SyncBadge status="assets-missing" onClick={() => {}} />);
expect(screen.getByText('Assets missing')).toBeInTheDocument();
expect(screen.getByRole('button').className).toMatch(/port-warning/);
});

it('renders "Local only" for local-only status', () => {
render(<SyncBadge status="local-only" onClick={() => {}} />);
expect(screen.getByText('Local only')).toBeInTheDocument();
expect(screen.getByRole('button').className).toMatch(/port-warning/);
});

it('renders "On peer only" for peer-only status', () => {
render(<SyncBadge status="peer-only" onClick={() => {}} />);
expect(screen.getByText('On peer only')).toBeInTheDocument();
expect(screen.getByRole('button').className).toMatch(/port-warning/);
});

it('renders "Not syncing" with distinct neutral styling for not-syncing status', () => {
render(<SyncBadge status="not-syncing" onClick={() => {}} />);
const btn = screen.getByRole('button', { name: /not syncing/i });
expect(btn).toBeInTheDocument();
// Must not use warning or success colors — visually distinct/neutral
expect(btn.className).not.toMatch(/port-warning/);
expect(btn.className).not.toMatch(/port-success/);
expect(btn.className).toMatch(/gray/);
});

it('renders "Sync unknown" with neutral styling for unknown status', () => {
render(<SyncBadge status="unknown" onClick={() => {}} />);
const btn = screen.getByRole('button', { name: /sync unknown/i });
expect(btn).toBeInTheDocument();
// Neutral, like not-syncing — not a warning/success state.
expect(btn.className).not.toMatch(/port-warning/);
expect(btn.className).not.toMatch(/port-success/);
expect(btn.className).toMatch(/gray/);
});

it('calls onClick when clicked', () => {
const onClick = vi.fn();
render(<SyncBadge status="in-parity" onClick={onClick} />);
fireEvent.click(screen.getByRole('button'));
expect(onClick).toHaveBeenCalledTimes(1);
});

it('renders nothing for null status', () => {
const { container } = render(<SyncBadge status={null} onClick={() => {}} />);
expect(container.firstChild).toBeNull();
});

it('renders nothing for undefined status', () => {
const { container } = render(<SyncBadge onClick={() => {}} />);
expect(container.firstChild).toBeNull();
});

it('has a descriptive title attribute', () => {
render(<SyncBadge status="diverged" onClick={() => {}} />);
const btn = screen.getByRole('button');
expect(btn.title).toBeTruthy();
expect(btn.title.length).toBeGreaterThan(5);
});
});
Loading