feat!: fanout tree protocol + large-network sims + interactive sandbox (#582)

* refactor(stream): extract seek routing helpers

* pubsub: add large-network sims and protocol sandbox

* ci: fix site build and bootstrap test

* docs: gate remote bootstrap example in CI

* docs: rewrite fanout visualizer article with references

* site: fix sandbox canvas overflow on wide layouts

* pubsub: add formation metrics + score to fanout-tree-sim

* site/docs: clarify subscribe gossip vs data plane

* peerbit-server: allow API bootstrap address override

* test: reduce flakiness (bootstrap, e2e, stream dial)

* peerbit-server: fix lint in bootstrap handler

* docs: align fanout visualizer defaults with article

* peerbit-server: tidy bootstrap handler formatting

* fix(shared-log): de-flake isLeader reachableOnly test

* feat(shared-log)!: enforce fanout data path for target-all

Remove legacy rpc fallback for target:"all" append path and make fanout failures explicit.\nForward fanout options through Documents->SharedLog and update tests/spec notes for root-only fanout publish/control-plane behavior.

* feat(pubsub)!: add routed unicast control-plane

Bound route-token cache with TTL+eviction, add route announce/query/reply control messages, expose fanout helpers on Peerbit instead of DirectSub wiring, and add route/cache metrics in sims + protocol docs.

* test(document): de-flake slow redundancy/update cases

Increase timing tolerance for known flaky scenarios and bound late-drop wait to avoid callback timing hangs in CI.

* test(document): reduce redundancy test flakiness further

Scale scenario size and widen convergence window for min-replica search test to avoid CI variance while preserving intent.

* feat(shared-log): use fanout membership for subscriber discovery

* feat!: scalable fanout delivery + interactive protocol sandbox

\nBREAKING CHANGE: FanoutTree no longer uses periodic route announcements; root route resolution is now demand-driven via subtree route queries.

* feat(log,shared-log): inject provider hints for remote entry fetch

* feat(blocks)!: bound remote get via provider hints

* feat(log,shared-log): resolve remote.from candidates for joins

* feat(shared-log): hint block providers from prune acks

* feat(pubsub)!: fanout-tree v0.5 cache hardening

- Bump fanout protocol multicodec to /peerbit/fanout-tree/0.5.0\n- Make route cache + peer hints bounded and prune in O(expired)\n- Reduce TopicControlPlane subscription tracking retention (store timestamps only)\n- Update integration helpers/tests/docs accordingly

* test(pubsub): add churn maintenance objective metrics

* feat(pubsub): add join candidate scoring modes

* feat(stream): bound route cache memory growth

* feat(pubsub): harden fanout-tree sims and tests

* fix(shared-log,program): stabilize close/resume and readiness

* fix(peerbit-react): prevent lockstorage timer leak

* test(document,sqlite3): reduce flakiness

* feat(pubsub,blocks,peerbit,shared-log): use fanout provider directory for block fetch

* test(stream,sqlite3,server,docs): reduce CI flakiness

* style(shared-log): fix accidental indentation

* test(pubsub): deflake fully-connected routing

* test(pubsub): remove fixed waits in fully-connected routing

* feat(peerbit-org): add fanout formation sandbox

Adds an interactive join/capacity visualizer (FanoutTree over in-memory libp2p) and embeds it in the fanout blog post.

* fix(peerbit-org): render formation nodes reliably

Make the canvas draw loop read latest state from refs to avoid stale closures (previously could draw only the root node during joins).

* feat(peerbit-org): simulate churn in formation sandbox

Allow clicking nodes to drop them (offline), then visualize FanoutTree auto-reparenting via join loop. Adds InMemoryNetwork.unregisterPeer and updates the blog instructions.

* feat(peerbit-org): improve formation churn recovery demo

* fix(peerbit-org): harden formation layout for disconnected components

* fix(pubsub): avoid cold-cache no-op publish on rejoin

* feat(pubsub): harden fanout tree join and routing

* docs(peerbit-org): add interactive fanout demos

* fix(peerbit-org): make fanout formation demo typecheck

* fix(shared-log): harden sync convergence in CI

* fix(pubsub): emit peer disconnect events in in-memory shim

* ci: avoid playwright install-deps in node-only CI

* fix(stream): make waitFor result order deterministic

* feat!(transport): rename/bump control-plane and blocks multicodecs

* fix(peerbit-org): close connections on unregister

* docs(peerbit-org): fix inline code rendering + update fanout article

* feat(network): add fanout metadata + shared-log fanout envelope

* fanout: add unicast ACK + shared-log directed delivery via fanout

* docs: clarify demos and expand comparisons (Iroh/Netflix/Tor)

* pubsub: opt-in fanout topics without subscriber gossip

* network v2: shared routes + abuse guards + fanout topic bench

* pubsub/shared-log: fanout sharding + checked-prune resiliency

Deflake integration tests and refresh docs/examples

* document: deflake outOfOrder queue late-results test

* time: deflake debounceAccumulator leading timing test

* shared-log: prevent duplicate replicator leave on unsubscribe races

* shared-log: bound explicit prune timeout + fix fanout sandbox

* trusted-network: deflake replicator wait in CI

* shared-log: announce offline on drop (fix restart sync)

* trusted-network: deflake trusted by chain test

* pubsub: shard-only control plane + bounded teardown

* test(shared-log): harden restart replication wait timeout

* test(shared-log): stabilize restart replication in CI

* test(shared-log): stabilize restart sync source in ci

* test(shared-log): stabilize restart and rateless sync assertions

* test(shared-log): stabilize restart and sharding ci flakes

* Fix shared-log sharding flake on peer leave and convergence

* chore: retrigger CI for PR 582

* refactor(sim): dedupe in-memory shims and benchmark helpers

* refactor(sim): move in-memory shim to libp2p-test-utils

* refactor(sim): remove legacy benchmark and dedupe fanout scaffolding

* docs(blog): clarify fanout claims, assumptions, and tradeoffs

* docs(fanout): migrate network todo items to issues and remove handoff file

* docs(fanout): reference parent tracker issue instead of child issue list

* docs(fanout): convert scalable-fanout into spec-only contract

* refactor(sim): remove benchmark in-memory libp2p re-export shims

* docs(sqlite3): annotate write barrier perf follow-ups

* chore(github): remove scalable fanout issue template

* Improve fanout visualizer runtime links and setup reliability

* Improve fanout join liveness and benchmark scale controls

* test(pubsub): stabilize mild-churn sim threshold in CI

* stream: harden keepalive under churn and add delivery regressions

* stream test: stabilize re-seek assertion after disconnect

* fix(stream): restore ack-based keepalive probing

* Stabilize fanout ci-loss benchmark and stream ACK cleanup

* Fix ci lint rule in fanout peerbit sim cleanup

Author: marcus-pousette

.github/workflows/ci.yml

@@ -40,7 +40,8 @@ jobs:
         run: |
           curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
           pnpm install --frozen-lockfile=false
-          pnpm exec playwright install-deps
+          # NOTE: CI runs node-only tests (`-t node`). Playwright system deps are not required
+          # here and have proven flaky on GitHub runners due to transient apt repository errors.
       - name: Build
         run: |
           pnpm run build
@@ -86,7 +87,8 @@ jobs:
         run: |
           curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
           pnpm install --frozen-lockfile=false
-          pnpm exec playwright install-deps
+          # NOTE: CI runs node-only tests (`-t node`). Playwright system deps are not required
+          # here and have proven flaky on GitHub runners due to transient apt repository errors.
       - name: Build
         run: |
           pnpm run build

.github/workflows/nightly-sims.yml

@@ -0,0 +1,57 @@
+name: Nightly sims
+on:
+  schedule:
+    # 03:30 UTC every day
+    - cron: '30 3 * * *'
+  workflow_dispatch: {}
+
+jobs:
+  pubsub_sims:
+    name: PubSub sims
+    runs-on: ubuntu-22.04
+    timeout-minutes: 60
+    env:
+      NODE_OPTIONS: --no-warnings
+      PEERBIT_TEST_SESSION: mock
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: pnpm/action-setup@v4
+        with:
+          run_install: false
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22.x
+          cache: pnpm
+          cache-dependency-path: pnpm-lock.yaml
+
+      - name: Install deps
+        run: |
+          curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
+          pnpm install --frozen-lockfile=false
+
+      - name: Build pubsub workspace
+        run: |
+          pnpm --filter @peerbit/pubsub... run build
+
+      - name: Run fanout-tree-sim (scale-5k)
+        run: |
+          mkdir -p sim-results
+          pnpm -C packages/transport/pubsub run bench -- fanout-tree-sim --preset scale-5k --profile 1 > sim-results/fanout-tree-scale-5k.txt
+
+      - name: Run DirectSub topic-sim (500 nodes)
+        run: |
+          pnpm -C packages/transport/pubsub run bench -- topic-sim \
+            --nodes 500 --degree 6 --subscribers 400 \
+            --messages 50 --msgSize 256 --intervalMs 0 \
+            --seed 1 --subscribeModel preseed --silent 1 \
+            --timeoutMs 600000 > sim-results/topic-sim-500.txt
+
+      - name: Upload artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: nightly-sims-${{ github.run_id }}
+          path: sim-results
+

.github/workflows/site.yml

@@ -54,7 +54,7 @@ jobs:
           if [ -n "$SUPABASE_UPDATES_SYNC_URL" ]; then
             export VITE_UPDATES_EMAIL_FORM_ACTION="${SUPABASE_UPDATES_SYNC_URL%/updates-sync}/updates-subscribe"
           fi
-          pnpm --filter peerbit-org build
+          pnpm --filter peerbit-org... build
 
       - name: Typecheck site
         run: pnpm --filter peerbit-org typecheck

apps/peerbit-org/package.json

@@ -7,11 +7,18 @@
 		"dev": "vite",
 		"build": "vite build",
 		"preview": "vite preview",
-		"typecheck": "tsc -p tsconfig.json --noEmit"
+		"typecheck": "tsc -p tsconfig.json --noEmit",
+		"test": "vitest run --config vitest.config.ts"
 	},
 	"dependencies": {
+		"@libp2p/interface": "^3.1.0",
+		"@multiformats/multiaddr": "^13.0.1",
+		"@peerbit/crypto": "workspace:*",
+		"@peerbit/pubsub": "workspace:*",
+		"@peerbit/stream-interface": "workspace:*",
 		"highlight.js": "^11.11.1",
 		"iconoir-react": "^7.11.0",
+		"it-pushable": "^3.1.4",
 		"react": "^18.3.1",
 		"react-dom": "^18.3.1",
 		"react-markdown": "^9.0.3",
@@ -27,6 +34,7 @@
 		"@vitejs/plugin-react": "^4.3.4",
 		"tailwindcss": "^4.0.0",
 		"typescript": "^5.6.3",
+		"vitest": "^4.0.12",
 		"vite": "^7.0.0"
 	}
 }