Skip to content

feat(cluster): wire multi-shard cross-shard communication for server-ng#3269

Open
hubcio wants to merge 1 commit into
masterfrom
crossfire-server-ng
Open

feat(cluster): wire multi-shard cross-shard communication for server-ng#3269
hubcio wants to merge 1 commit into
masterfrom
crossfire-server-ng

Conversation

@hubcio
Copy link
Copy Markdown
Contributor

@hubcio hubcio commented May 15, 2026

core/server-ng bootstrapped exactly one shard with SHARD_ID=0 and
senders=vec![sender] hardcoded; the multi-shard path was dead code.
Cross-shard primitives copied from legacy core/server also did not
fit core/shard's crossfire model (bounded MTx + try_send-or-drop,
fd-transfer coordinator instead of SO_REUSEPORT).

bootstrap() now spawns N OS threads from sharding.cpu_allocation,
each pinned via nix::sched + hwlocality and running its own compio
runtime + IggyMessageBus + IggyShard for the partitions hashed to
that shard. Cross-thread shutdown rides an Arc polled by
a per-shard watchdog, since the bus' Shutdown is !Send and cannot be
triggered from the main thread directly. Partial shard-spawn failure
and shard-thread panic now signal cluster-wide shutdown instead of
hanging; the shutdown watchdog is detached from the bus drain.

ShardFrame becomes a concrete enum (Consensus + Lifecycle); the R
generic is lifted off IggyShard. Named routers (route_metadata /
route_partition / route_consensus_control) replace the duplicated
MessageBag match blocks, and a debug_assert at pump entry catches
receiver-thread mis-binding that the ctor's assert_sender_ordering
cannot see.

ShardMetrics records frame_drops_total (counter, variant+reason
labels), bumped at every inter-shard try_send rejection; without it,
drop-and-recover under VSR retransmit is operationally
indistinguishable from a livelock. The counter is atomic, so it is
safe to bump from !Send compio reactor contexts.

The legacy shard-mapping broadcast subsystem (periodic snapshot
refresh task, three-state MappingSlot table, ReplicaMappingUpdate /
ReplicaMappingClear frames) is retired entirely. Cross-shard replica
routing now flows through the cluster-shared ReplicaOwnerTable: the
owning shard's installer stamps its slot on a successful registry
insert and CAS-clears it on disconnect, so every bus' send_to_replica
slow path reads authoritative state with no broadcast or reconcile
loop. Builder accepts the coordinator at ctor; IggyShard stays
immutable post-construction.

message_bus forward-fn types widen to carry replica/client id, and
send_to_replica routes via the shared owner table so non-zero shards
reconcile against shared state rather than shard 0's private bus.

WAL recovery is serialized across shards: non-zero shards open the
WAL read-only, reject mutating ops (drain, set_snapshot_op) on a
read-only storage, route an invalid WAL header to truncate_or_fail,
and close the read-only fd once recovery completes.

Storage milestones (durable PartitionJournal, durable
(view, commit_op) watermark) and SDK (client_id, request_id)
durability across reconnect remain out of scope; tracked separately.

@hubcio hubcio force-pushed the crossfire-server-ng branch from e0e289d to 84e79d3 Compare May 15, 2026 09:33
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label May 15, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 15, 2026

Codecov Report

❌ Patch coverage is 38.61038% with 698 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.99%. Comparing base (fd4dbf6) to head (92db4e4).

Files with missing lines Patch % Lines
core/server-ng/src/bootstrap.rs 18.69% 384 Missing and 3 partials ⚠️
core/shard/src/router.rs 0.00% 133 Missing ⚠️
core/shard/src/builder.rs 0.00% 54 Missing ⚠️
core/message_bus/src/lib.rs 78.90% 22 Missing and 5 partials ⚠️
core/server-ng/src/main.rs 0.00% 26 Missing ⚠️
core/shard/src/lib.rs 38.09% 26 Missing ⚠️
core/simulator/src/bus.rs 0.00% 12 Missing ⚠️
core/shard/src/coordinator.rs 86.25% 11 Missing ⚠️
core/consensus/src/plane_helpers.rs 0.00% 6 Missing ⚠️
core/journal/src/prepare_journal.rs 90.90% 3 Missing and 3 partials ⚠️
... and 5 more

❌ Your patch check has failed because the patch coverage (38.61%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3269       +/-   ##
=============================================
- Coverage     73.87%   62.99%   -10.89%     
  Complexity      943      943               
=============================================
  Files          1193     1192        -1     
  Lines        108970    99041     -9929     
  Branches      85988    76076     -9912     
=============================================
- Hits          80505    62393    -18112     
- Misses        25733    33771     +8038     
- Partials       2732     2877      +145     
Components Coverage Δ
Rust Core 60.74% <38.61%> (-14.20%) ⬇️
Java SDK 60.14% <ø> (ø)
C# SDK 69.16% <ø> (-0.28%) ⬇️
Python SDK 81.43% <ø> (ø)
Node SDK 91.53% <ø> (+0.12%) ⬆️
Go SDK 39.80% <ø> (ø)
Files with missing lines Coverage Δ
core/configs/src/server_config/sharding.rs 84.24% <100.00%> (+1.76%) ⬆️
core/journal/src/file_storage.rs 70.58% <ø> (ø)
core/message_bus/src/client_listener/tcp.rs 80.00% <ø> (ø)
core/message_bus/src/client_listener/tcp_tls.rs 84.61% <ø> (ø)
core/message_bus/src/client_listener/ws.rs 80.76% <ø> (ø)
core/message_bus/src/installer/mod.rs 62.50% <ø> (+9.86%) ⬆️
core/message_bus/src/installer/replica.rs 85.97% <100.00%> (+0.53%) ⬆️
core/message_bus/src/installer/tcp_tls.rs 90.00% <ø> (ø)
core/message_bus/src/installer/wss.rs 90.47% <ø> (ø)
core/metadata/src/stm/mod.rs 52.67% <100.00%> (+11.16%) ⬆️
... and 21 more

... and 210 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

core/server-ng bootstrapped exactly one shard with SHARD_ID=0 and
senders=vec![sender] hardcoded; the multi-shard path was dead code.
Cross-shard primitives copied from legacy core/server also did not
fit core/shard's crossfire model (bounded MTx + try_send-or-drop,
fd-transfer coordinator instead of SO_REUSEPORT).

bootstrap() now spawns N OS threads from sharding.cpu_allocation,
each pinned via nix::sched + hwlocality and running its own compio
runtime + IggyMessageBus + IggyShard for the partitions hashed to
that shard. Cross-thread shutdown rides an Arc<AtomicBool> polled by
a per-shard watchdog, since the bus' Shutdown is !Send and cannot be
triggered from the main thread directly. Partial shard-spawn failure
and shard-thread panic now signal cluster-wide shutdown instead of
hanging; the shutdown watchdog is detached from the bus drain.

ShardFrame becomes a concrete enum (Consensus + Lifecycle); the R
generic is lifted off IggyShard. Named routers (route_metadata /
route_partition / route_consensus_control) replace the duplicated
MessageBag match blocks, and a debug_assert at pump entry catches
receiver-thread mis-binding that the ctor's assert_sender_ordering
cannot see.

ShardMetrics records frame_drops_total (counter, variant+reason
labels), bumped at every inter-shard try_send rejection; without it,
drop-and-recover under VSR retransmit is operationally
indistinguishable from a livelock. The counter is atomic, so it is
safe to bump from !Send compio reactor contexts.

The legacy shard-mapping broadcast subsystem (periodic snapshot
refresh task, three-state MappingSlot table, ReplicaMappingUpdate /
ReplicaMappingClear frames) is retired entirely. Cross-shard replica
routing now flows through the cluster-shared ReplicaOwnerTable: the
owning shard's installer stamps its slot on a successful registry
insert and CAS-clears it on disconnect, so every bus' send_to_replica
slow path reads authoritative state with no broadcast or reconcile
loop. Builder accepts the coordinator at ctor; IggyShard stays
immutable post-construction.

message_bus forward-fn types widen to carry replica/client id, and
send_to_replica routes via the shared owner table so non-zero shards
reconcile against shared state rather than shard 0's private bus.

WAL recovery is serialized across shards: non-zero shards open the
WAL read-only, reject mutating ops (drain, set_snapshot_op) on a
read-only storage, route an invalid WAL header to truncate_or_fail,
and close the read-only fd once recovery completes.

Storage milestones (durable PartitionJournal, durable
(view, commit_op) watermark) and SDK (client_id, request_id)
durability across reconnect remain out of scope; tracked separately.
@hubcio hubcio force-pushed the crossfire-server-ng branch from 84e79d3 to 92db4e4 Compare May 15, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant