test: add nightly connectivity tests with stronger assertions

[iduartgomez]

Problem

The current test_connection_growth_stall_regression test uses a small simulation (15 nodes, 20 virtual minutes) with moderate assertions to keep CI fast. This limits how tightly we can assert on connection growth speed and final topology quality due to run-to-run variance in small simulations.

Proposal

Add nightly-only connectivity tests with:

• Larger scale: 50+ nodes, min_connections=10+, 1 virtual hour (per testing.md guidance)
• Growth rate assertions: Checkpoint connection counts at multiple time intervals (e.g., 5m, 15m, 30m, 60m) and assert monotonic growth toward min_connections
• Stronger final-state assertions: >= 90% of nodes at min_connections (per testing.md)
• Fault recovery speed: After injecting faults, assert recovery to pre-fault levels within a bounded time window

These would run on a schedule (not per-PR) since they take several minutes each.

Context

PR #3510 tightened the existing regression test assertions but was limited by the small simulation size. Nightly tests can use realistic parameters without blocking CI.

Checklist

• Add #[cfg(feature = "nightly_tests")] gated tests
• 50-node topology formation test with median >= min_connections assertion
• Growth rate checkpoint test (no plateau detection)
• Fault injection + recovery speed test
• CI schedule configuration for nightly runs

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-enhancement (95% confidence)
• E-medium (82% confidence)
• A-developer-xp (90% confidence)
• A-networking (88% confidence)

Reasoning: The issue proposes adding and strengthening nightly connectivity tests (larger-scale simulations, stronger assertions, fault-injection and CI scheduling). This is an improvement to existing test infrastructure rather than a bug or entirely new runtime feature, so T-enhancement is appropriate. Implementing gated tests, larger simulations and CI scheduling is moderate in complexity (E-medium). The change impacts developer experience/CI (A-developer-xp) and networking/topology testing (A-networking).

Previous labels: none

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[sanity]

Issue Priority Assessment

Generated: 2026-03-12T22:24:16.107108317+00:00
Issue hash: b4777efa

Value Assessment

Relevance is still high because the only regression test that digs into connection growth—test_connection_growth_stall_regression—runs a 2‑gateway, 15‑node topology for 20 virtual minutes and merely insists that 10 % of nodes touch min_connections, so it can’t enforce the stronger growth rates or 90 % final-state guarantees that the issue asks for when min_connections climbs above 5. The repository’s only #[cfg(feature = "nightly_tests")] hooks are for test_long_running_deterministic, test_subscription_renewal_at_scale, and test_subscribe_renewal_long_running, so there is currently no nightly coverage specifically exercising connectivity growth or recovery targets for larger topologies outside of those three suites (crates/core/tests/simulation_integration.rs:3479-3507, :4425-4502, :5221-5246). That means the problem the issue describes—small simulations limiting how tightly the assertions can be written—is still present in the codebase.

User impact remains material because without stronger nightly checks, real nodes in the wild could regress back to slow-growth or incomplete topologies before anyone notices. The nightly workflow already runs several fdev test scenarios, but those invocations simply wait for partial connectivity, run operations, and assert on overall success rates (.github/workflows/simulation-nightly.yml:79-172 and crates/fdev/src/testing/network.rs:208-250); they do not track per-node connection count milestones, recovery speed after a fault injection, or the percentage of peers that reach min_connections over 60 minutes. If a regression causes 50‑node networks to plateau at nine connections or recover sluggishly from NAT failures, it would go uncaught by the current suite and would eventually degrade user experience via sterile routing or gateway flapping.

Strategically, this proposal aligns with the project’s documented intent to push heavier simulations into the nightly suite (docs/architecture/testing/simulation-testing.md:344-357) and to gate expensive, deterministic scenarios behind nightly_tests. The new tests would slot into the same nightly pipeline that already fires the long-running and subscription-at-scale checks, providing another deterministic regression guard that targets an essential property of the routing/topology stack. Ensuring monotonic growth toward min_connections and quick recovery from injected faults complements the existing long-duration and contract-focused suites by exercising client-facing connectivity guarantees instead of purely long-term state or contract throughput.

There is no blocking dependency on this work—current CI still runs, and developers can land other features while the nighttime suite stays unchanged—but adding these assertions has moderate priority because they would catch hard-to-reproduce regressions before they reach real nodes. There is no rush tied to a release date, but the nightly scheduler already exists (.github/workflows/simulation-nightly.yml) so the tests would immediately benefit from being run every day once they are in place.

Cost Assessment

The regression scenario described in #3511 still maps cleanly onto the existing test_connection_growth_stall_regression infrastructure in crates/core/tests/simulation_integration.rs:6403 that exercises the same SimNetwork simulation stack, so the approach hasn’t become obsolete. The code still uses deterministic seeds, let_network_run to advance virtual time, and SimNetwork::connection_count to measure topology state, so all of the building blocks cited in the issue are present. The nightly guidance in docs/architecture/testing/simulation-testing.md:344 confirms that this repository already expects larger-scale scenarios (50+ nodes, real traffic) to live behind nightly_tests and that those runs are scheduled via .github/workflows/simulation-nightly.yml, so executing the proposed enhancements in a nightly-only context remains practicable and doesn’t conflict with recent architectural drift.

Implementing the issue means touching crates/core/tests/simulation_integration.rs again, ideally adding multiple #[cfg(feature = "nightly_tests")] functions near the existing nightly section (see test_long_running_deterministic around :3479 for the current pattern). Each new test require instrumentation to sample the full set of nodes, so you’ll still rely on SimNetwork::connection_count (crates/core/src/node/testing_impl.rs:1398) and probably wrap that logic in helper utilities that collect medians, per-node tallies, and growth profiles over the requested 5/15/30/60 minute checkpoints. The feature gate already exists in crates/core/Cargo.toml:189, so no build-system churn is necessary—only the tests plus any helper utilities. Scheduling-wise, the nightly job already invokes cargo nextest run -p freenet --features "simulation_tests,testing,nightly_tests" --test simulation_integration --profile nightly (.github/workflows/simulation-nightly.yml:157), so the extra tests will run there once they are present; no new workflow is required beyond ensuring the new tests are covered in that invocation.

Technically, the hardest part is writing deterministic assertions for both intermediate growth and post-fault recovery. You’ll need to drive the simulation through repeated let_network_run phases, log connection counts at precise virtual times, and then validate monotonic growth toward the advertised min_connections (with thresholds inspired by the 90%-nodes-at-min mandate documented in docs/architecture/testing/simulation-testing.md:357). The fault-recovery test requires injecting deterministic message loss or latency with SimNetwork::with_fault_injection (crates/core/src/node/testing_impl.rs:1011) and confidently measuring recovery to pre-fault connection levels within a bounded window, which means capturing a baseline, introducing faults, observing the dip, and then verifying the climb within another let_network_run interval. There’s also the danger of flakiness because these large simulations still depend on seed outcomes and timing, so the tests should likely log diagnostics when the assertions fail to accelerate debugging.

Beyond code, no external approvals or services are needed—everything runs inside the existing nightly workflow, and its cargo nextest invocation already leverages the nightly_tests feature so the gating checklist (“add #[cfg(feature = "nightly_tests")]” and “nightly schedule configuration”) is satisfied once the tests exist. The main external cost is runtime: locally exercising the new tests for verification requires the same long cargo test -p freenet --features "simulation_tests,testing,nightly_tests" --test simulation_integration run that currently takes minutes per test (docs/architecture/testing/simulation-testing.md:361-367), and the nightly job may need a few extra minutes per run, which raises the risk that a flaky test could repeatedly fail and require investigation. All told, the scope is moderate (extra functions plus a bit of helper logic) but the complexity and runtime cost are high enough that this should remain a medium-priority effort.

---

Generated by issue-prioritizer