Skip to content

Fix flaky test_maximum_full_file_count by eliminating race condition#20704

Open
TheLastCicada wants to merge 1 commit intomainfrom
fix/flaky-test-maximum-full-file-count
Open

Fix flaky test_maximum_full_file_count by eliminating race condition#20704
TheLastCicada wants to merge 1 commit intomainfrom
fix/flaky-test-maximum-full-file-count

Conversation

@TheLastCicada
Copy link
Contributor

@TheLastCicada TheLastCicada commented Mar 20, 2026

Purpose:

Eliminate a race condition that causes test_maximum_full_file_count to fail intermittently on macOS CI with a misleading 30-second timeout error.

Current Behavior:

The test loop (9 iterations of batch_updatefarm_block_with_spend) has a race condition. After farming a block and waiting for wallet sync, the DataLayer wallet's internal singleton state may not have caught up before the next batch_update builds a new spend. The wallet creates a spend against stale singleton state → full node rejects it (INVALID_SPEND_BUNDLE, status 3) → check_mempool_spend_count polls until timeout → test fails after 30 seconds with AssertionError: Timed assertion timed out.

New Behavior:

Three fixes:

  1. Fail-fast on spend rejection: New check_mempool_spend_count_or_fail() helper inspects the transaction's sent_to field on each poll iteration. If any entry has MempoolInclusionStatus.FAILED, it raises a RuntimeError immediately with the rejection reason instead of burning 30 seconds on a doomed timeout.

  2. Singleton readiness check: Added time_out_assert(10, check_singleton_confirmed, True, data_layer, store_id) after farm_block_with_spend in the test loop. This ensures the DataLayer wallet has processed the previous block's singleton update before the next iteration calls batch_update, closing the race condition. (farm_block_check_singleton already does this; farm_block_with_spend did not.)

  3. Relaxed mempool count check: Changed check_mempool_spend_count from == to >= so auto-resends or ancillary transactions adding extra mempool entries don't break the check.

The farm_block_with_spend function signature is unchanged — all ~40 existing callers are unaffected.

Testing Notes:

  • All 6 parametrized variants of test_maximum_full_file_count pass locally (611s total)
  • ruff lint, ruff format, and mypy all pass

Made with Cursor


Note

Low Risk
Low risk: changes are isolated to test helpers and assertions, primarily improving synchronization and error reporting for flaky CI behavior.

Overview
Fixes flakiness in test_maximum_full_file_count by ensuring each batch_update spend is fully processed before the next iteration runs.

The test now fails fast when a transaction is rejected (via check_mempool_spend_count_or_fail inspecting sent_to for MempoolInclusionStatus.FAILED), relaxes mempool size checks from == to >=, and adds an explicit check_singleton_confirmed wait after farm_block_with_spend in the update loop to avoid racing on stale singleton state.

Written by Cursor Bugbot for commit 0c9b1bb. This will update automatically on new commits. Configure here.

The test fails intermittently on macOS CI because farm_block_with_spend
polls the mempool with a blind timeout. When the wallet builds a spend
against stale singleton state, the full node rejects it
(INVALID_SPEND_BUNDLE, status 3), the mempool never reaches count 1,
and the test burns 30 seconds before failing with a misleading timeout.

Three fixes:

1. Add check_mempool_spend_count_or_fail() that inspects the
   transaction's sent_to field on each poll. If any entry has
   MempoolInclusionStatus.FAILED, raise immediately with the rejection
   reason instead of waiting for the timeout.

2. Add a singleton readiness check (check_singleton_confirmed) after
   farm_block_with_spend in the test loop, ensuring the DataLayer
   wallet has processed the previous block's singleton update before
   the next batch_update builds a new spend.

3. Change check_mempool_spend_count to use >= instead of == so
   auto-resends or ancillary transactions don't break the check.

Made-with: Cursor
@TheLastCicada TheLastCicada requested a review from a team as a code owner March 20, 2026 20:29
@TheLastCicada TheLastCicada added the Fixed Required label for PR that categorizes merge commit message as "Fixed" for changelog label Mar 20, 2026
@coveralls-official
Copy link

Pull Request Test Coverage Report for Build 23361347167

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 11 of 14 (78.57%) changed or added relevant lines in 1 file are covered.
  • 7 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.01%) to 91.159%

Changes Missing Coverage Covered Lines Changed/Added Lines %
chia/_tests/core/data_layer/test_data_rpc.py 11 14 78.57%
Files with Coverage Reduction New Missed Lines %
chia/timelord/timelord_api.py 1 93.02%
chia/full_node/full_node.py 2 87.57%
chia/timelord/timelord.py 2 74.25%
chia/wallet/wallet_node.py 2 86.25%
Totals Coverage Status
Change from base Build 23360405816: 0.01%
Covered Lines: 105830
Relevant Lines: 115928

💛 - Coveralls

@github-actions
Copy link
Contributor

File Coverage Missing Lines
chia/_tests/core/data_layer/test_data_rpc.py 78.6% lines 209-211
Total Missing Coverage
14 lines 3 lines 78%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coverage-diff Fixed Required label for PR that categorizes merge commit message as "Fixed" for changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant