Skip to content

Comments

Speed up test_weakref_leak and test_step_args#8107

Open
mcwanza wants to merge 1 commit intopymc-devs:mainfrom
mcwanza:speedup-slow-tests
Open

Speed up test_weakref_leak and test_step_args#8107
mcwanza wants to merge 1 commit intopymc-devs:mainfrom
mcwanza:speedup-slow-tests

Conversation

@mcwanza
Copy link

@mcwanza mcwanza commented Feb 12, 2026

Addresses #7686 — profiling-backed optimizations targeting tests where iteration reduction yields meaningful savings.

Changes

test_weakref_leak (89s → ~30s estimated on CI)

Object count profiling shows memory state stabilizes at iteration 2:

Iteration | Growing objects | Details
    0     | 10 growing      | function:+67554, tuple:+50400, dict:+30266
    1     |  1 growing      | list:+1
    2     |  0 growing      | STABLE
    3     |  0 growing      | STABLE
   ...    |  0 growing      | STABLE (all remaining iterations)

The original test ran 20 iterations but only checked after iteration 15 — 16 warmup iterations when 3 is sufficient. Reduced to 6 total (3 warmup + 3 check). Each conditional_logp call costs ~4.5s on CI, so removing 14 iterations saves ~63s.

test_step_args (62s → ~25s estimated on CI)

This test verifies target_accept argument plumbing, checking acceptance_rate.mean() ≈ 0.5 with decimal=1 precision. The default pm.sample() uses 1000 draws/tune, but 200 is more than sufficient for this loose tolerance (verified stable over 10 runs with different seeds).

Profiling: first pm.sample() with 1000 draws takes 2.20s locally, with 200 draws takes 0.29s. The test calls pm.sample() 5 times.

Changes NOT made (profiling showed negligible impact)

Test CI time Bottleneck Iteration savings
test_default_value_transform_logprob 35s Compile: 8.85s, Loop(10): 0.0014s 0.6ms total
TestMatchesScipy::test_interpolated 33s Compile: 91.8%, Array: 7.9% ~296ms across 56 iters

Compilation dominates these tests — reducing iteration counts has near-zero impact, consistent with @ricardoV94's review feedback.

@mcwanza mcwanza marked this pull request as ready for review February 12, 2026 02:17
Copy link
Member

@ricardoV94 ricardoV94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes are fine, other sacrifice too much on the tests. Left some commenst

"size, shape",
[
((10,), None),
(None, (10, 6)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should remove any of the parametrizations in test_multivariate. If these are slow the best thing is to analyze why instead

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted — all test_multivariate.py changes have been removed from this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworked the entire PR based on this feedback. Profiled all the slow tests to understand where time is actually spent — you were right that compilation dominates.

Removed all the original changes (parametrization reduction, iteration count tweaks on compile-dominated tests). The PR now only contains 2 profiling-backed changes:

  1. test_weakref_leak (20→6 iterations): Object count profiling shows state stabilizes at iteration 2 — the original 16 warmup iterations were overkill. Each conditional_logp call costs ~4.5s on CI, so this saves ~63s.

  2. test_step_args (explicit draws=200, tune=200): This test checks acceptance_rate.mean() ≈ 0.5 with decimal=1 precision — 200 draws is sufficient. The default 1000 was unnecessary for this tolerance level.

"draws": 10,
"tune": 10,
"draws": 5,
"tune": 5,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be very surprised if this played a role. The cost will be mostly compile time, not running 5 +- steps. Let me know if I am wrong

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted — the progress bar change has been removed from this PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed by profiling. The test_default_value_transform_logprob loop takes 0.0014s total for 10 iterations while compilation takes 8.85s — compile/loop ratio of 6480x. Removed this change from the PR.

Profiling-backed changes targeting the two tests where iteration
reduction yields meaningful savings.

test_weakref_leak (89s on CI → estimated ~30s):
Object count profiling shows memory state stabilizes at iteration 2.
The original test used 16 warmup iterations before checking — reduced
to 3 warmup + 3 check = 6 total iterations (from 20). Each
conditional_logp call costs ~4.5s on CI, so removing 14 iterations
saves ~63s.

test_step_args (62s on CI → estimated ~25s):
This test verifies target_accept argument plumbing, checking
acceptance_rate.mean() ≈ 0.5 with decimal=1 precision. The default
pm.sample() uses 1000 draws/tune, but 200 is sufficient for this
loose tolerance. Verified stable over 10 runs with different seeds.

Changes NOT made (profiling showed negligible impact):
- test_default_value_transform_logprob range(10)→range(3): compile
  time is 99.98% of cost, loop saves 0.6ms total
- test_interpolated x_points 100k→20k: compilation is 91.8% of cost,
  array reduction saves ~296ms across 56 iterations

Addresses pymc-devs#7686
@mcwanza mcwanza changed the title Reduce unnecessary iteration counts in slow tests Speed up test_weakref_leak and test_step_args Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants