Release v0.37 #901

mhauru · 2025-04-24T16:47:50Z

0.37 checklist

Accumulators

Accumulators broke Enzyme. This is a known issue and already reported upstream:

deepcopying NamedTuples seems to break EnzymeAD/Enzyme.jl#2429

Fixes for particle methods

VariableOrderAccumulator #940
Fix behaviour of set_retained_vns_del! for num_produce == 0 #1000
Remove set_retained_vns_del!, cf "Fixes" for PG-in-Gibbs Turing.jl#2629

Remove `context` from `evaluate!!` and simplify submodel code

Replacing `SamplingContext` with `InitContext`

InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict #980

Fixes to AD testing

Improve API for AD testing #964

Shuffle code around

For 0.38

Improve performance of values_as_in_model #986
InitContext, part 3 - Introduce InitContext #981
InitContext, part 4 - Use init!! to replace evaluate_and_sample!!, predict, returned, and initialize_values #984
InitContext, part 5 - Remove SamplingContext, SampleFrom{Prior,Uniform}, {tilde_,}assume #985
A remaining PR on documentation
Break up contexts and possibly accumulator code

codecov · 2025-04-24T17:00:35Z

Codecov Report

❌ Patch coverage is 86.19855% with 114 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (0b7213f) to head (c98a67c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/simple_varinfo.jl	61.11%	21 Missing ⚠️
src/varinfo.jl	89.11%	16 Missing ⚠️
src/abstract_varinfo.jl	84.09%	14 Missing ⚠️
src/default_accumulators.jl	87.01%	10 Missing ⚠️
src/logdensityfunction.jl	65.21%	8 Missing ⚠️
src/threadsafe.jl	83.33%	8 Missing ⚠️
src/debug_utils.jl	90.27%	7 Missing ⚠️
src/compiler.jl	53.84%	6 Missing ⚠️
src/extract_priors.jl	75.00%	6 Missing ⚠️
src/test_utils/ad.jl	66.66%	6 Missing ⚠️
... and 6 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #901      +/-   ##
==========================================
- Coverage   82.97%   81.91%   -1.07%     
==========================================
  Files          36       38       +2     
  Lines        3965     4025      +60     
==========================================
+ Hits         3290     3297       +7     
- Misses        675      728      +53

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

coveralls · 2025-04-24T17:56:44Z

Pull Request Test Coverage Report for Build 15395885376

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

510 of 604 (84.44%) changed or added relevant lines in 24 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on breaking at 81.15%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/context_implementations.jl	11	12	91.67%
src/utils.jl	7	8	87.5%
src/values_as_in_model.jl	12	13	92.31%
src/accumulators.jl	27	29	93.1%
src/debug_utils.jl	50	52	96.15%
ext/DynamicPPLJETExt.jl	0	3	0.0%
src/extract_priors.jl	12	15	80.0%
src/model.jl	30	37	81.08%
src/threadsafe.jl	42	49	85.71%
src/simple_varinfo.jl	27	36	75.0%

Totals
Change from base Build 15391414350:	81.2%
Covered Lines:	3302
Relevant Lines:	4069

💛 - Coveralls

* Release 0.36 * AbstractPPL 0.11 + change prefixing behaviour (#830) * AbstractPPL 0.11; change prefixing behaviour * Use DynamicPPL.prefix rather than overloading * Remove VarInfo(VarInfo, params) (#870) * Unify `{untyped,typed}_{vector_,}varinfo` constructor functions (#879) * Unify {Untyped,Typed}{Vector,}VarInfo constructors * Update invocations * NTVarInfo * Fix tests * More fixes * Fixes * Fixes * Fixes * Use lowercase functions, don't deprecate VarInfo * Rewrite VarInfo docstring * Fix methods * Fix methods (really) * Draft of accumulators * Fix some variable names * Fix pointwise_logdensities, gut tilde_observe, remove resetlogp!! * Map rather than broadcast Co-authored-by: Tor Erlend Fjelde <[email protected]> * Start documenting accumulators * Use Val{symbols} instead of AccTypes to index * More documentation for accumulators * Link varinfo by default in AD testing utilities; make test suite run on linked varinfos (#890) * Link VarInfo by default * Tweak interface * Fix tests * Fix interface so that callers can inspect results * Document * Fix tests * Fix changelog * Test linked varinfos Closes #891 * Fix docstring + use AbstractFloat * Fix resetlogp!! and type stability for accumulators * Fix type rigidity of LogProbs and NumProduce * Fix uses of getlogp and other assorted issues * setaccs!! nicer interface and logdensity function fixes * Revert back to calling the macro @addlogprob! * Remove a dead test * Clarify a comment * Implement split/combine for PointwiseLogdensityAccumulator * Switch ThreadSafeVarInfo.accs_by_thread to be a tuple * Fix `condition` and `fix` in submodels (#892) * Fix conditioning in submodels * Simplify contextual_isassumption * Add documentation * Fix some tests * Add tests; fix a bunch of nested submodel issues * Fix fix as well * Fix doctests * Add unit tests for new functions * Add changelog entry * Update changelog Co-authored-by: Hong Ge <[email protected]> * Finish docs * Add a test for conditioning submodel via arguments * Clean new tests up a bit * Fix for VarNames with non-identity lenses * Apply suggestions from code review Co-authored-by: Markus Hauru <[email protected]> * Apply suggestions from code review * Make PrefixContext contain a varname rather than symbol (#896) --------- Co-authored-by: Hong Ge <[email protected]> Co-authored-by: Markus Hauru <[email protected]> * Revert ThreadSafeVarInfo back to Vectors and fix some AD type casting in (Simple)VarInfo * Improve accumulator docs * Add test/accumulators.jl * Docs fixes * Various small fixes * Make DynamicTransformation not use accumulators other than LogPrior * Fix variable order and name of map_accumulator!! * Typo fixing * Small improvement to ThreadSafeVarInfo * Fix demo_dot_assume_observe_submodel prefixing * Typo fixing * Miscellaneous small fixes * HISTORY entry and more miscellanea * Add more tests for accumulators * Improve accumulators docstrings * Fix a typo * Expand HISTORY entry * Add accumulators to API docs * Remove unexported functions from API docs * Add NamedTuple methods for get/set/acclogp * Fix setlogp!! with single scalar to error * Export AbstractAccumulator, fix a docs typo * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Rename LogPrior -> LogPriorAccumulator, and Likelihood and NumProduce * Type bound log prob accumulators with T<:Real * Add @addlogprior! and @addloglikelihood! * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Move default accumulators to default_accumulators.jl * Fix some tests * Introduce default_accumulators() * Go back to only having @addlogprob! * Fix tilde_observe!! prefixing * Fix default_accumulators internal type * Make unflatten more type stable, and add a test for it * Always print all benchmark results * Move NumProduce VI functions to abstract_varinfo.jl --------- Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Tor Erlend Fjelde <[email protected]> Co-authored-by: Hong Ge <[email protected]>

* Implement values_as_in_model using an accumulator * Make make_varname_expression a function * Refuse to combine ValuesAsInModelAccumulators with different include_colon_eqs * Fix nested context test

github-actions · 2025-06-02T11:57:14Z

Benchmark Report for Commit `c98a67c`

Computer Information

Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

|                 Model | Dimension |  AD Backend |      VarInfo Type | Linked | Eval Time / Ref Time | AD Time / Eval Time |
|-----------------------|-----------|-------------|-------------------|--------|----------------------|---------------------|
| Simple assume observe |         1 | forwarddiff |             typed |  false |                 13.4 |                 1.4 |
|           Smorgasbord |       201 | forwarddiff |             typed |  false |                763.4 |                39.4 |
|           Smorgasbord |       201 | forwarddiff | simple_namedtuple |   true |                532.0 |                50.4 |
|           Smorgasbord |       201 | forwarddiff |           untyped |   true |               1076.4 |                35.4 |
|           Smorgasbord |       201 | forwarddiff |       simple_dict |   true |               7500.8 |                25.4 |
|           Smorgasbord |       201 | reversediff |             typed |   true |               1582.6 |                27.2 |
|           Smorgasbord |       201 |    mooncake |             typed |   true |               1117.2 |                13.3 |
|    Loop univariate 1k |      1000 |    mooncake |             typed |   true |               7065.4 |                52.0 |
|       Multivariate 1k |      1000 |    mooncake |             typed |   true |               1051.1 |                 8.7 |
|   Loop univariate 10k |     10000 |    mooncake |             typed |   true |              84142.9 |                49.6 |
|      Multivariate 10k |     10000 |    mooncake |             typed |   true |               9127.2 |                 9.5 |
|               Dynamic |        10 |    mooncake |             typed |   true |                144.9 |                23.9 |
|              Submodel |         1 |    mooncake |             typed |   true |                 18.0 |                25.9 |
|                   LDA |        12 | reversediff |             typed |   true |               1101.7 |                 2.0 |

This reverts commit 3cb47cd.

github-actions · 2025-06-02T16:57:53Z

DynamicPPL.jl documentation for PR #901 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR901/

…952) * Change `evaluate!!` API, add `sample!!` * Fix literally everything else that I broke * Fix some docstrings * fix ForwardDiffExt (look, multiple dispatch bad...) * Changelog * fix a test * Fix docstrings * use `sample!!` * Fix a couple more cases * Globally rename `sample!!` -> `evaluate_and_sample!!`, add changelog warning

* Move submodel code to submodel.jl * Remove `@submodel`

* Clean up submodel code, remove 3-arg `_evaluate!!` * Remove 3-argument `evaluate!!` as well * Update changelog * Improve submodel error message * Fix doctest * Add error hint for three-argument evaluate!!

* Rework API for AD testing * Fix test * Add `rng` keyword argument * Use atol and rtol * remove unbound type parameter (?) * Don't need to do elementwise check * Update changelog * Fix typo

* Turn NumProduceAccumulator into VariableOrderAccumulator * Add comparison methods * Make VariableOrderAccumulator use regular Dict * Use copy rather than deepcopy for accumulators * Minor docstring touchup * Remove unnecessary use of NumProduceAccumulator * Fix split(VariableOrderAccumulator) * Remove NumProduceAcc from Debug * Fix set_retained_vns_del! --------- Co-authored-by: Penelope Yong <[email protected]>

* Give LogDensityFunction the getlogdensity field * Allow missing LogPriorAccumulator when linking * Trim whitespace * Run formatter * Fix a few typos * Fix comma -> semicolon * Fix `LogDensityAt` invocation * Fix one last test * Fix tests --------- Co-authored-by: Penelope Yong <[email protected]>

…nAccumulator` (#998) * logjac accumulator * Fix tests * Fix a whole bunch of stuff * Fix final tests * Fix docs * Fix docs/doctests * Fix maths in LogJacobianAccumulator docstring * Twiddle with a comment * Add changelog * Fix accumulator docstring * logJ -> logjac * Fix logjac accumulation for StaticTransformation

…; enforce key type of `AbstractDict` (#980) * point to unmerged AbstractPPL branch * Remove code that was moved to AbstractPPL * Remove Dictionaries with Any key type * Fix bad merge conflict resolution * Fix doctests * Point to [email protected] This reverts commit 709dc9e. * Fix doctests * Fix docs AbstractPPL bound * Remove stray `Pkg.update()`

…or (#999) * logjac accumulator * Fix tests * Fix a whole bunch of stuff * Fix final tests * Fix docs * Fix docs/doctests * Fix maths in LogJacobianAccumulator docstring * Twiddle with a comment * Add changelog * Simplify accs with LogProbAccumulator * Replace + with accumulate for LogProbAccs * Introduce merge and subset for accs * Improve acc tests * Fix docstring typo. Co-authored-by: Penelope Yong <[email protected]> * Fix merge --------- Co-authored-by: Penelope Yong <[email protected]>

mhauru · 2025-08-01T15:41:18Z

@yebai, @penelopeysm and I think this is ready to go. Do you want to check changes to HISTORY.md and make sure you are happy?

The corresponding Turing.jl PR, TuringLang/Turing.jl#2550, should also be either ready or very close to ready. We'll need this released first though, to see CI pass on the final version.

mhauru · 2025-08-01T15:58:07Z

Oh, should note that this PR includes VariableOrderAccumulator, be we agreed with @penelopeysm that it should be removed before making a release, since Turing.jl no longer needs it and its functionality. We'll merge this PR into main and then make another PR to remove VOA, just to have a commit in the commit history of main that includes it, in case we ever want to go back.

yebai · 2025-08-01T16:00:48Z

Excellent work and improvements!

yebai · 2025-08-01T16:01:37Z

@sunxd3 and @AoifeHughes, can you also take a look?

yebai · 2025-08-01T17:22:25Z

should note that this PR includes VariableOrderAccumulator, be we agreed with @penelopeysm that it should be removed before making a release, since Turing.jl no longer needs it and its functionality.

Can you add a note on why VariableOrderAccumulator is no longer needed?

penelopeysm · 2025-08-01T17:45:09Z

@yebai Happy to expand on it, do you want that in the changelog or the PR (or both)?

yebai · 2025-08-01T18:04:12Z

The changelog is more discoverable.

mhauru · 2025-08-04T08:18:31Z

@yebai a note about VariableOrderAccumulator being removed is added to HISTORY.md in #1005 (to be merged after this one and before making a release). Does that do it?

mhauru · 2025-08-06T09:31:11Z

If anyone wants to give this a look but hasn't found time yet, could you please leave a comment here? Happy to wait, but if no one is still wanting to review then I'd merge in the next day or two.

HISTORY.md

Bump minor version to 0.37.0

8135113

mhauru marked this pull request as draft April 24, 2025 16:47

github-actions bot assigned mhauru Apr 24, 2025

mhauru and others added 5 commits May 2, 2025 14:12

Replace PriorExtractorContext with PriorDistributionAccumulator (#907)

326d7ed

Implement values_as_in_model using an accumulator (#908)

d4ef1f2

* Implement values_as_in_model using an accumulator * Make make_varname_expression a function * Refuse to combine ValuesAsInModelAccumulators with different include_colon_eqs * Fix nested context test

Merge remote-tracking branch 'origin/main' into breaking

adfdda0

Bump DynamicPPL versions

d043103

penelopeysm added 6 commits June 2, 2025 15:08

Fix merge (1)

d9545c6

Add benchmark Pkg source

a243bbb

[no ci] Don't need to dev again

e2272a5

Disable use_closure for ReverseDiff

3cb47cd

Revert "Disable use_closure for ReverseDiff"

2d11ad7

This reverts commit 3cb47cd.

Fix LogDensityAt struct

0445092

penelopeysm added 2 commits June 2, 2025 18:00

Try not duplicating

ff7f8a2

Update comment pointing to closure benchmarks

80db9e2

mhauru mentioned this pull request Jun 3, 2025

Harmonise our use of !! #653

Open

Merge remote-tracking branch 'origin/main' into breaking

bec523a

penelopeysm mentioned this pull request Jun 19, 2025

Remove context from model evaluation (use model.context instead) #952

Merged

penelopeysm added 7 commits June 19, 2025 17:45

Mark function as Const for Enzyme tests (#957)

8b67e96

Move submodel code to submodel.jl; remove @submodel (#959)

1882f72

* Move submodel code to submodel.jl * Remove `@submodel`

Fix missing field tests for 1.12 (#961)

7f20709

Remove 3-argument {_,}evaluate!!; clean up submodel code (#960)

f20e86c

* Clean up submodel code, remove 3-arg `_evaluate!!` * Remove 3-argument `evaluate!!` as well * Update changelog * Improve submodel error message * Fix doctest * Add error hint for three-argument evaluate!!

Merge branch 'main' into breaking

57a53e1

Improve API for AD testing (#964)

a0289db

* Rework API for AD testing * Fix test * Add `rng` keyword argument * Use atol and rtol * remove unbound type parameter (?) * Don't need to do elementwise check * Update changelog * Fix typo

mhauru and others added 6 commits July 18, 2025 16:33

Fix behaviour of set_retained_vns_del! for num_produce == 0 (#1000)

05cd886

mhauru marked this pull request as ready for review August 1, 2025 15:39

mhauru requested a review from yebai August 1, 2025 15:39

yebai approved these changes Aug 1, 2025

View reviewed changes

mhauru mentioned this pull request Aug 1, 2025

Remove VariableOrderAccumulator and subset and merge on accs #1005

Merged

Merge branch 'main' into breaking

2ffc311

penelopeysm reviewed Aug 6, 2025

View reviewed changes

HISTORY.md Outdated Show resolved Hide resolved

Minor tweak to changelog wording

c98a67c

penelopeysm approved these changes Aug 6, 2025

View reviewed changes

mhauru added this pull request to the merge queue Aug 7, 2025

Merged via the queue into main with commit f0ac109 Aug 7, 2025
19 of 21 checks passed

mhauru deleted the breaking branch August 7, 2025 09:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Release v0.37 #901

Release v0.37 #901

Uh oh!

mhauru commented Apr 24, 2025 •

edited by penelopeysm

Loading

Uh oh!

codecov bot commented Apr 24, 2025 •

edited

Loading

Uh oh!

coveralls commented Apr 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 2, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

mhauru commented Aug 1, 2025

Uh oh!

mhauru commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

penelopeysm commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

mhauru commented Aug 4, 2025

Uh oh!

mhauru commented Aug 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Release v0.37 #901

Release v0.37 #901

Uh oh!

Conversation

mhauru commented Apr 24, 2025 • edited by penelopeysm Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

0.37 checklist

Accumulators

Fixes for particle methods

Remove context from evaluate!! and simplify submodel code

Replacing SamplingContext with InitContext

Fixes to AD testing

Shuffle code around

For 0.38

Uh oh!

codecov bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coveralls commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 15395885376

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Report for Commit c98a67c

Computer Information

Benchmark Results

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

mhauru commented Aug 1, 2025

Uh oh!

mhauru commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

penelopeysm commented Aug 1, 2025

Uh oh!

yebai commented Aug 1, 2025

Uh oh!

mhauru commented Aug 4, 2025

Uh oh!

mhauru commented Aug 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhauru commented Apr 24, 2025 •

edited by penelopeysm

Loading

Remove `context` from `evaluate!!` and simplify submodel code

Replacing `SamplingContext` with `InitContext`

codecov bot commented Apr 24, 2025 •

edited

Loading

coveralls commented Apr 24, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

Benchmark Report for Commit `c98a67c`