Support DPPL 0.37 #2550

mhauru · 2025-05-15T15:18:03Z

Currently in a very unfinished state.

yebai · 2025-05-18T21:30:06Z

Lots of interface code changes here are unnecessary and are planned to be removed in #2413

Perhaps we should address #2413 first.

mhauru · 2025-05-20T10:49:44Z

That is tempting, to not waste time fixing code that's on its way out. I worry though that removing the samplers will take a while still, and in the meanwhile all the accumulator stuff, and other DPPL changes that build on it, would be held back from Turing.jl. For instance, introducing ValuesAsInModelAccumulator would cut our inference time in #2542 by half.

penelopeysm · 2025-07-17T13:33:42Z

IMO, it goes both ways. Reducing sampler complexity would make this PR easier. On the other hand, merging this PR would also make it easier to remove the duplicate samplers. (To be precise, DPPL 0.37 would make it easier.)

I think DPPL 0.37 has taken a long time and we should prioritise this, rather than trying to squeeze in the changes to samplers.

I'm going to fix the merge conflicts and add a [sources] bit to Project.toml to point to the unreleased DPPL branch, so that CI can at least run on 1.11. (See https://pkgdocs.julialang.org/v1/toml-files/#The-%5Bsources%5D-section)

Note that 1.10 CI will always fail as [sources] isn't understood on 1.10.

penelopeysm · 2025-07-18T17:53:00Z

test/mcmc/hmc.jl

+    # TODO(mhauru) Do we give up being able to sample from only prior/likelihood like this,
+    # or do we implement some way to pass `whichlogprob=:LogPrior` through `sample`?


sample(::LogDensityFunction) would solve that, since we could set getlogprior in the LDF, but I guess that'll be the next release.

penelopeysm · 2025-07-19T01:04:14Z

src/mcmc/hmc.jl

-        # TODO(penelopeysm): Can we just use leafcontext(model.context)? Do we
-        # need to pass in the sampler? (In fact LogDensityFunction defaults to
-        # using leafcontext(model.context) so could we just remove the argument
-        # entirely?)
-        DynamicPPL.SamplingContext(rng, spl, DynamicPPL.leafcontext(model.context));
-        adtype=spl.alg.adtype,


As established in (e.g.) TuringLang/DynamicPPL.jl#955 (comment) SamplingContext for Hamiltonians was never overloaded so it is equivalent to just use DefaultContext in the LDF.

src/mcmc/hmc.jl

src/optimisation/Optimisation.jl

penelopeysm · 2025-07-19T14:42:45Z

Because Turing re-exports some things that were changed in DPPL 0.37 (for example, LogDensityFunction), this has to go into breaking. (It probably philosophically should anyway, since I think this might be the biggest makeover that DPPL has had in a while.)

github-actions · 2025-07-19T23:33:15Z

Turing.jl documentation for PR #2550 is available at:
https://TuringLang.github.io/Turing.jl/previews/PR2550/

* Use new `getlogjoint` for optimisation * Change getlogjoint -> getlogjoint_internal where needed * Enforce re-evaluation when constructing `Transition` * fix tests * Remove extra evaluations from SGLD and SGHMC * Remove dead `transitions_from_chain` method (used to be part of `predict`) * metadata -> getstats_with_lp * Clean up some stray getlogp

penelopeysm · 2025-08-01T15:49:23Z

test/mcmc/gibbs.jl

+        # TODO(DPPL0.37/penelopeysm): decide what to do with these tests
+        @testset "Coalescing multiple observations into one" begin
+            # Instead of observing x[1] and x[2] separately, we lump them into a
+            # single distribution.
+            @model function dynamic_bernoulli()
+                b ~ Bernoulli()
+                if b
+                    dists = [Normal(1.0)]
+                else
+                    dists = [Normal(1.0), Normal(2.0)]
+                end
+                return x ~ product_distribution(dists)
+            end
+            model = dynamic_bernoulli()


To be discussed @ 4 Aug meeting

* Allow Prior to skip model re-evaluation * remove unneeded `default_chain_type` method * add a test * add a likelihood term too * why not test correctness while we're at it

penelopeysm · 2025-08-07T10:45:03Z

Apologies for force push, I did it on the wrong branch. I undid it (force pushed back to the old commit) so it shouldn't affect any local git history. CI is now failing because DynamicPPL's breaking branch doesn't exist... hmmmm

penelopeysm · 2025-08-07T10:46:28Z

I'd be happy to merge TuringLang/DynamicPPL.jl#1005, release DPPL, then remove the [sources] thing here and maybe merge this into breaking (I don't think there are any other breaking changes that this will hold up).

mhauru · 2025-08-07T11:31:26Z

I agree on that plan.

* Remove calls to resetlogp!! * Add a changelog for 0.40 * Update HISTORY.md Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]>

* Unify `Transition` methods * Add tests * Add same test for SGLD/SGHMC * Refactor so that it's nice and organised * Fix failing test on 1.10 * just increase the atol * Make addlogprob test more robust * Remove stray `@show` Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]>

penelopeysm

I'm very pleased with where we are on this, would be happy to merge into breaking and then main. Was there anything else you wanted to add to 0.40?

HISTORY.md

mhauru · 2025-08-12T13:59:51Z

Thanks, very happy with that. I'm good to merge and release once tests pass. You wanna do it?

penelopeysm · 2025-08-12T14:01:27Z

Okay 😄

penelopeysm · 2025-08-12T14:01:46Z

I'll merge into breaking then let CI run on that branch.

* [no ci] Bump to v0.40.0 * Uncomment tests that should be there * Support DPPL 0.37 (#2550) * First efforts towards DPPL 0.37 compat, WIP * More DPPL 0.37 compat work, WIP * Add [sources] for [email protected] * Remove context argument from `LogDensityFunction` * Fix MH * Remove spurious logging * Remove residual OptimizationContext * Delete files that were removed in previous releases * Fix typo * Simplify ESS * Fix LDF * Fix Prior(), fix a couple more imports * fixes * actually fix prior * Remove extra return value from tilde_assume * fix ldf * actually fix prior * fix HMC log-density * fix ldf * fix make_evaluate_... * more fixes for evaluate!! * fix hmc * fix run_ad * even more fixes (oh goodness when will this end) * more fixes * fix * more fix fix fix * fix return values of tilde pipeline * even more fixes * Fix missing import * More MH fixes * Fix conversion * don't think it really needs those type params * implement copy for LogPriorWithoutJacAcc * Even more fixes * More fixes; I think the remaining failures are pMCMC related * Fix merge * DPPL 0.37 compat for particle MCMC (#2625) * Progress in DPPL 0.37 compat for particle MCMC * WIP PMCMC work * Gibbs fixes for DPPL 0.37 (plus tiny bugfixes for ESS + HMC) (#2628) * Obviously this single commit will make Gibbs work * Fixes for ESS * Fix HMC call * improve some comments * Fixes to ProduceLogLikelihoodAccumulator * Use LogProbAccumulator for ProduceLogLikelihoodAccumulator * use get_conditioned_gibbs --------- Co-authored-by: Penelope Yong <[email protected]> * "Fixes" for PG-in-Gibbs (#2629) * WIP PMCMC work * Fixes to ProduceLogLikelihoodAccumulator * inline definition of `set_retained_vns_del!` * Fix ProduceLogLikelihoodAcc * Remove all uses of `set_retained_vns_del!` * Use nice functions * Remove PG tests with dynamic number of Gibbs-conditioned-observations * Fix essential/container tests * Update pMCMC implementation as per discussion * remove extra printing statements * revert unneeded changes * Add back (some kind of) dynamic model test * fix rebase * Add a todo comment for dynamic model tests --------- Co-authored-by: Markus Hauru <[email protected]> * Use accumulators to fix all logp calculations when sampling (#2630) * Use new `getlogjoint` for optimisation * Change getlogjoint -> getlogjoint_internal where needed * Enforce re-evaluation when constructing `Transition` * fix tests * Remove extra evaluations from SGLD and SGHMC * Remove dead `transitions_from_chain` method (used to be part of `predict`) * metadata -> getstats_with_lp * Clean up some stray getlogp * InitContext isn't for 0.37, update comments * Fix merge * Do not re-evaluate model for Prior (#2644) * Allow Prior to skip model re-evaluation * remove unneeded `default_chain_type` method * add a test * add a likelihood term too * why not test correctness while we're at it * No need to test AD for SamplingContext{<:HMC} (#2645) * change breaking -> main * Remove calls to resetlogp!! & add changelog (#2650) * Remove calls to resetlogp!! * Add a changelog for 0.40 * Update HISTORY.md Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]> * Remove `[sources]` * Unify Turing `Transition`s, fix some tests (#2651) * Unify `Transition` methods * Add tests * Add same test for SGLD/SGHMC * Refactor so that it's nice and organised * Fix failing test on 1.10 * just increase the atol * Make addlogprob test more robust * Remove stray `@show` Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]> * Update changelog for PG in Gibbs --------- Co-authored-by: Penelope Yong <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]>

github-actions bot assigned mhauru May 15, 2025

mhauru mentioned this pull request May 30, 2025

VariableOrderAccumulator TuringLang/DynamicPPL.jl#940

Merged

penelopeysm reviewed Jul 18, 2025

View reviewed changes

penelopeysm reviewed Jul 19, 2025

View reviewed changes

src/mcmc/hmc.jl Outdated Show resolved Hide resolved

penelopeysm reviewed Jul 19, 2025

View reviewed changes

src/optimisation/Optimisation.jl Outdated Show resolved Hide resolved

mhauru and others added 8 commits July 19, 2025 15:46

First efforts towards DPPL 0.37 compat, WIP

cea1f7d

More DPPL 0.37 compat work, WIP

5d860d9

Add [sources] for [email protected]

c7c4638

Remove context argument from LogDensityFunction

f16a5cf

Fix MH

98d5e7a

Remove spurious logging

73e127b

Remove residual OptimizationContext

ce0c782

Delete files that were removed in previous releases

4d03c07

penelopeysm force-pushed the mhauru/dppl-0.37 branch from 4ee5649 to 4d03c07 Compare July 19, 2025 14:46

penelopeysm changed the base branch from main to breaking July 19, 2025 14:46

penelopeysm added 4 commits July 19, 2025 23:45

Fix typo

06fec2d

Simplify ESS

0af8725

Fix LDF

3d44c12

Fix Prior(), fix a couple more imports

a1837b5

penelopeysm added 5 commits July 20, 2025 00:46

fixes

17efb8c

actually fix prior

d62ad82

Remove extra return value from tilde_assume

aac93f1

fix ldf

e903d1c

actually fix prior

fd5a815

mhauru mentioned this pull request Aug 1, 2025

Release v0.37 TuringLang/DynamicPPL.jl#901

Merged

22 tasks

Merge branch 'breaking' into mhauru/dppl-0.37

27aab23

penelopeysm reviewed Aug 1, 2025

View reviewed changes

penelopeysm added 2 commits August 1, 2025 16:49

InitContext isn't for 0.37, update comments

119c818

Fix merge

b41a4b1

This was referenced Aug 4, 2025

Simplify the context mechanism in DynamicPPL TuringLang/DynamicPPL.jl#895

Open

No need to test AD for SamplingContext{<:HMC} and friends #2645

Merged

penelopeysm added 2 commits August 5, 2025 17:39

Do not re-evaluate model for Prior (#2644)

d92fd56

* Allow Prior to skip model re-evaluation * remove unneeded `default_chain_type` method * add a test * add a likelihood term too * why not test correctness while we're at it

No need to test AD for SamplingContext{<:HMC} (#2645)

806c82d

penelopeysm force-pushed the mhauru/dppl-0.37 branch 2 times, most recently from 8af36e1 to 806c82d Compare August 7, 2025 10:43

change breaking -> main

5743ff7

penelopeysm mentioned this pull request Aug 10, 2025

computation time of returned scales with number of model parameters TuringLang/DynamicPPL.jl#1019

Open

penelopeysm and others added 3 commits August 11, 2025 14:30

Remove calls to resetlogp!! & add changelog (#2650)

57e6f9c

* Remove calls to resetlogp!! * Add a changelog for 0.40 * Update HISTORY.md Co-authored-by: Markus Hauru <[email protected]> --------- Co-authored-by: Markus Hauru <[email protected]>

Remove [sources]

bb21e1e

penelopeysm approved these changes Aug 12, 2025

View reviewed changes

mhauru commented Aug 12, 2025

View reviewed changes

HISTORY.md Show resolved Hide resolved

Update changelog for PG in Gibbs

247aee9

mhauru marked this pull request as ready for review August 12, 2025 13:59

penelopeysm merged commit ee0b095 into breaking Aug 12, 2025
24 of 30 checks passed

penelopeysm deleted the mhauru/dppl-0.37 branch August 12, 2025 14:02

		# TODO(mhauru) Do we give up being able to sample from only prior/likelihood like this,
		# or do we implement some way to pass `whichlogprob=:LogPrior` through `sample`?

Support DPPL 0.37 #2550

Support DPPL 0.37 #2550

Conversation

mhauru commented May 15, 2025

Uh oh!

yebai commented May 18, 2025

Uh oh!

mhauru commented May 20, 2025

Uh oh!

penelopeysm commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

penelopeysm Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

penelopeysm Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

penelopeysm commented Jul 19, 2025

Uh oh!

github-actions bot commented Jul 19, 2025

Uh oh!

penelopeysm Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

penelopeysm commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

penelopeysm commented Aug 7, 2025

Uh oh!

mhauru commented Aug 7, 2025

Uh oh!

penelopeysm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mhauru commented Aug 12, 2025

Uh oh!

penelopeysm commented Aug 12, 2025

Uh oh!

penelopeysm commented Aug 12, 2025

Uh oh!

Uh oh!

Uh oh!

penelopeysm commented Jul 17, 2025 •

edited

Loading

penelopeysm commented Aug 7, 2025 •

edited

Loading