Future option: Express mass/normalization of densities

[phipsgabler]

I've been previously thinking about whether and how a density interface should include the possibility to require or check for normalization, so I'm just posing this for discussion here.

Options that come to my mind:

# extra argument and dispatch
logdensityof(d, x; normalized=Val{false}())

Or maybe include a whole machinery for dealing with normalization constants?

# like eltype/length traits
hasnormalizingconstant(d) = false
normalizingconstant(d::SomeDensity) = ...
isnormalized(d) = normalizingconstant(d) == 1

# probably should be its own struct
normalize(d) = let logZ = log(normalizingconstant(d))
    logfuncdensity(x -> logdensityof(d, x) - logZ)
end

[oschulz]

Hm, I wonder if something like this could go into Distributions?

[phipsgabler]

The real use case of this IMHO is the query "is something a probability density". But that should then be available to PPLs too, since there we have all kinds of densities that we know are not normalized. So if this is deemed a good idea, I'd rather have it here.

[oschulz]

The real use case of this IMHO is the query "is something a probability density".

What do you think about my proposal at the (current) end of #5 regarding adding ismeasure and isdensity, as more specific queries than hasdensity? Would that serve this use case as well?

[phipsgabler]

I wouldn't think that makes a difference; being a measure doesn't tell you anything about normalization (or being a probability measure).

A third approach, where you only have to define normalizingconstant:

hasnormalizingconstant(d) = normalizingconstant(d) isa Nothing
normalizingconstant(d) = nothing
normalizingconstant(d::SomeDensity) = ...
isnormalized(d) = normalizingconstant(d) == 1

[oschulz]

We could call it logdensitynorm, and default it to logdensitynorm(d) == missing. Not sure we'd need a haslogdensitynorm, with a default of missing.

[oschulz]

Having a logdensitynorm could be useful in applications that need to operate in non-log space (e.g. subvolume integration) but need to renormalize (shift log-space) before doing exp(logdensityof(...)) because they would be out of the dynamic range of their float-type otherwise.

[phipsgabler]

I think I like that idea. It seemed to me that it would be nice also be able to express "infinite norms" (as there is a semantic difference between "finite but unknown constant", as in posteriors, and "known to be non-normalizable"). With your approach, it would work out, right?

logdensitynorm(d) = missing
logdensitynorm(d::Distribution) = 1
densitynorm(d) = exp(logdensitynorm(d))

will propagate the missing, as do subsequent computations. And then we can also have

densitynorm(d::Lebesgue) = Inf
# default: logdensitynorm(d::Lebesgue) = missing

with somewhat consistent behaviour.

So it seems nice on the first glance, but I'm not confident enough to judge downstream effects.

[oschulz]

Yes, I think so. But maybe the name logdensitynorm would be weird when used on measures/distributions - their norm is independent from which density is derived for them using different base measures. May we should just use LinearAlgebra.norm, and add lognorm on our own?

Then, for Bayesian inference, we'd have lognorm(∫exp(log_likelihood, prior)) == log_evidence.

[oschulz]

Hm, maybe using LinearAlgebra.norm is a bad idea, though, it's not specific enough. One might expect that norm(f_density) == sqrt(∫f_density(x)^2 dx), or something else. There are so many norms ...

[phipsgabler]

Yeah, I really wouldn't use LinearAlgebra.norm; for distributions, it's also already taken (JuliaStats/Distributions.jl#1321). And for that reason, if something like this is implemented, we should probably refrain from "norm" as the name.

[oschulz]

What could we call it so that it would also make sense for densities (as the integral of the density function over it's support)?

[oschulz]

What is the norm of MeasureTheory.Dirichlet([2,3]), and can I compute it with `MeasureTheory?

[cscherrer]

"Norm" and "normalize" are very overloaded, which is why we went with mass and logmass
cc @sethaxen

[oschulz]

I like those. I think defining those as abstract function in DensityInterface could be very handy. We often do use the term "norm of a density", we say the "PDF is normalized" and so on. So it's not just useful for MeasureBase, it's also useful for densities (density functions). We do of course mean the integral in respect to an implied base measure - but that's the same as using an implied base measure in `[log]densityof], so it would fit in quite neatly.

What do you think about this, @devmotion ?

[cscherrer]

Here's some more context:
JuliaMath/MeasureTheory.jl#128

For a trait-based approach, I can see a few situations we'd want to distinguish:

1. Measures that are normalized
2. Measures with a known mass, which may or may not be one
3. Measures known to be finite, with unknown mass
4. Measures known to have infinite mass
5. Measures where the mass is unknown (maybe finite, maybe infinite)