[API] multivariate distributions

[fkiraly]

Discussion issue for supporting multivariate distributions.

The key issue for support is API-wise: it would be nice if we could supply a unified API for distributions that are joint marginals, and those that are correlated.

I see two options to add multivariate support and maintain a unified API:

• new methods for joints, e.g., pdfj for joint row-wise pdf - as showcased in this PR: WIP [ENH] Multivariate distribution framework support #621
• include an argument marginal to all the methods, and it could be "entry" (default and current behaviour), "row", or "none"?
• or an argument joint, which could be "none" (current), "row", or "all"?

In both cases we will have the problem where there is a lack of clear defaults when going from joint to marginal - marginalization for pdf, for instance, requires integration over an infinite domain, but it could default to evaluates of cdfj via derivatives.

In either case, we need to define:

• method API for multivariate objects
• defaulting behaviour
• possibly dispatch frameworks as methods call each other and default to each other.
  • I wanted to avoid this, but we may have to introduce a default operation graph to avoid manual transitivitity checks etc

[fkiraly]

FYI @joshdunnlime, @felipeangelimvieira, your opinions would be appreciated.

[fkiraly]

Secondary design questions:

• should Normal be able to represent both marginal products of normals and the multivariate normal? We could use the init input to detect which of the two it is. Feels fiddly though.
• should we redefine the internals of losses? For instance, LogLoss could simply call log_pdfj (or pdf(joint="rows")) instead of the current implementation which sums row-wise.

[felipeangelimvieira]

Franz, regarding the models, should we really support marginals? I guess most models that output multi-variate probabilities try to approximate the joint density.

About the probabilities themselves, I understand it might be nice to have this feature. Couldn't we implement a method that marginalizes a distribution with respect to a dimension, and returns another distribution without that dimension?

Roughly something like:

dist = MultivariateNormal(0, [[1, 0.2],[-0.3,2]])

univariate_normal = dist.marginalize(axis=1)

Considering that .pdf will always be the joint density. LogLoss etc should always use the joint... I can't imagine an usecase where we would like logloss of the marginal?

[fkiraly]

Franz, regarding the models, should we really support marginals? I guess most models that output multi-variate probabilities try to approximate the joint density.

Marginals are needed by many ways to evaluate and are aligned with the current default returns of BaseDistribution methods.
So, I do think we need to support at least marginals by entry for distributions in general, or have some mechanism to deal with them.

About the probabilities themselves, I understand it might be nice to have this feature. Couldn't we implement a method that marginalizes a distribution with respect to a dimension, and returns another distribution without that dimension?

Very interesting idea. That would lead to a product joint?

This would internally use the same logic that gets pdf from pdfj (in the above naming)

Considering that .pdf will always be the joint density. LogLoss etc should always use the joint... I can't imagine an usecase where we would like logloss of the marginal?

pdf cannot be the joint without major deprecation, because currently it returns a DataFrame of same shape as self. Were it the joint, it would return a DataFrame with the same index but only a single column.

LogLoss etc should always use the joint

I do agree with this.

I can't imagine an usecase where we would like logloss of the marginal?

It might be a modelling choice?

[felipeangelimvieira]

This would internally use the same logic that gets pdf from pdfj (in the above naming)

yes

pdf cannot be the joint without major deprecation, because currently it returns a DataFrame of same shape as self. Were it the joint, it would return a DataFrame with the same index but only a single column.

currently the pdf for multivariate probabilistic problems assumes independence, is that right? It returns one column for each variable?

It might be a modelling choice?

yes, there can be use-cases, it's true. But for the losses, I understand that most methods would probably minimize the product of the marginals, which is equivalent of joint with assumption of independence? So logloss etc using the joint still seems better than using the average of the marginals or something like this

[felipeangelimvieira]

Should we go with .marginalize(axis) method to reduce M-variate distributions to N-variate? (N < M)

[fkiraly]

currently the pdf for multivariate probabilistic problems assumes independence, is that right? It returns one column for each variable?

That is not fully correct, it depends on what pdf should stand for.

The object that assumes column-wise independence is the Normal distribution currently.

Under the current design, even for a hypothetical MultivariateNormal object, the pdf - following a unified API - would compute evaluates at the independent marginal pdf.

[fkiraly]

Idea: we leave it at a single pdf method, but we introduce an argument axis, which means, which axes are joint.

pdf(x, axis=None) -> current row- and column marginals.

pdf(x, axis=1) -> marginals by row; columns are joint. The result is a single-column df.

pdf(x, axis=0) -> marginals by columns; rows are joint. The result is a single-row df.

pdf(x, axis=(0,1)) or pdf(x, axis="all") -> joint by rows and columns. The result is a np.float.

[felipeangelimvieira]

Sounds good to me. Internally, this would use the .marginalize method?
The marginalize could also be used when we want to marginalize only by a subset of columns

[fkiraly]

Sounds good to me. Internally, this would use the .marginalize method?
The marginalize could also be used when we want to marginalize only by a subset of columns

That, or some dispatch logic.
(we need to think about exact implementation)

[fkiraly]

@tingiskhan, do you have an opinion on the above?