Skip to content

Add Fisher Information Matrix support and natural-gradient training#60

Open
ujohn33 wants to merge 3 commits intoStatMixedML:masterfrom
ujohn33:feat/fim-all-dists
Open

Add Fisher Information Matrix support and natural-gradient training#60
ujohn33 wants to merge 3 commits intoStatMixedML:masterfrom
ujohn33:feat/fim-all-dists

Conversation

@ujohn33
Copy link

@ujohn33 ujohn33 commented Dec 22, 2025

This PR adds natural gradient descent support to LightGBMLSS by introducing per-distribution Fisher Information Matrix (FIM) scaling of gradients.

What’s included

  1. Extend DistributionClass with a natural_gradient flag and a default compute_fisher_information_matrix() method (fallback returns ones).
  2. Apply natural-gradient updates in compute_gradients_and_hessians() when natural_gradient=True (gradient scaled by FIM diagonal).
  3. Implement closed-form / efficient FIM diagonal computations for several univariate distributions based on the source [Miyamoto et al.] https://arxiv.org/abs/2304.14885 (e.g., Gaussian, Cauchy, Laplace, LogNormal, Logistic, Poisson, StudentT, Weibull, Gumbel), with an optimized branch for response_fn="exp" and a generic autograd-based derivative fallback for other response functions.
  4. Add unit tests aligned with the existing test structure to validate:
    • FIM output shapes and positivity/finite values across distributions
    • presence/behavior of the natural_gradient parameter
    • basic end-to-end training sanity for natural_gradient=True

Notes
Natural gradient is not enabled by default; existing behavior is unchanged unless natural_gradient=True.
CRPS currently falls back to standard gradients (a warning is emitted), as natural gradient scaling is implemented for NLL.

@ujohn33
Copy link
Author

ujohn33 commented Jan 27, 2026

@StatMixedML @nucflash please have a look once you find time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant