refactor(estimation): explicit array and weight domains - #1500
Draft
s3alfisc wants to merge 10 commits into
Draft
Conversation
s3alfisc
force-pushed
the
refactor/weight-within-domains
branch
from
September 3, 2026 20:57
f5acf74 to
94c8c07
Compare
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Linear and IV fits now consume unpremultiplied within arrays plus observation weights; the square-root transform is local to the solvers. Residuals are stored in response units and scores carry the weights explicitly, so covariance code no longer undoes a stored transform.
The IRLS solver now returns a typed working state instead of overwriting the observation-weight alias, so a weighted GLM keeps its user weights available after the fit. Leverage and fixed-effect recovery ask the model which weights their estimating equation uses, and square-root-weighted arrays stay local to the solver.
Separating observation weights from IRLS working weights fixes the shape error reported in #367, so the skipped Poisson frequency-weight comparison runs again and a new R fixest comparison checks the aggregate fit against the literal row expansion. The R comparison sample keeps only fixed-effect levels with more than one physical row. A level holding a single row with a frequency weight above one is a singleton for pyfixest but not after expansion, and that deliberate deviation is not what this test is meant to measure.
Separation removes rows after the observation weights have been fixed, so the effective sample size must keep following those weights: it feeds the small-sample corrections, and a frequency-weighted aggregate fit must keep matching its literal row expansion after separated rows are dropped. Unweighted GLMs are unaffected because the effective and physical counts coincide.
Compare frequency-weighted OLS and 2SLS coefficients and heteroskedastic covariance, with and without fixed effects, against R fixest fitted on the literal row expansion. The sample keeps only fixed-effect levels with more than one physical row, since singleton detection deliberately counts rows.
Both methods read the retained design and response arrays, which are now kept on within scale without the square-root weight transform. A weighted ccv() would therefore compute on the wrong scale, and an in-place update() cannot rebuild the formula, prediction, and inference state that the appended rows would invalidate. Reject both explicitly, and turn the unsupported-estimator assert in ccv() into a NotImplementedError so it is not stripped under python -O.
The typed state objects become the only stored representation of a fit. `_X`, `_Y`, `_Z`, `_weights`, `_endogvar` and the GLM `_irls_weights` and `_Xbeta` are now read-only views over `_within_data`, `_observation_weights` and `_working_state`, so no code path can publish a second, differently scaled copy of the same array under a familiar name. `_weights` materializes its ones column on access instead of at fit time, and the remaining internal readers take `_observation_weights` directly. The response-scale dependent variable moves from the `_Y_untransformed` frame to a flat `_response` array. `_weights_df`, `_offset_df`, `_scores_response` and `_scores_working` had no readers and are dropped. Deletion lists now name the state objects rather than the aliases, because `delattr` on a property raises; the lean contract is unchanged.
Storage options discard fit arrays and the estimation sample, and every method that needed them failed with a bare `AttributeError` naming a private field. Two guards on `Feols` now name the method, say which storage option removed what, and give the remedy. `fixef()` checks its own support conditions before touching weights, so its `ValueError` and `NotImplementedError` still surface first. `vcov(..., data=)` becomes usable after `store_data=False`: the explicit sample is validated against the fitted row count and threaded through the HAC and cluster paths, and the update is refused before any inference state is overwritten, so a rejected call leaves the previous covariance estimate intact. The lean guard tracks the cleanup rather than the `lean` request, because cleanup runs at the end of the fit while estimation itself calls `vcov()`, `resid()`, `get_performance()` and `residualize()`.
The array aliases are read-only properties now, so the cleanup lists name the state objects that back them rather than the alias names; deleting a property would raise. Lean fits additionally drop the demeaning caches, the GLM working state, the Poisson null fit, and the quantile solver outputs, none of which any post-estimation path reads. A retained IV first stage is a full fitted model, so it inherits the parent's store_data setting instead of keeping a copy of data the parent was asked to discard. Effective-F diagnostics keep working because they read the first stage's retained arrays, not its input frame.
Records the state contract the refactor actually implements: which value carries which domain, that the legacy names are read-only views over those values, and how the storage options now apply across the result graph. Singleton detection stays physical-row based under frequency weights, so an aggregate row alone in its level is dropped although its literal expansion would survive. That deviation was previously undocumented and untested; a documenting test in tests/test_wls_types.py now pins it.
s3alfisc
force-pushed
the
refactor/weight-within-domains
branch
from
September 3, 2026 21:23
94c8c07 to
509e9ef
Compare
s3alfisc
force-pushed
the
refactor/weight-within-domains
branch
from
September 3, 2026 21:49
509e9ef to
ebe65a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 2 of 3, based on
refactor/formula-data-state.internals/model_state.py:ObservationWeights,WithinLinearData,GlmWorkingState, and array-nativeDemeanedDatacache entries.weights_type = "fweights"andvcov = "hetero"#367)._Y,_X,_Z, and_weightsbecome read-only properties over the typed state.store_data=Falseandlean=Trueapply to the whole result graph, including retained IV first stages. Methods that need discarded state raise an informativeRuntimeError.ccv()and in-placeupdate()are rejected. The effective sample size is kept after separation.