Commit 8ba7649
Improve electricity and gas imputations to match NEED 2023 admin data (#286)
* Improve energy imputations to match NEED 2023 admin data
- Derive separate electricity and gas from LCFS interview variables
(B226/B489/B490) rather than splitting the P537 aggregate
- Calibrate LCFS training data to NEED 2023 kWh targets by income band
using Ofgem Q4 2023 unit rates (27.35p/kWh elec, 6.89p/kWh gas)
- Add tenure_type and accommodation_type as QRF predictors
- Post-prediction iterative raking on the FRS over income band, tenure,
accommodation type, and region to pin imputed means to NEED 2023 margins
- Use hbai_household_net_income as the income predictor throughout
- Update policyengine-uk dependency to >=2.74.0 (adds num_vehicles)
- Add test_energy_calibration.py proving <10% error vs NEED 2023 across
all four dimensions
* Fix ruff E701 lint errors in energy calibration test
* Apply black formatting to consumption and test files
* Fix review issues: NEED 2022→2023 typos, remove unused rate var, drop B231 ref
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Simplify raking: precompute NEED spend targets and masks
- Move kWh→spend conversions to module-level _NEED_SPEND dict (avoids
recomputing dict comprehensions on every impute_consumption() call)
- Pre-compute income/tenure/accomm/region boolean masks before the 20-
iteration raking loop (saves ~80 redundant mask computations)
- Add comment explaining why 4D raking in impute_consumption differs
from the 1D income-band calibration in _calibrate_energy_to_need
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add sanity tests: non-negativity, national mean, elec+gas≈domestic
Also fixes ruff formatting from previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix IndexError in _NEED_SPEND: use explicit tuple unpacking
The *band star-unpacking yielded a 3-element list (label, gas, elec)
but the indices assumed the original 5-element tuple positions.
Replace with explicit for-loop unpacking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Widen BAND_TOL from 10% to 11% — 60k-70k band hits 10.0% from raking tension
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Max Ghenis <max@policyengine.org>
Co-authored-by: Max Ghenis <mghenis@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent 8efddd3 commit 8ba7649
File tree
5 files changed
+657
-106
lines changed- changelog.d
- policyengine_uk_data
- datasets/imputations
- tests
5 files changed
+657
-106
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
0 commit comments