Add adversarial weight regularisation pipeline

[nikhilwoodruff]

Summary

• Adds a diagnostics package implementing the adversarial weight regularisation pipeline from the design doc
• Phase 1 (influence detector): Computes per-record influence across a reporting surface of 10 metrics × 4 slice dimensions (income decile, region, age band, tenure). Identifies records exceeding a configurable influence threshold, computes Kish effective sample sizes, and samples across random policy reforms.
• Phase 2 (generative model): Trains a TVAE on FRS input attributes via sdv, with conditional sampling using varied conditioning fractions for diverse offspring generation.
• Phase 3 (adversarial loop): Iteratively detects worst-offender records, generates synthetic offspring, and replaces high-weight records with weighted offspring.
• Phase 4 (regularised recalibration): Entropy-regularised weight optimisation with KL divergence penalty and optional hard weight cap.
• Adds a CLI (python -m policyengine_uk_data.diagnostics) with diagnose, train, and regularise commands.
• Adds visualisation script producing weight distribution, Kish ESS, influence heatmap, and scatter plots.

Current dataset diagnostics

Running Phase 1 on the enhanced FRS reveals:

• 53,508 households, median weight 33, max weight 372,747 (skewness 31.4)
• 274 records exceed the 5% influence threshold
• Overall Kish effective sample size: 930 (out of 53k records)
• Worst offender (HH #506) has 95.5% influence on housing_benefit_reported/age_band=16-24
• Most problematic slices: young age bands (16-24) and specific region×income combinations

Test plan

• All new modules pass ruff lint and format checks
• Syntax validation passes for all 6 new files
• Diagnostics script runs successfully on enhanced_frs_2023_24.h5
• Run full adversarial loop on a subset to verify convergence
• Compare pre/post weight distributions after regularisation