Conversation
New script `create_default_structure_tariffs.py` reads monthly_rates YAML
and builds URDB v7 tariff JSON that preserves the actual utility rate
structure (flat-with-variation, seasonal tiered, seasonal TOU) instead
of collapsing everything into a single volumetric rate.
Also: generalize `copy_flat_to_nonhp_flat.py` with a `--pattern` arg so
it can mirror any `_{pattern}.json` to `_nonhp_{pattern}.json`, and add
`create_seasonal_tiered_tariff()` to `create_tariff.py`.
Made-with: Cursor
New recipes: `create-default-structure-tariffs` and `copy-default-structure-to-nonhp-all`. Both are wired into `all-pre`. New variable `base_tariff_pattern` (env `BASE_TARIFF_PATTERN`, default "flat") controls which base tariff runs 1-4 use and which calibrated tariff runs 5/6 derive the seasonal discount from. Parameterize `create-seasonal-tou-tariffs` to read the fixed charge from the configured base pattern instead of always using flat. Made-with: Cursor
Set BASE_TARIFF_PATTERN=default in ri/state.env. Update scenarios_rie.yaml to reference rie_default/rie_nonhp_default tariffs and tariff maps for runs 1-4 (precalc/default) and 5-16 (seasonal discount and TOU vs default). Add generated default-structure tariff JSONs (3-period delivery, 4-period supply) and corresponding tariff maps for RIE. Made-with: Cursor
Calibrated default tariffs and tariff maps from precalc runs 1-2. Updated seasonal and TOU tariffs now derive from default-calibrated base (instead of flat-calibrated), changing rates and fixed charges. Subclass revenue requirements updated with new run source directory. Made-with: Cursor
6 tasks
Generated from monthly_rates YAMLs via create-default-structure-tariffs. Delivery and supply variants for all 7 utilities: ConEd, CenHud, NiMo, NYSEG, RG&E, O&R, PSEG-LI. Made-with: Cursor
Sets BASE_TARIFF_PATTERN=default so runs 1-4 use the actual utility rate structure, and USE_RESSTOCK_LOADS=true for monthly rate x load covariance in revenue topups. Made-with: Cursor
Plan A (superseded): structure-preserving approach that clones default tariff and scales winter rates. Rejected due to mixed-period splitting complexity across 6/7 NY utilities. Plan B (active): simplified flat seasonal using revenue-based rate derivation from run-1 bill outputs. Fixes the _extract_default_rate bug when BASE_TARIFF_PATTERN=default while preserving existing flat behavior. Includes revenue neutrality proof, adversarial review, TOU/flex independence analysis, and comparison table. Made-with: Cursor
- compute_subclass_rr: monthly bills + loads for summer/winter flat rates; --base-tariff-json for URDB fixed charge; new CSV columns (summer_rate, rev_*) - Thread base_tariff_json through Justfiles (run-5/run-6 calibrated paths) - Tests: URDB fixtures, flat equivalence, missing base tariff error Made-with: Cursor
Non-HP default-structure tariffs are not seasonal; checking them caused false failures. Add regression test. Made-with: Cursor
- run_orchestration: BASE_TARIFF_PATTERN and revenue-based flat derivation - Update Plan B plan; remove superseded Plan A doc - context/README: plans index matches repo (drop Plan A row) Made-with: Cursor
Made-with: Cursor
- rev_requirement: new source_run_dir (ri_20260323) and subclass RR floats - Calibrated/seasional tariff rate tweaks from regenerated outputs - scenarios YAML: drop stray blank lines; JSON EOF fixes where applicable Made-with: Cursor
…ed flat + winter discount
Replaces the season-specific revenue-weighted rate formula in
`compute_hp_seasonal_discount_inputs` with a cleaner construction that is
directly grounded in the HP subclass revenue requirement:
equivalent_flat_rate = annual_energy_rev_HP / annual_kWh_HP
summer_rate = equivalent_flat_rate
winter_rate = equivalent_flat_rate − CS_HP / winter_kWh_HP
where annual_energy_rev_HP = Σ weight_i × (annual_bill_i − 12×FC), read
from the Annual row of elec_bills_year_target.csv — the same row that
compute_subclass_rr uses to derive RR_HP.
**Why this matters**
The old formula split revenue by season (rev_summer/summer_kWh and
(rev_winter−CS)/winter_kWh). For flat base tariffs the two are identical,
but for structured tariffs (BASE_TARIFF_PATTERN=default) the seasonal revenue
split reflected the default tariff's asymmetry rather than the HP subclass
revenue requirement. This caused unnecessary per-building redistribution
during CAIRO precalc and the hp_seasonal tariff drifted away from rate_unity=1.
The new formula is revenue-neutral by construction:
Rev_HP = 12·FC·N + flat·total_kWh − CS
= 12·FC·N + annual_energy_rev_HP − CS
= Total_Bills_HP − CS = RR_HP
Because the Annual row is exactly what produced RR_HP, rate_unity at CAIRO
precalc is 1.0 up to floating-point precision. Any residual drift comes
only from CAIRO minimum bill floors, not from the rate derivation formula.
Using the Annual row (one row per building) also simplifies the code: the
monthly-bills seasonal grouping, winter_month_names computation, and
rev_summer/rev_winter intermediates are all removed.
**Output CSV changes**
Old columns removed: rev_summer_energy_hp, rev_winter_energy_hp
New columns added: annual_energy_rev_hp, equivalent_flat_rate, winter_discount, annual_kwh_hp
**Tests**
- Updated all seasonal-discount test fixtures to supply correct Annual bill
values (previously placeholder 0.0) so the new Annual-row read has real data.
- Updated assertions to check annual_energy_rev_hp, equivalent_flat_rate,
and winter_discount instead of the removed seasonal revenue columns.
- Added test_compute_hp_seasonal_discount_inputs_structured_tariff: verifies
that a tariff with asymmetric winter/summer filed rates produces the blended
flat rate (not the season-specific rate) and passes the revenue-neutrality
identity Rev_HP = RR_HP.
…lling kWh Our seasonal-rate derivation, TOU derivation, and validation were using out.electricity.net which diverges from out.electricity.total even for non-solar buildings, causing ~10% calibration drift. CAIRO bills on grid_cons = max(total - abs(pv), 0), so replicate that formula via a shared grid_consumption_expr() helper in utils/loads.py. Update all consumers (compute_subclass_rr, validate/load, hourly_resstock_load) and tests.
_extract_tou_supply_monthly processed charges in YAML iteration order. For PSEGLI, flat riders (merchant_function_charge, securitization) appeared before the TOU-structured supply_commodity_bundled, causing them to accumulate into a _flat key that was silently ignored by the caller. Net effect: ~$0.018/kWh missing from every supply rate. Rewrite as two-pass: collect TOU and flat charges separately, then distribute flat charges across all established TOU slot keys. This is order-independent. Made-with: Cursor
Regenerated default-structure, flat, nonhp_flat, and seasonal TOU tariffs for all 7 NY utilities using resstock monthly loads (USE_RESSTOCK_LOADS=true). Includes PSEGLI default_supply fix (flat riders now correctly folded into TOU slot rates). Made-with: Cursor
Switched from EIA-861 annual kWh to resstock monthly loads for volumetric budget calculation (USE_RESSTOCK_LOADS=true). Updates total_residential_kwh, topup budgets, and per-charge breakdowns for all 7 NY utilities. Made-with: Cursor
Replace _flat with _default across all 7 NY utility scenario YAMLs (run names, tariff paths, tariff map references, output paths). Runs 1-4 now use default-structure tariffs that preserve seasonal/tiered/TOU rate structure; runs 5-16 use nonhp_default instead of nonhp_flat for the non-HP subclass. Made-with: Cursor
New nonhp_default and nonhp_default_supply tariff JSONs for runs 5-16 where non-HP customers retain the utility's actual rate structure instead of a simplified flat rate. Made-with: Cursor
Made-with: Cursor
- Fix TOU supply monthly extraction (flat riders into slots) - Build energyratestructure only for seasons used per period group - Add tests for dead-rate entries and flat rider inclusion Made-with: Cursor
- Switch scenario paths from _flat to _default where applicable - Update hp_vs_nonhp rev requirement YAMLs for default structure - Refresh OR gas tariff map CSVs Made-with: Cursor
- Refresh delivery/supply tariffs for all utilities after generator fix - Add calibrated and nonhp_default calibrated copies where generated - Add flex TOU tariffs for OR, PSEGLI, RGE where missing Made-with: Cursor
- Default / calibrated / supply variants per utility - HP seasonal and seasonal-TOU vs default and flex-vs-default maps Made-with: Cursor
Replace the managed `aws_ebs_volume` Terraform resource with a data source lookup filtered by Name and Persistent=true tags. When the local state file was lost (git-ignored, never committed), `terraform apply` would see no existing volume and create a new one — orphaning the old one. This happened at least four times, accumulating three abandoned 500 GB volumes. With the data source approach, losing state is harmless: Terraform looks up the existing volume by tag instead of owning it, so it can never accidentally create a duplicate or destroy data on teardown. Add `infra/create-volume.sh` and `just create-volume` as the one-time volume creation path. The script is idempotent and tags the volume Persistent=true so the data source finds exactly one match. Made-with: Cursor
…ture-tariff-generation-from-monthly_rates-yaml
Runs 17-20 are not part of the current NY batch (r1-16). Comment them out in both run-all-sequential and run-all-parallel-tracks so a batch can be kicked off without manually bypassing those recipes. Made-with: Cursor
Documents the full Genability → monthly_rates YAML → URDB v7 JSON pipeline: charge classification (already_in_drr / add_to_drr / add_to_srr), monthly_rates YAML structure, create_flat_tariffs.py (top-down from RR), create_default_structure_tariffs.py (bottom-up from filed rates), BASE_TARIFF_PATTERN wiring, CAIRO calibration, and why flat vs default rates differ. Made-with: Cursor
- hp_vs_nonhp YAMLs: update source_run_dir to ny_20260325b_r1-16 and refresh subclass revenue requirements from new run-1 BAT outputs - NYSEG gas tariff maps: row-order change only (regenerated by all-pre) - Pre-calibrated tariff JSONs: minor float precision tweaks and trailing newline fixes from re-running all-pre across all utilities - Calibrated tariff JSONs: updated rates from ny_20260325b CAIRO runs - New tariff files: cenhud/rge _hp_flat.json, coned_nonhp_flat and nimo_hp_seasonalTOU_flex_supply calibrated Made-with: Cursor
The patch's fallback condition checked only solar_compensation_df (the sell_rate dict) to decide whether to fall back to CAIRO's per-building Dask loop. When run_scenario.py passes solar_pv_compensation=None to simulate(), the sell_rate dict is still populated, triggering an unnecessary fallback for all NY utilities (~15k buildings processed one-by-one instead of vectorized). Fix: check solar_compensation_style in addition to solar_compensation_df. When style is None, CAIRO's calculate_compensation returns an empty DataFrame regardless of sell_rate, so no fallback is needed. When style is "net_metering", compute compensation inline: build a sell-rate lookup from ur_ec_tou_mat, merge with aggregated_solar net_exports, and add credits to monthly_wide before fixed charges and min_charge. net_billing still falls back (unimplemented in CAIRO itself). Reduces ConEd run time from ~60-90 min to ~7 min per run. Tests: two new tests in test_patches.py verify the vectorized path matches CAIRO's output for both net_metering and style=None cases. Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Runs 13-16 (elasticity=-0.1, upgrade 0 and 2) were added to the ConEd batch but not registered in the hardcoded run-pair constants. This caused `ValueError: Invalid run pair 15+16` in build_master_bills and build_master_bat. Add 13/14 to UPGRADE_00_RUNS and 15/16 to UPGRADE_02_RUNS, and register (13,14) and (15,16) in VALID_RUN_PAIRS. Made-with: Cursor
Promote calibrated tariff outputs from the completed ny_20260325b_r1-16 ConEd batch (runs 1-20, including demand-flex runs 13-16). Updates all 10 ConEd delivery and supply calibrated tariff files. Made-with: Cursor
…YAML cenhud.yaml was updated with revised rate-case values; sync the hardcoded expected values in test_scalar_delivery_only and test_scalar_delivery_plus_supply. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
create_default_structure_tariffs.pyreads filed monthly rates YAML and builds URDB v7 tariff JSON preserving the actual utility rate structure (flat-with-monthly-variation, seasonal tiered, seasonal TOU) instead of collapsing to a single volumetric rate. Dispatches based on therate_structurefield in the YAML.base_tariff_patternJustfile variable (envBASE_TARIFF_PATTERN, default"flat") controls which base tariff runs 1–4 use and which calibrated tariff runs 5/6 derive seasonal discounts from. RI is configured withdefault.copy_flat_to_nonhp_flat.pywith--patternarg so it can mirror any_{pattern}.jsonto_nonhp_{pattern}.json.Reviewer focus
_extract_default_rate_from_tariff_configinutils/mid/compute_subclass_rr.py(lines 211–247). It hardcodes toperiod=1, tier=1when extracting the default rate from a calibrated tariff'sur_ec_tou_mat. With a multi-period default tariff, this picks one period's rate instead of a consumption-weighted annual average — ~12% too high for RIE supply, biasing the seasonal HP discount. This is a pre-existing design assumption that only becomes visible with multi-period tariffs and should be addressed as a follow-up._detect_periods_from_monthly_ratesmerges months with identical rates into contiguous periods. Non-contiguous months with the same rate become separate URDB periods (required by the schedule format). Verify this is acceptable for CAIRO.Closes #371
Closes #380