fix: Added tracking of started planned dispatches for intelligent tariffs to improve cost sensor accuracy (3.5 hours dev time)

BREAKING CHANGE: The rate adjustments for intelligent tariffs have moved from using completed dispatches to started dispatches. This means that cost sensors may be in accurate for a couple of days while data is gathered.

Author: BottlecapDave

_docs/entities/intelligent.md

@@ -30,6 +30,7 @@ This sensor is used to determine if you're currently in a planned dispatch perio
 |-----------|------|-------------|
 | `planned_dispatches` | `array` | An array of the dispatches that are currently planned by Octopus Energy. |
 | `completed_dispatches` | `array` | An array of the dispatches that have been completed by Octopus Energy. This will only store up to the last 3 days worth of completed dispatches. |
+| `started_dispatches` | `array` | An array of the dispatches that have been planned by Octopus Energy and upon API refresh are still planned when the current 30 minute period started and is not in a boosting state. A planned dispatch will be added one 30 minute period at a time. This will only store up to the last 3 days worth of started dispatches. This is used to determine historic off peak rates. For example if you have a planned dispatch of `2025-04-01T10:00:00`-`2025-04-01T11:00:00`, at `2025-04-01T10:01:00` if the planned dispatch is still available the period of `2025-04-01T10:00:00`-`2025-04-01T10:30:00` will be added. |
 | `provider` | `string` | The provider of the intelligent features |
 | `vehicle_battery_size_in_kwh` | `float` | The size of the target vehicle battery in kWh. |
 | `charge_point_power_in_kw` | `float` | The power of the charge point battery in kW. |
@@ -45,9 +46,16 @@ Each item in `planned_dispatch` or `completed_dispatches` have the following att
 | `start` | `datetime` | The start date/time of the dispatch |
 | `end` | `datetime` | The end date/time of the dispatch |
 | `charge_in_kwh` | `float` | The amount to be charged within the dispatch period. |
-| `source` | `string` | Determines what has caused the dispatch to be generated. Will be `smart-charge` or `bump-charge`. |
+| `source` | `string` | Determines what has caused the dispatch to be generated. Will be `smart-charge`, `bump-charge` or None. |
 | `location` | `string` | The location of the smart charge |
 
+Each item in `started_dispatch` have the following attributes
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `start` | `datetime` | The start date/time of the dispatch |
+| `end` | `datetime` | The end date/time of the dispatch |
+
 !!! info
 
     You can use the [data_last_retrieved sensor](./diagnostics.md#intelligent-dispatches-data-last-retrieved) to determine when the underlying data was last retrieved from the OE servers.

_docs/services.md

@@ -339,6 +339,10 @@ This service is only available for the following sensors
 
     Due to limitations with Home Assistant entities, this service will only refresh data for the associated statistic ids used for the recommended approach in the [energy dashboard](./setup/energy_dashboard.md#previous-day-consumption). This will not update the history of the entities themselves.
 
+!!! warn
+
+    If you are on intelligent, the cost data will not be correct for charges outside of the normal off peak times. This is because this data isn't available.
+
 ### octopus_energy.register_rate_weightings
 
 Allows you to configure weightings against rates at given times using factors external to the integration. These are applied when calculating [target rates](./setup/target_rate.md#external-rate-weightings) or [rolling target rates](./setup/rolling_target_rate.md#external-rate-weightings).

_docs/setup/account.md

@@ -53,25 +53,23 @@ By default, intelligent dispatches are retrieved [periodically](../faq.md#how-of
 
 ## Intelligent rates mode
 
-If you are on an intelligent tariff then it's possible for you to get cheaper rates outside of your normal off peak periods if Octopus Energy schedules the charges and your car accepts the charges. The rate information provides by Octopus Energy doesn't take these periods into account, so the integration has to use the pending/completed dispatch information to adjust the rates appropriately. Due to the quality of the available data, this _can_ be off sometimes. This feature allows how the rate information is adjusted in these scenarios.
+If you are on an intelligent tariff then it's possible for you to get cheaper rates outside of your normal off peak periods if Octopus Energy schedules the charges and your car accepts the charges. The rate information provides by Octopus Energy doesn't take these periods into account, so the integration has to use the planned/completed dispatch information to adjust the rates appropriately. Due to the quality of the available data, this _can_ be off sometimes. This feature allows how the rate information is adjusted in these scenarios.
 
-### Pending and completed dispatches will turn into off peak rates
+### Planned and started dispatches will turn into off peak rates
 
-This is the default behaviour. In this scenario, all pending dispatches will be assumed to be converted into successful off peak charges by the car and therefore all rates during these periods will be converted into the off peak rate. This will be indicated by the `is_intelligent_adjusted` property. This is useful when planning other devices to turn on in the future during these cheap periods (e.g. by using a [target rate sensor](./target_rate.md)). 
+This is the default behaviour. In this scenario, all planned dispatches will be assumed to be converted into successful off peak charges by the car and therefore all rates during these periods will be converted into the off peak rate. This will be indicated by the `is_intelligent_adjusted` property. This is useful when planning other devices to turn on in the future during these cheap periods (e.g. by using a [target rate sensor](./target_rate.md)). 
 
 !!! warning
 
-    One side effect of this is around cost sensors, where if a pending dispatch does not turn into a completed dispatch, the cost sensor can increase in value when the pending dispatch is removed.
+    One side effect of this is around cost sensors, where if a planned dispatch does not turn into a started dispatch, the cost sensor can increase in value when the planned dispatch is removed.
 
-### Only completed dispatches will turn into off peak rates
+### Only started dispatches will turn into off peak rates
 
-In this scenario only completed dispatches will be taken into account for adjustments meaning all rates during only completed dispatch periods will be converted into the off peak rate. This will be indicated by the `is_intelligent_adjusted` property. This means no future planning can be made to take advantage of these cheap periods by rates alone.
+In this scenario only started dispatches will be taken into account for adjustments meaning all rates during only started dispatch periods will be converted into the off peak rate. This will be indicated by the `is_intelligent_adjusted` property. This means no future planning can be made to take advantage of these cheap periods by rates alone.
 
 !!! warning
 
-    One side effect of this is around cost sensors, where when a completed dispatch arrives the cost sensor will decrease in value.
-
-    There have also been reports of some successful charges not resolving into completed dispatches.
+    One side effect of this is around cost sensors, where when a started dispatch arrives the cost sensor will decrease in value.
 
 ## Home Pro
 

custom_components/octopus_energy/__init__.py

@@ -53,7 +53,7 @@
   CONFIG_MAIN_HOME_PRO_API_KEY,
   CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES,
   CONFIG_MAIN_INTELLIGENT_RATE_MODE,
-  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES,
+  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES,
   CONFIG_MAIN_OLD_API_KEY,
   CONFIG_VERSION,
   DATA_DISCOVERY_MANAGER,
@@ -504,7 +504,7 @@ async def async_setup_dependencies(hass, config):
                                                         is_smart_meter,
                                                         is_export_meter,
                                                         planned_dispatches_supported,
-                                                        config[CONFIG_MAIN_INTELLIGENT_RATE_MODE] if CONFIG_MAIN_INTELLIGENT_RATE_MODE in config else CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES,
+                                                        config[CONFIG_MAIN_INTELLIGENT_RATE_MODE] if CONFIG_MAIN_INTELLIGENT_RATE_MODE in config else CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES,
                                                         tariff_override)
 
   mock_heat_pump = account_debug_override.mock_heat_pump if account_debug_override is not None else False

custom_components/octopus_energy/api_client/intelligent_dispatches.py

@@ -39,32 +39,63 @@ def from_dict(data):
       data["source"],
       data["location"],
     )
+  
+class SimpleIntelligentDispatchItem:
+  start: datetime
+  end: datetime
+
+  def __init__(
+    self,
+    start: datetime,
+    end: datetime
+  ):
+    self.start = start
+    self.end = end
+
+  def to_dict(self):
+    return {
+      "start": self.start,
+      "end": self.end,
+    }
+  
+  def from_dict(data):
+    return SimpleIntelligentDispatchItem(
+      parse_datetime(data["start"]),
+      parse_datetime(data["end"]),
+    )
 
 class IntelligentDispatches:
   current_state: str | None
   planned: list[IntelligentDispatchItem]
   completed: list[IntelligentDispatchItem]
 
+  # Bit of a smell this being on the API client object as it's never returned from the API
+  started: list[SimpleIntelligentDispatchItem]
+
   def __init__(
     self,
     current_state: str | None,
     planned: list[IntelligentDispatchItem],
-    completed: list[IntelligentDispatchItem]
+    completed: list[IntelligentDispatchItem],
+    started: list[IntelligentDispatchItem] = []
   ):
     self.current_state = current_state
     self.planned = planned
     self.completed = completed
+    self.started = started
 
   def to_dict(self):
     return {
       "current_state": self.current_state,
       "planned": list(map(lambda x: x.to_dict(), self.planned)),
       "completed": list(map(lambda x: x.to_dict(), self.completed)),
+      "started": list(map(lambda x: x.to_dict(), self.started)),
     }
 
   def from_dict(data):
     return IntelligentDispatches(
       data["current_state"],
       list(map(lambda x: IntelligentDispatchItem.from_dict(x), data["planned"])),
       list(map(lambda x: IntelligentDispatchItem.from_dict(x), data["completed"])),
+      list(map(lambda x: SimpleIntelligentDispatchItem.from_dict(x), data["started"] if "started" in data else [])),
     )

custom_components/octopus_energy/config_flow.py

@@ -27,8 +27,8 @@
   CONFIG_MAIN_HOME_PRO_API_KEY,
   CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES,
   CONFIG_MAIN_INTELLIGENT_RATE_MODE,
-  CONFIG_MAIN_INTELLIGENT_RATE_MODE_COMPLETED_DISPATCHES_ONLY,
-  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES,
+  CONFIG_MAIN_INTELLIGENT_RATE_MODE_STARTED_DISPATCHES_ONLY,
+  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES,
   CONFIG_ROLLING_TARGET_HOURS_LOOK_AHEAD,
   CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE,
   CONFIG_TARGET_TARGET_TIMES_EVALUATION_MODE_ALL_IN_FUTURE_OR_PAST,
@@ -840,8 +840,8 @@ async def __async_setup_main_schema__(self, config, errors):
           vol.Required(CONFIG_MAIN_INTELLIGENT_RATE_MODE): selector.SelectSelector(
           selector.SelectSelectorConfig(
               options=[
-                selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES, label="Pending and completed dispatches will turn into off peak rates"),
-                selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_COMPLETED_DISPATCHES_ONLY, label="Only completed dispatches will turn into off peak rates"),
+                selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES, label="Planned and started dispatches will turn into off peak rates"),
+                selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_STARTED_DISPATCHES_ONLY, label="Only stared dispatches will turn into off peak rates"),
               ],
               mode=selector.SelectSelectorMode.DROPDOWN,
           )
@@ -860,7 +860,7 @@ async def __async_setup_main_schema__(self, config, errors):
           CONFIG_MAIN_GAS_PRICE_CAP: config[CONFIG_MAIN_GAS_PRICE_CAP] if CONFIG_MAIN_GAS_PRICE_CAP in config else None,
           CONFIG_MAIN_FAVOUR_DIRECT_DEBIT_RATES: config[CONFIG_MAIN_FAVOUR_DIRECT_DEBIT_RATES] if CONFIG_MAIN_FAVOUR_DIRECT_DEBIT_RATES in config else True,
           CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES: config[CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES] if CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES in config else False,
-          CONFIG_MAIN_INTELLIGENT_RATE_MODE: config[CONFIG_MAIN_INTELLIGENT_RATE_MODE] if CONFIG_MAIN_INTELLIGENT_RATE_MODE in config else CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES,
+          CONFIG_MAIN_INTELLIGENT_RATE_MODE: config[CONFIG_MAIN_INTELLIGENT_RATE_MODE] if CONFIG_MAIN_INTELLIGENT_RATE_MODE in config else CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES,
           CONFIG_MAIN_AUTO_DISCOVER_COST_TRACKERS: config[CONFIG_MAIN_AUTO_DISCOVER_COST_TRACKERS] if CONFIG_MAIN_AUTO_DISCOVER_COST_TRACKERS in config else False,
         }
       ),

custom_components/octopus_energy/const.py

@@ -45,8 +45,8 @@
 CONFIG_MAIN_INTELLIGENT_MANUAL_DISPATCHES = "intelligent_manual_dispatches"
 CONFIG_MAIN_AUTO_DISCOVER_COST_TRACKERS = "auto_discover_cost_trackers"
 CONFIG_MAIN_INTELLIGENT_RATE_MODE = "intelligent_rate_mode"
-CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES = "pending_and_completed_dispatches"
-CONFIG_MAIN_INTELLIGENT_RATE_MODE_COMPLETED_DISPATCHES_ONLY = "completed_dispatches_only"
+CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES = "pending_and_started_dispatches"
+CONFIG_MAIN_INTELLIGENT_RATE_MODE_STARTED_DISPATCHES_ONLY = "started_dispatches_only"
 
 CONFIG_DEFAULT_LIVE_ELECTRICITY_CONSUMPTION_REFRESH_IN_MINUTES = 1
 CONFIG_DEFAULT_LIVE_GAS_CONSUMPTION_REFRESH_IN_MINUTES = 2
@@ -202,8 +202,8 @@
   vol.Required(CONFIG_MAIN_INTELLIGENT_RATE_MODE): selector.SelectSelector(
     selector.SelectSelectorConfig(
         options=[
-          selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES, label="Pending and completed dispatches will turn into off peak rates"),
-          selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_COMPLETED_DISPATCHES_ONLY, label="Only completed dispatches will turn into off peak rates"),
+          selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES, label="Planned and started dispatches will turn into off peak rates"),
+          selector.SelectOptionDict(value=CONFIG_MAIN_INTELLIGENT_RATE_MODE_STARTED_DISPATCHES_ONLY, label="Only started dispatches will turn into off peak rates"),
         ],
         mode=selector.SelectSelectorMode.DROPDOWN,
     )

custom_components/octopus_energy/coordinators/electricity_rates.py

@@ -9,7 +9,7 @@
 from homeassistant.helpers import issue_registry as ir
 
 from ..const import (
-  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES,
+  CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES,
   COORDINATOR_REFRESH_IN_SECONDS,
   DATA_ACCOUNT_COORDINATOR,
   DATA_INTELLIGENT_DEVICE,
@@ -67,7 +67,7 @@ async def async_refresh_electricity_rates_data(
     unique_rates_changed: Callable[[Tariff, int], Awaitable[None]] = None,
     raise_no_active_rate: Callable[[], Awaitable[None]] = None,
     remove_no_active_rate: Callable[[], Awaitable[None]] = None,
-    intelligent_rate_mode: str = CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_COMPLETED_DISPATCHES
+    intelligent_rate_mode: str = CONFIG_MAIN_INTELLIGENT_RATE_MODE_PENDING_AND_STARTED_DISPATCHES
   ) -> ElectricityRatesCoordinatorResult: 
   if (account_info is not None):
     period_from = as_utc((current - timedelta(days=1)).replace(hour=0, minute=0, second=0, microsecond=0))
@@ -122,7 +122,7 @@ async def async_refresh_electricity_rates_data(
         if dispatches_result is not None and dispatches_result.dispatches is not None and is_export_meter == False:
           new_rates = adjust_intelligent_rates(new_rates,
                                                dispatches_result.dispatches.planned if planned_dispatches_supported else [],
-                                               dispatches_result.dispatches.completed,
+                                               dispatches_result.dispatches.started,
                                                intelligent_rate_mode)
 
           _LOGGER.debug(f"Rates adjusted: {new_rates}; dispatches: {dispatches_result.dispatches}")
@@ -189,7 +189,7 @@ async def async_refresh_electricity_rates_data(
           dispatches_result.last_evaluated > existing_rates_result.rates_last_adjusted):
       new_rates = adjust_intelligent_rates(existing_rates_result.original_rates,
                                            dispatches_result.dispatches.planned,
-                                           dispatches_result.dispatches.completed,
+                                           dispatches_result.dispatches.started,
                                            intelligent_rate_mode)
 
       _LOGGER.debug(f"Rates adjusted: {new_rates}; dispatches: {dispatches_result.dispatches}")