feat: power input methods — docs, staleness, HW energy, manual IP, diagnostics (#71-#75)

AGENT_PROMPT.md

@@ -0,0 +1,105 @@
+# Plan 09 — Power Input Methods: Documentation & Feature Completeness
+
+## Your Role: Network & MQTT Specialist + State Machine Specialist
+
+You are implementing Plan 09: comprehensive documentation for all 5 power input
+methods, plus feature gap fixes (API staleness detection, HomeWizard P1 energy data,
+manual IP fallback, and diagnostic counters).
+
+## Plan File
+
+Read the full plan at: `/Users/basmeerman/Downloads/EVSE-team-planning/plan-09-power-input-methods.md`
+
+## Branch & Workflow
+
+- **Branch:** `work/plan-09`
+- **Remote:** Push to `myfork` (basmeerman/SmartEVSE-3.5), NEVER to `origin`
+- **Workflow:** Specification-first for code changes (SbE → test → implement → verify)
+- **Verify after each code increment:** `cd SmartEVSE-3/test/native && make clean test`
+- **Verify firmware builds:** `pio run -e release -d SmartEVSE-3/`
+- **Commit and push after each increment** — do not batch multiple increments into one commit
+
+## GitHub Issues (in order)
+
+1. **#71** — Power Input Methods Documentation (Increment 1) — START HERE
+2. **#72** — API/MQTT Staleness Detection (Increment 2)
+3. **#73** — HomeWizard P1 Energy Data (Increment 3)
+4. **#74** — HomeWizard P1 Manual IP Fallback (Increment 4)
+5. **#75** — Diagnostic Counters (Increment 5)
+
+## File Ownership — YOU OWN THESE
+
+### New files (create these):
+- `docs/power-input-methods.md` — Comprehensive power input guide (Increment 1)
+- `SmartEVSE-3/test/native/tests/test_metering_diagnostics.c` — Diagnostic counter tests (Increment 5)
+
+### Files you modify (YOUR sections only):
+- `SmartEVSE-3/src/mqtt_parser.c` / `mqtt_parser.h` — Add staleness timeout parsing,
+  HomeWizardIP command
+- `SmartEVSE-3/src/evse_ctx.h` — Add `api_mains_last_update_ms`, `api_mains_timeout_ms`,
+  diagnostic counter fields
+- `SmartEVSE-3/src/evse_state_machine.c` — Staleness check in tick function (CRITICAL —
+  must have tests)
+- `SmartEVSE-3/test/native/Makefile` — Add test_metering_diagnostics build rule
+- `README.md` — Add "Power Input Methods" to documentation table
+
+## SHARED FILES — COORDINATE CAREFULLY
+
+These files are also modified by Plan 06 and/or Plan 07:
+
+- **`network_common.cpp`**: You modify `getMainsFromHomeWizardP1()` to read energy
+  fields (Increment 3) and add HomeWizardIP manual fallback in mDNS discovery
+  (Increment 4). Keep your changes in clearly marked sections with comments like
+  `// BEGIN PLAN-09: HomeWizard energy data` and `// END PLAN-09`.
+
+## Architecture Constraints
+
+1. **CRITICAL: `evse_state_machine.c` changes MUST have tests** — this is safety-critical
+2. Staleness detection must use pure C, testable natively
+3. Use `snprintf` everywhere, never `sprintf`
+4. All test functions MUST have SbE annotations (@feature, @req, @scenario, etc.)
+5. Use `test_framework.h` (NOT unity.h)
+6. New `evse_ctx.h` fields must be documented with comments
+7. Never change existing MQTT topic names — only add new ones
+8. Never change existing `/settings` JSON field names — only add new fields
+9. Requirement prefix: `REQ-MTR-` for metering, `REQ-MQTT-` for MQTT
+
+## Increment 1 Detailed Steps (Start Here — Documentation Only)
+
+1. Read the plan's Section 1 (Power Input Method Analysis) thoroughly
+2. Read existing code to verify the data flow descriptions:
+   - `main.cpp`: `ModbusRequestLoop()` for Modbus RTU and Sensorbox
+   - `esp32.cpp`: `homewizard_loop()` / `getMainsFromHomeWizardP1()` for HomeWizard
+   - `network_common.cpp`: MQTT callback for API/MQTT feed
+3. Create `docs/power-input-methods.md` with:
+   - Reliability ranking table (most reliable first)
+   - Decision tree flowchart (text-based)
+   - Per-method setup guide (5 methods)
+   - Comparison table
+   - Troubleshooting section
+4. Update `README.md` documentation table with link
+5. Commit and push
+
+## Increment 2 Detailed Steps (API/MQTT Staleness — Most Important Code Change)
+
+1. Write SbE scenarios:
+   - Given API metering, when no update for 120s, then fall back to MaxMains
+   - Given API metering, when update received, then reset staleness timer
+   - Given API metering, when staleness detected, then set diagnostic flag
+   - Given non-API metering, then staleness check is skipped
+2. Add fields to `evse_ctx.h`: `api_mains_last_update_ms`, `api_mains_timeout_ms`
+3. Write tests in existing test file or new test file
+4. Implement staleness check in `evse_state_machine.c`
+5. Add MQTT parser for timeout setting
+6. Run `make clean test`
+7. Verify firmware builds
+
+## Key References
+
+- MQTT parser pattern: `src/mqtt_parser.c` (see existing `Set/MainsMeter` handler)
+- HomeWizard code: search for `HomeWizard` in `esp32.cpp` and `network_common.cpp`
+- Modbus metering: `main.cpp` `ModbusRequestLoop()`
+- State machine: `src/evse_state_machine.c`
+- Context struct: `src/evse_ctx.h`
+- Test framework: `test/native/include/test_framework.h`
+- Existing meter tests: `test/native/tests/test_meter_decode.c`

README.md

@@ -251,6 +251,7 @@ Connect to your WiFi network, then browse to `http://smartevse-xxxx.local/update
 | Document | Description |
 |----------|-------------|
 | [Hardware installation](docs/installation.md) | Wiring, mounting, contactor setup |
+| [Power Input Methods](docs/power-input-methods.md) | Metering options: Modbus, Sensorbox, HomeWizard, MQTT — reliability ranking, setup, troubleshooting |
 | [Configuration](docs/configuration.md) | LCD menu settings reference |
 | [Operation](docs/operation.md) | Day-to-day usage guide |
 | [Solar & Smart Mode Stability](docs/solar-smart-stability.md) | EMA smoothing, dead bands, phase switch timers, cycling prevention |

SmartEVSE-3/src/esp32.cpp

@@ -816,6 +816,21 @@ void mqtt_receive_callback(const String topic, const String payload) {
             break;
         // END PLAN-06
 
+        // BEGIN PLAN-09: API staleness timeout
+        case MQTT_CMD_MAINS_METER_TIMEOUT:
+            // Applied via bridge layer to evse_ctx.api_mains_timeout
+            break;
+        // END PLAN-09
+
+        // BEGIN PLAN-09: HomeWizard manual IP fallback
+        case MQTT_CMD_HOMEWIZARD_IP:
+            homeWizardManualIP = cmd.homewizard_ip;
+            // Clear cached mDNS result so next poll uses the new IP
+            homeWizardHost = "";
+            _LOG_A("HomeWizard manual IP set to '%s'\n", cmd.homewizard_ip);
+            break;
+        // END PLAN-09
+
         default:
             return;
     }
@@ -1020,6 +1035,12 @@ void SetupMQTTClient() {
     MQTTclient.announce("LoadBl", "sensor", optional_payload);
     MQTTclient.announce("PairingPin", "sensor", optional_payload);
     MQTTclient.announce("Firmware Version", "sensor", optional_payload);
+    // BEGIN PLAN-09: Metering diagnostic counters
+    optional_payload = MQTTclient.jsna("entity_category","diagnostic") + MQTTclient.jsna("state_class","total_increasing") + MQTTclient.jsna("entity_registry_enabled_default","False");
+    MQTTclient.announce("MeterTimeoutCount", "sensor", optional_payload);
+    MQTTclient.announce("MeterRecoveryCount", "sensor", optional_payload);
+    MQTTclient.announce("ApiStaleCount", "sensor", optional_payload);
+    // END PLAN-09
 
 #if MODEM
         optional_payload = MQTTclient.jsna("unit_of_measurement","%") + MQTTclient.jsna("value_template", R"({{ (value | int / 1024 * 100) | round(0) }})");
@@ -1266,6 +1287,11 @@ void mqttPublishData() {
         // Diagnostic: free heap and MQTT message counter
         mqtt_pub_int(MQTT_SLOT_FREE_HEAP, "/FreeHeap", (int32_t)ESP.getFreeHeap(), false, now_s);
         mqtt_pub_int(MQTT_SLOT_MQTT_MSG_COUNT, "/MQTTMsgCount", (int32_t)MQTTMsgCount, false, now_s);
+        // BEGIN PLAN-09: Metering diagnostic counters
+        mqtt_pub_int(MQTT_SLOT_METER_TIMEOUT_COUNT, "/MeterTimeoutCount", (int32_t)g_evse_ctx.meter_timeout_count, false, now_s);
+        mqtt_pub_int(MQTT_SLOT_METER_RECOVERY_COUNT, "/MeterRecoveryCount", (int32_t)g_evse_ctx.meter_recovery_count, false, now_s);
+        mqtt_pub_int(MQTT_SLOT_API_STALE_COUNT, "/ApiStaleCount", (int32_t)g_evse_ctx.api_stale_count, false, now_s);
+        // END PLAN-09
 
         // MQTT config settings — publish for HA switch/number entity state
         MQTTclient.publish(MQTTprefix + "/MQTTChangeOnly", MQTTChangeOnly ? "1" : "0", true, 0);
@@ -2668,16 +2694,22 @@ bool fwNeedsUpdate(char * version) {
     _LOG_A("homewizard_loop(): start HomeWizrd P1 reading.");
     lastCheck_homewizard = currentTime;
 
-    const auto currents = getMainsFromHomeWizardP1();
+    const auto result = getMainsFromHomeWizardP1();
 #if SMARTEVSE_VERSION < 40 //v3
-    for (int i = 0; i < currents.first; i++)
-        MainsMeter.Irms[i] = currents.second[i];
-    if (currents.first) {
+    for (int i = 0; i < result.phases; i++)
+        MainsMeter.Irms[i] = result.currents[i];
+    if (result.phases) {
+        // BEGIN PLAN-09: HomeWizard energy data
+        if (result.import_energy_wh > 0)
+            MainsMeter.Import_active_energy = result.import_energy_wh;
+        if (result.export_energy_wh > 0)
+            MainsMeter.Export_active_energy = result.export_energy_wh;
+        // END PLAN-09
         CalcIsum();
         MainsMeter.setTimeout(COMM_TIMEOUT);
     }
 #else
-    Serial1.printf("@Irms:%03u,%d,%d,%d\n", MainsMeter.Address, currents.second[0], currents.second[1], currents.second[2]); //Irms:011,312,123,124 means: the meter on address 11(dec) has Irms[0] 312 dA, Irms[1] of 123 dA, Irms[2] of 124 dA
+    Serial1.printf("@Irms:%03u,%d,%d,%d\n", MainsMeter.Address, result.currents[0], result.currents[1], result.currents[2]); //Irms:011,312,123,124 means: the meter on address 11(dec) has Irms[0] 312 dA, Irms[1] of 123 dA, Irms[2] of 124 dA
 #endif
 }
 

SmartEVSE-3/src/evse_ctx.h

@@ -185,6 +185,12 @@ typedef enum {
 #ifndef SMART_DEADBAND_DEFAULT
 #define SMART_DEADBAND_DEFAULT  10   /* 1.0A in deciamps */
 #endif
+#ifndef API_MAINS_STALENESS_DEFAULT
+#define API_MAINS_STALENESS_DEFAULT 120  /* Default staleness timeout in seconds */
+#endif
+#ifndef EM_API_METER
+#define EM_API_METER  9   /* MainsMeterType value for API/MQTT feed (matches EM_API in meter.h) */
+#endif
 #ifndef RAMP_RATE_DIVISOR_DEFAULT
 #define RAMP_RATE_DIVISOR_DEFAULT  4 /* Symmetric /4 for both up and down */
 #endif
@@ -350,6 +356,16 @@ typedef struct {
     uint8_t  MainsMeterTimeout;
     uint8_t  EVMeterTimeout;
 
+    // --- API mains staleness detection ---
+    uint16_t api_mains_staleness_timer; /* Countdown in seconds, reset on API data arrival */
+    uint16_t api_mains_timeout;        /* Configurable staleness timeout: 0=disabled, default 120s */
+    bool     api_mains_stale;          /* true when API mains data is stale */
+
+    // --- Metering diagnostic counters ---
+    uint32_t meter_timeout_count;      /* Number of CT_NOCOMM timeout events */
+    uint32_t meter_recovery_count;     /* Number of CT_NOCOMM recovery events */
+    uint32_t api_stale_count;          /* Number of API staleness detection events */
+
     // --- Error handling ---
     uint8_t ErrorFlags;
     uint8_t ChargeDelay;

SmartEVSE-3/src/evse_state_machine.c

@@ -130,6 +130,16 @@ void evse_init(evse_ctx_t *ctx, evse_hal_t *hal) {
     ctx->MainsMeterTimeout = COMM_TIMEOUT;
     ctx->EVMeterTimeout = COMM_EVTIMEOUT;
 
+    // API mains staleness detection
+    ctx->api_mains_staleness_timer = API_MAINS_STALENESS_DEFAULT;
+    ctx->api_mains_timeout = API_MAINS_STALENESS_DEFAULT;
+    ctx->api_mains_stale = false;
+
+    // Metering diagnostic counters
+    ctx->meter_timeout_count = 0;
+    ctx->meter_recovery_count = 0;
+    ctx->api_stale_count = 0;
+
     // Error handling
     ctx->ErrorFlags = NO_ERROR;
     ctx->ChargeDelay = 0;
@@ -1568,15 +1578,21 @@ void evse_tick_1s(evse_ctx_t *ctx) {
     // MainsMeter timeout (lines 1732-1755)
     if (ctx->MainsMeterType && ctx->LoadBl < 2) {
         if (ctx->MainsMeterTimeout == 0 && !(ctx->ErrorFlags & CT_NOCOMM) && ctx->Mode != MODE_NORMAL) {
-            evse_set_error_flags(ctx, CT_NOCOMM);
-            evse_set_power_unavailable(ctx);
+            // For API metering with staleness enabled, suppress CT_NOCOMM —
+            // the staleness mechanism handles the timeout with graceful fallback.
+            if (!(ctx->MainsMeterType == EM_API_METER && ctx->api_mains_timeout > 0)) {
+                evse_set_error_flags(ctx, CT_NOCOMM);
+                evse_set_power_unavailable(ctx);
+                ctx->meter_timeout_count++;
+            }
         } else if (ctx->MainsMeterTimeout > 0) {
             ctx->MainsMeterTimeout--;
         }
     } else if (ctx->LoadBl > 1) {
         if (ctx->MainsMeterTimeout == 0 && !(ctx->ErrorFlags & CT_NOCOMM)) {
             evse_set_error_flags(ctx, CT_NOCOMM);
             evse_set_power_unavailable(ctx);
+            ctx->meter_timeout_count++;
         } else if (ctx->MainsMeterTimeout > 0) {
             ctx->MainsMeterTimeout--;
         }
@@ -1599,13 +1615,39 @@ void evse_tick_1s(evse_ctx_t *ctx) {
     // CT_NOCOMM recovery (line 1769)
     if ((ctx->ErrorFlags & CT_NOCOMM) && ctx->MainsMeterTimeout > 0) {
         evse_clear_error_flags(ctx, CT_NOCOMM);
+        ctx->meter_recovery_count++;
     }
 
     // EV_NOCOMM recovery (line 1771)
     if ((ctx->ErrorFlags & EV_NOCOMM) && ctx->EVMeterTimeout > 0) {
         evse_clear_error_flags(ctx, EV_NOCOMM);
     }
 
+    // API mains staleness detection
+    // For API metering (EM_API_METER) with staleness enabled, count down a
+    // separate timer. On expiry, fall back to MaxMains (safe limit that still
+    // allows charging) instead of CT_NOCOMM (which stops charging entirely).
+    if (ctx->MainsMeterType == EM_API_METER && ctx->api_mains_timeout > 0 && ctx->LoadBl < 2) {
+        // Recovery: clear stale flag when timer is reset by incoming data
+        if (ctx->api_mains_stale && ctx->api_mains_staleness_timer > 0) {
+            ctx->api_mains_stale = false;
+        }
+        // Countdown
+        if (ctx->api_mains_staleness_timer > 0) {
+            ctx->api_mains_staleness_timer--;
+        }
+        // On expiry: set stale flag and fall back to MaxMains (once)
+        if (ctx->api_mains_staleness_timer == 0 && !ctx->api_mains_stale) {
+            ctx->api_mains_stale = true;
+            ctx->api_stale_count++;
+            for (int i = 0; i < 3; i++) {
+                ctx->MainsMeterIrms[i] = (int16_t)(ctx->MaxMains * 10);
+            }
+        }
+    } else if (ctx->MainsMeterType != EM_API_METER) {
+        ctx->api_mains_stale = false;
+    }
+
     // Temperature check (lines 1773-1778)
     if (ctx->TempEVSE > ctx->maxTemp && !(ctx->ErrorFlags & TEMP_HIGH)) {
         evse_set_error_flags(ctx, TEMP_HIGH);

SmartEVSE-3/src/mqtt_parser.c

@@ -148,6 +148,29 @@ bool mqtt_parse_command(const char *prefix, const char *topic,
                                    &out->ev_meter.L3, &out->ev_meter.W, &out->ev_meter.Wh);
     }
 
+    /* API mains meter staleness timeout: 0=disabled, 10-3600 seconds */
+    if (match_topic(prefix, topic, "/Set/MainsMeterTimeout")) {
+        out->cmd = MQTT_CMD_MAINS_METER_TIMEOUT;
+        int val = atoi(payload);
+        if (val == 0 || (val >= 10 && val <= 3600)) {
+            out->mains_meter_timeout = (uint16_t)val;
+            return true;
+        }
+        return false;
+    }
+
+    /* HomeWizard P1 manual IP: skip mDNS, connect directly. Empty = use mDNS */
+    if (match_topic(prefix, topic, "/Set/HomeWizardIP")) {
+        out->cmd = MQTT_CMD_HOMEWIZARD_IP;
+        size_t len = strlen(payload);
+        if (len >= sizeof(out->homewizard_ip))
+            return false;
+        /* Allow empty string to clear manual IP (re-enable mDNS) */
+        strncpy(out->homewizard_ip, payload, sizeof(out->homewizard_ip) - 1);
+        out->homewizard_ip[sizeof(out->homewizard_ip) - 1] = '\0';
+        return true;
+    }
+
     /* Home battery current: positive = charging, negative = discharging */
     if (match_topic(prefix, topic, "/Set/HomeBatteryCurrent")) {
         out->cmd = MQTT_CMD_HOME_BATTERY_CURRENT;

SmartEVSE-3/src/mqtt_parser.h

@@ -30,6 +30,8 @@ typedef enum {
     MQTT_CMD_MQTT_CHANGE_ONLY,
     MQTT_CMD_SOLAR_DEBUG,
     MQTT_CMD_DIAG_PROFILE,
+    MQTT_CMD_MAINS_METER_TIMEOUT,
+    MQTT_CMD_HOMEWIZARD_IP,
 } mqtt_cmd_type_t;
 
 // Mode values matching firmware MODE_NORMAL/MODE_SOLAR/MODE_SMART
@@ -68,6 +70,8 @@ typedef struct {
         bool mqtt_change_only;                  // MQTT_CMD_MQTT_CHANGE_ONLY (0/1)
         bool solar_debug;                       // MQTT_CMD_SOLAR_DEBUG (0/1)
         uint8_t diag_profile;                   // MQTT_CMD_DIAG_PROFILE (0-5)
+        uint16_t mains_meter_timeout;           // MQTT_CMD_MAINS_METER_TIMEOUT (0, 10-3600)
+        char homewizard_ip[16];                 // MQTT_CMD_HOMEWIZARD_IP (IPv4 or empty)
     };
 } mqtt_command_t;
 

SmartEVSE-3/src/mqtt_publish.h

@@ -9,7 +9,7 @@
 extern "C" {
 #endif
 
-#define MQTT_CACHE_MAX_SLOTS 84  /* 74 current topics + headroom */
+#define MQTT_CACHE_MAX_SLOTS 88  /* 81 current topics + headroom */
 
 /* One slot per published MQTT topic, in mqttPublishData() call order */
 typedef enum {
@@ -91,6 +91,9 @@ typedef enum {
     MQTT_SLOT_OCPP_TX_ACTIVE,
     MQTT_SLOT_OCPP_CURRENT_LIMIT,
     MQTT_SLOT_OCPP_SMART_CHARGING,
+    MQTT_SLOT_METER_TIMEOUT_COUNT,
+    MQTT_SLOT_METER_RECOVERY_COUNT,
+    MQTT_SLOT_API_STALE_COUNT,
     MQTT_SLOT_COUNT  /* must be <= MQTT_CACHE_MAX_SLOTS */
 } mqtt_slot_t;