[ZH/ZHC/Z2M/WindFront] OTA refactor

[Nerivec]

This one is going to be a bit "involved", so, drafting some guidelines, and gathering some feedback beforehand...

Move OTA logic from ZHC to ZH

Most of the code should go in Device.ts.

Helpers can be added separately (request builders, validators & co).

Need to rework the update logic so it can be properly tested (tests were disabled a while back in ZHC, vitest crumbles due to the amount of faked setTimeout or something...). Maybe a stream-like logic would work better?

Move OTA logic from Z2M to ZH

Moving scheduling stuff in ZH would decrease code duplication in Z2M extension and ensure unified logic (+ built-in support for other projects using ZH). Also would add database support.

Support all known OTA file tags

Support spec tags, see 11.4.4 of ZCL spec.

[Manu-spe] Refactor Koenkk/zigbee-herdsman-converters#9984 - 0xf000 should parse as AES tag for Telink per source. No need for custom properties in OTA index.

snippet

/** see https://github.com/telink-semi/telink_zigbee_sdk/blob/d5bc2f7b0c1f8536fe21c8127ca680ea8214bc8e/tl_zigbee_sdk/zigbee/ota/ota.h#L38 */
const TELINK_AES_TAG_ID = 0xf000;

    if (tagID === TELINK_AES_TAG_ID) {
        // OTA_FLAG_IMAGE_ELEM_INFO1 (1-byte) + OTA_FLAG_IMAGE_ELEM_INFO2 (1-byte)
        const tagMeta = buffer.subarray(position + 6, position + 8);
        const data = buffer.subarray(position + 8, position + 8 + length);

        return [{tagID, length, tagMeta, data}, 8];
    }

Should also be able to remove suppressElementImageParseFailure property from ZHC. (e.g. LEDVANCE use is Telink AES, same as above).

Cleanup zigbee-OTA customParseLogic.

Firmware release notes in mqtt/frontend

Add releaseNotes property to OTA state payload (if available).

Source feedback in mqtt/frontend (local index vs repo index)

Add source property to OTA state payload.

Updating via custom firmware URL

Passing url as part of update payload would seem the easiest. Can branch out to "use that firmware instead of matching in index".

Can spec-match the image after downloading.

Problem of "no index":

• unable to do off-spec matching for manufacturers that make a mess of the OTA spec (e.g. Tuya, Gledopto, etc.).
  • potential for "wrong firmware flashed" increased substentially
• no SHA, no integrity checking possible, can only validate that the image is passing OTA spec format, but cannot ensure it wasn't tampered with
• no release notes (presumably, user is already aware in this scenario)

Must bypass ota: true used in ZHC definition.

Must allow bypassing fileVersion check (to ease cases like Lumi that does not respect fileVersion incrementing...).

Will need UI support in specific device page rather than OTA page (since OTA page is dependent on supports_ota which we want to bypass here).

Updating via custom index URL

Similar to above, but for index. Should help with projects building custom firmware. Provides a middleground between "have to be in zigbee-OTA repo" and "use custom firmware URL" when extra off-spec matching is desired, better safety, and only one URL to share for all devices supported by given project.

Improve downgrading support

Add new buttons tied to proper topics in WindFront.

Need proper feedback of "is upgrade" vs "is downgrade" when a "check" was done and the OTA state has a match.

Better error handling for update

Should clear OTA state as appropriate when errors indicate a problem like "no image available" (could have been detected by a previous, now stale, check). cf https://github.com/Koenkk/zigbee2mqtt/blob/2721b0ea78f4e86e6fe3b5f4831c5787a28585a8/lib/extension/otaUpdate.ts#L366-L371

Improve options

Support disable_automatic_update_check per-device. Due to how Zigbee OTA works, we cannot prevent devices from requesting OTA, but with this option we can always reply "not available".

Post-update changes

Currently only reconfigure is done. This causes problems with some devices, should just full interview.

Tooling

https://nerivec.github.io/zigbee-ota-file-editor/

---

CC: @romasku @andrei-lazarov @pvvx @slacky1965 @thk-socal @sjorge @burmistrzak

[Ltek]

very happy to see this happening, thank you. When I encountered needing to manually update (testing unreleased/custom FW from the manufacture) I ran into some challenges.

[Comment] recipient of a 'bad' firmware and there was no way to fix it other than hack the Z2M database to trick Z2M into thinking the firmware was reporting correctly, which it was not. Being able to 'force' a file to be flashed would have been really nice. Dangerous, sure - but would have saved me a ton of time and headaches. Glad you are putting this in!

[Comment & FR] what is displayed on the Z2M page is confusing as all get-out. See this example...
2095120 is the Firmware ID, cool... but there is no way to know that "2095120" is magically one of the listed firmwares... "3.1" or is it "2.9"? The screenshot below shows its BOTH yet the, what is it? Dont know because it doesn't currently do any of these things. Would love to see you fix this to have more clear info

[FR] Flash History ... log of what devices were flashed, with what file, and when. Logging exact file specs (name, CRC, etc), Date, FW version, etc... all in a nice little table so we can see it all by filtering on the device. An the ability to make / edit user notes on each so we have a record of comments we may need for later, like "this version reports value changes when changed by a 3rd party system - no other FW does, this one is custom for me" (real comment for the last FW I used, the developer made for me)

[andrei-lazarov]

Looks good! 🙂 Some thoughts:

• I don't mind losing the safeguards when updating with firmware URL

  • No matter how many checks you add, some users will go around them, so the wrong update goes through. This usually doesn't brick the device, so they could just update again with the correct image.

• We might still need the legacy custom index

  • If the user wants to deploy many custom devices, it will be annoying to go through each one with the new approach. Also you probably wouldn't update them at once

• Solve Lumi / Aqara lower fileVersion

  • given the example oldFwV: 2839, newFwV: 25 (both in index)
  • We must find an elegant way to solve this automatically.. After we force the update, it will just update again to the previous one
  • Same method should be usable for other devices, like ikea styrbar

• Downgrade also needs blocking further updates

  • The red update button bothers people after they downgraded, or when they don't want to update at all
  • So there must be a way to clear it and ignore/block updates

• Update selection

  • A selector might be nice.. if there are multiple versions available, pick one from a list instead of having only 2 options: up & down
  • This would solve the lower fileVersion and downgrade issues
  • Not sure how to implement. It would need a behind-the-scenes force-index that changes whenever you make a selection

• Auto-interview after update

  • I believe we already have auto-reconfiguration after updates
  • But we also need interview (before config)!
  • ZBMINIR2 received binds in update, but they don't work until interview
  • same for STYRBAR
  • custom firmware also needs this all the time

• Correlate device to index entry

  • It's a little difficult to find what updates will match a device
  • Mainly because you need debug logs to obtain imageType. Saying again, should be obtainable from the UI, available after user performs a check, with manufacturerCode
  • And the OTA file names are not giving enough information
  • Maybe add some words in the index, just some strings for information: device name, device type, description, anything

@Ltek

• I agree multiple versions are confusing, but that's just what the device reports..
• Some kind of history might be useful

[pvvx]

1. Each firmware file comes with a ".json" file. A comment can be added to the ".json" file.
2. Updates for Tuya devices cannot be identified or processed via standard Zigbee OTA. There are already many Tuya devices on the market for which standard Zigbee OTA renders the device completely inoperable. (The reason is that chips with small Flash are used, into which OTA loading is carried out using other algorithms, but the block loading process itself uses the standard Zigbee OTA function.)
3. A menu should be provided for the user to select a custom OTA file. The Zigbee OTA program should automatically rebuild the identifiers in the file provided by the user.

Displaying OTA identifiers for the current running device firmware is a secondary task and is required by developers. Users are only interested in the version number and revision of the software.

Par. 1 and 3 solve all user problems.
Displaying the version or other descriptions for a specific Zigbee OTA file is resolved in par. 1.
Par. 2 has no solutions. In this case, the user makes their own decision and uses par. 3.

---

Currently, Telink (chipset) device users are provided with a program to correct identifiers in the Zigbee OTA update file to be able to upgrade from any FW version to any other. The program also extracts OTA code from the Flash firmware binary file.
https://pvvx.github.io/ATC_MiThermometer/ZigbeeTlcIdxOTA.html

For example, in the file selection menu you can use the link insertion -
https://github.com/pvvx/ZigbeeTLc/raw/refs/heads/master/bin/1141-020a-01323001-Z03MMC.zigbee

[Nerivec]

After we force the update, it will just update again to the previous one

It won't for zigbee-OTA provided images, the previous image will have been moved out of index and into index1 when the new one is added to the repo.
Presumably, if you flash a custom image, you know you should not use regular update after that (be it upgrade or downgrade).

We've mentioned the firmware ID vs firmware version quite a few times already but here it is again:

• Firmware ID: purely cosmetic (and completely unreliable, many times manufacturers forget to update it...). This is a manufacturer-provided string that is optional. When provided, it usually matches what manufacturers will for e.g. display on their websites / in release notes. This is only available when a device is joined to a network (is a Zigbee attribute, not an OTA file metadata).
• Firmware version: this is the actual version used by OTA (hence the two columns, one for current, one for available), this is mandatory.

About a history: between resets/re-pairings and devices updated outside of Z2M, would be unreliable at best.
Device description can be used for notes as needed (e.g. "custom firmware from abcd").

About device metadata in index: this has to be maintained manually, with 800+ images, it's a massive task, and in any case, several images match too many devices, likely to bring more confusion. It would also grow the index quite a bit (already has a lot of metadata for off-spec matching...). I suggest custom firmware projects make use of the OTA header string (see 11.4.2 OTA Header Format of ZCL spec), this is already in zigbee-OTA index (auto-added for each submitted image), and often enough is a default/empty string.

About update selection: this is beyond the use of 99% of users, would be too confusing. Custom firmware URL will allow to do this for users that want to pick specific versions. The zigbee-OTA index and index1 have only one purpose each, upgrade, downgrade, no multi-image is possible (except for step-updates with min/max version matchers).

Also have to remember:

• frontend is using WebSocket for communication with Z2M.
• most users want a simple display, too much advanced information can have adverse effects (e.g. the mess made by the misunderstood firmware ID), we have to pick what to display carefully.

[thk-socal]

I would love the ability to mark a device as "DO NOT Check for Update" so that it is not reporting an update is available. The reason is that I have some Inovelli switches that will not take the latest update and I am going to need to use a custom wire harness to update them offline. Seventy of my switches are this way. I want to mark those switches as DO NOT Check so that I do not try to update them as they will not take the update.

Would also be happy with a "Keep Current Firmware" as an option if that wording is better. What I fear is that users will see the update and keep trying to update a device and get frustrated.

[Ltek]

1. Each firmware file comes with a ".json" file. A comment can be added to the ".json" file.

I'm not sure what you are referring to but the OTA file I received from the the manufacture did not include a JSON file. It was just 1 OTA file.

I would love the ability to mark a device as "DO NOT Check for Update" so that it is not reporting an update is available. The reason is that I have some Inovelli switches that will not take the latest update and I am going to need to use a custom wire harness to update them offline. Seventy of my switches are this way. I want to mark those switches as DO NOT Check so that I do not try to update them as they will not take the update.

ditto!

[burmistrzak]

Here are my two cents:

• We should definitely keep the ability to have a local index. Useful for tinkering and air-gapped systems.
• Having the option to directly provide an OTA image is great for advanced users and/or emergencies, but should come with an appropriate frontend warning.
• Release notes are a huge plus. Should be pushed to HA (i.e. release_summary) as well.
• Preventing updates per device is mostly cosmetic, but still a thoughtful addition.
• A full interview (and configuration) after update would fix a lot of "issues".

@Nerivec Regarding testing, how about sampling the mock OTA process? IIRC, the device can request a specific block of bytes from the coordinator during an update. That should, in theory, be something we can test against.

[pvvx]

About device metadata in index: this has to be maintained manually, with 800+ images, it's a massive task, and in any case, several images match too many devices, likely to bring more confusion.

Most of the 800 devices listed are outdated and have long been absent from the market.

Currently, there are over a hundred update files for alternative firmware. If we include transitional firmware versions, the number of alternative firmware files would exceed three hundred variants.
However, further development of alternative, custom (DIY) firmware is limited by the current ZigBee OTA processing policy in ZHA and Z2M.
Most questions from users of alternative firmware are related to updating firmware in ZHA and Z2M. For this reason, the number of published alternative firmware versions is limited. It's impossible to explain to every user how to update the firmware on ZHA and Z2M. The current descriptions of how to Zigbee OTA on Z2M and ZHA resources are unclear to many users. And not all users know how to write specialized scripts to enable Zigbee OTA.
Likewise, not everyone needs updates to every new version, and the parameters offered in Z2M don't describe what exactly will change when upgrading to the proposed version.
Furthermore, a FW version update may include changes to clusters and device features that are not accounted for in Z2M.

In three years of working with Zigbee devices, out of the 800 firmware versions described, only one update file was used. And that was a Tuya update, which worked for one device but was also displayed for other devices (which it didn't support). And this update made no difference.