Troubles using photons from Home Assistant custom integration

[vendruscolo]

Hi Stephen, once again I want to first thank you for your effort with Photons. I've been using it for quite some time and it's worked flawlessly so far.

Some time ago I wrote a custom integration for LIFX Z strips, so that I could expose them to HA as "virtual" lights. The idea is to have certain zones of the strip to light up independently from others. The integration lives here https://github.com/vendruscolo/lifx_virtual_lights/blob/photons/light.py
Bear with me, I'm a Swift engineer so that Python is probably far from idiomatic.

It was working great with Python 3.6 and HA 2021.12. The other day I upgraded Python to 3.10 and HA to 2022.06 and it started complaining.

This is the error I get in the logs:

Task <Task pending name='Task-3573' coro=<LIFXVirtualLight.async_update() running at /home/homeassistant/.homeassistant/custom_components/lifx_virtual_lights/light.py:284> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.10/asyncio/tasks.py:475]> got Future <Future pending cb=[silent_reporter() at /srv/homeassistant/lib/python3.10/site-packages/photons_app/helpers.py:1349]> attached to a different loop

From what I understand (I might be wrong here), HA creates its asyncio run loop, and so does Photons. And then, when I run an async method from HA there's runloop mismatch.

I don't know if this is something that can be fixed with changes from Photons, though. Anyway, are you able to tell me more and provide some guidance on how to address this? I had an hard time even trying to understand the above error.

Thanks!

[delfick]

I want to first thank you for your effort with Photons

❤️

Bear with me, I'm a Swift engineer so that Python is probably far from idiomatic.

I'm sure it's a lot better than any swift I could write!

attached to a different loop

I also hate this error

HA creates its asyncio run loop, and so does Photons

Yeap! the library_setup function creates this object https://github.com/delfick/photons/blob/main/modules/photons_app/executor.py#L58 which is responsible for collecting configuration and setting up everything necessary to run photons and then cleanup afterwards. It's technically unnecessary but for simplicity the documentation pretends the only way you'd use photons is when photons is the main thing in control.

anyway, are you able to tell me more and provide some guidance on how to address this?

absolutely!

We stop using the collector and manually create the objects you're getting from it :)

from photons_transport.targets import LanTarget
from photons_messages import protocol_register

import asyncio


async def async_setup_platform(...):
    target = LanTarget.create({"protocol_register": protocol_register, "final_future": asyncio.Future()})
    sender = await target.make_sender()

    async_add_entities(....)

and you can use self._mac_address instead of creating a reference object. You get the same level of caching as long as you're not remaking the sender each time you send messages to a device.

Ideally you run await target.close_sender(sender) and future.cancel() on that final future when HASS shuts down. But that's mainly a cleanliness thing and avoids the possibility of some annoying python warnings when things shut down. So certainly not vital for what you're doing.

Also https://github.com/vendruscolo/lifx_virtual_lights/blob/photons/light.py#L268-L274 is completely unnecessary. You could be pessimistic and say that not reaching the device means the device changed IP address and do a await sender.forget(self._mac_address) so that next time it wants to send to that device it redoes discovery, but it's rare a router gives a different IP to the same MAC.

To demonstrate, change the mac address in the following and in a python environment that has photons installed run python toggle.py

toggle.py

from photons_transport.targets import LanTarget
from photons_messages import protocol_register

import asyncio


async def main():
    target = LanTarget.create(
        {"protocol_register": protocol_register, "final_future": asyncio.Future()}
    )

    sender = await target.make_sender()

    mac_address = "d073d567cc4f"

    # Annoyingly seems I have to import PowerToggle here because of a circular import sigh
    from photons_control.transform import PowerToggle

    await sender(PowerToggle(), mac_address)


asyncio.run(main())

[delfick]

and to clarify, the reference object is a way of translating some description of a device into MAC addresses (or serials as photons calls them) so if you already have serials, then you don't need to do anything else to turn them into serials.

[delfick]

Also, I recommend using SetZones

[vendruscolo]

Oh my god, I can use the lights as intended again. You're my hero, thanks! <3

I'll try to see if I can find the proper hook for closing the sender and canceling the future. Given how hass works (and how it works in my case) I don't think it really matters. It would only matter if people load and unload the integration often, otherwise it is intended to be always running. Still, I'll try doing the right thing.

Re: https://github.com/vendruscolo/lifx_virtual_lights/blob/photons/light.py#L268-L274 I always thought it was a hack/workaround. What happened is that often (especially after a full LAN-stack reboot due to power outage) HA would start before the light strips would connect to Wi-Fi, and that caused photons to never discover them. That did the trick (the light entities become available without having to restart HA after the light strips connected to Wi-Fi). However I noticed that it would cause a high memory usage (async_update is called every 5 seconds): sometimes I had to reboot HA anyway.
Do you think it's worth making sender forget the MAC address?

Lastly, is the SetZones a new message? I don't remember reading about it one year ago 🤔

Thanks again!

[delfick]

I can use the lights as intended again

Wooh!!

You're my hero

It would only matter if people load and unload the integration often,

exactly!

However I noticed that it would cause a high memory usage

hmmmm, weird. I'd imagine it'd maybe use some more memory till they're found and then that'd go away, but surely not a lot more.

Do you think it's worth making sender forget the MAC address?

The only time that is necessary is if the router gives a different IP to the device.

When photons sends a message to a device it figures out the IP of the device (either by lookup or if there's no ip to lookup by doing a discovery) and then throws bytes at that IP. The retry mechanism is based off getting back separate messages from the network and lining that up to sent bytes.

So when we haven't already discovered a device, it will retry discovery the next time we try send bytes to that device.

In the case of your async_update running every 5 seconds, then I'd change the discovery timeout to 5 seconds. You do this by passing in find_timeout (code) when you use the sender.

For example

sender(DeviceMessages.SetPower(level=0), "d073d5001337", find_timeout=5)

would give up trying to find the device after 5 seconds instead of the default. And every time it does async_update it'll redo discovery if it hasn't found the device yet

Looking at that again, I recommend using the gatherer

from photons_transport.targets import LanTarget
from photons_messages import protocol_register

import asyncio


async def main():
    target = LanTarget.create(
        {"protocol_register": protocol_register, "final_future": asyncio.Future()}
    )

    sender = await target.make_sender()

    mac_address = "d073d541125c"

    plans = sender.make_plans("zones")

    // I can assume this loop runs at most once because I know I'm only giving a single mac address and I only have one plan
    async for _, _, info in sender.gatherer.gather(plans, mac_address):
        if info is not sender.gatherer.Skip:
            zones = [z for _, z in sorted(info)]
            print(zones)


asyncio.run(main())

Lastly, is the SetZones a new message? I don't remember reading about it one year ago

It's a bit older than a year :) delfick/photons-core@71fadb4#diff-65b24694454ac40c75e74a25f614f74dba17665ca2fef792e298a107122f89a9R213

Thanks again!

You're welcome!

[delfick]

Also I recommend using an error catcher instead of a pokemon catch

def error_catcher(error):
    _LOGGER.error(f"Received error while updating color zones for {self._mac_address} ({error}). Possibly offline?")

async for _, _, info in sender.gatherer.gather(plans, self._mac_address, error_catcher=error_catcher):
   // Skip from a "zones" plan means we can talk to the device but it's not multizone
   if info is not sender.gatherer.Skip:
      self._available = True
      ...

[vendruscolo]

Hey Stephen, busy days but I was able to go through your feedback and make changes to my integration. If you're curious, it's all pushed to my repo. I like how the gatherer made things simpler. I think I can also ditch the sets and simply use max() to get the various HSBK values, having a single loop.

Things seem to be working fine. Had some hiccups with Wi-Fi recently, but I think it was caused by my access points. Some times lights would disconnect randomly (the official LIFX app showed the lights as unavailable as well), but a second firmware upgrade seems to have solved issues (Ubiquiti mentioned something about IoT/2.4GHz devices, so maybe that was it).

I have a question about the error catcher: your suggestion makes sense, however it's not clear to me whether I can make the sender forget about the device on error. I added that just to be safe, but when I switched to the error catcher I had to remove that (it's an async method). I don't know whether it is possible to have an async error catcher? (lazy question, I could just try). As you said, I can probably live without it…?

And finally one other question that bugged me for a while: if I animate the brightness change (pass a non-zero value for duration) when turning a virtual light off (which sends the SetZones message), some times the zones remain partially lit (as if it set the brightness to 1% instead of 0). I never understood what's causing it. Is it because I'm immediately grabbing zone info right after it, while the zone is transitioning to a different brightness?

Thanks again for your time!

[delfick]

I like how the gatherer made things simpler

definitely one of my favourite fever dreams hahah

but I think it was caused by my access points.

All network equipment is evil

but when I switched to the error catcher I had to remove that (it's an async method).

yeap, correct. So there are two ways to go about doing that.

The first approach, which I recommend is to look at self._available in the methods that use sender and forget the device there. Great opportunity for a decorator like https://gist.github.com/delfick/df1943aa0ce7e33e27c4de784474ec18

The second approach is better if you have the ability to make sure things get cleaned up and that's by using a TaskHolder like this https://gist.github.com/delfick/3d1fe6fac48b8eaa8a7ec1ef9b2af6b7

I can probably live without [forgetting devices]…?

extremely likely

Is it because I'm immediately grabbing zone info right after it, while the zone is transitioning to a different brightness?

The only thing that stops a transition is another transition.

[delfick]

also, I recommend setting up your editor to use https://black.readthedocs.io/en/stable/

[vendruscolo]

Wow, I noticed this issue is still open! Things have been working smoothly on my side, I guess we can finally close this one and do a GitHub sweeping of old issues 😄 But before doing that I have two quick questions for you, if you don't mind.

In the end, I decided to keep things simple as far as error recovery is concerned. When an error occurs, I flip a bool in my light entity so that this informs Home Assistant the light is not available (which should block some user interactions). Then the update process will continue on its schedule, and hopefully at some point things will resume working. When a successful update finally takes place, I flip that bool again and say the light is available for use.

This has served me well so far. I notice some times messages time out, but that only happens for a few update cycles (so 20 seconds at most?). I'm not sure if clearing the sender would make things work better. What do you think?

Second, which is something I noticed and fixed today. I'm using the zones plans as you suggested. The implementation is really simple, but I noticed sometimes the zone data is providing information about HSB using the 0.0–1.0 range (using floats) and some times is providing the same information using ints in the 0-65535 range. This threw my logic off, because I was expecting data in a format and when I did my color calculations Photons would then complain I was passing a huge value for brightness, out of the 0–65535 range (I was passing 43234528 or something like that).
I fixed that by checking if I'm getting an int or a float, but that feels unnecessary to me.