[Feature] Support Hailo for machine learning hardware acceleration

[arthurmcgregor]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

I run Immich on a Raspberry Pi 5 and would like to connect a Raspberry Pi AI Hat to improve machine learning performance. From what I understand, this would require supporting the Hailo modules for hardware acceleration. I suspect the Raspberry Pi 5 is a common entrypoint for people starting on their self-hosting journey (like myself!) so supporting the RPi AI hardware would be valuable.

The relevant hardware is currently:

• AI HAT with "13 and 26 tera-operations per second (TOPS) variants, built around the Hailo-8L and Hailo-8 neural network inference accelerators"
• AI Kit with "13 tera-operations per second (TOPS) neural network inference accelerator built around the Hailo-8L chip."

There was a related discussion here but that discussion is closed and the Hailo part was a side-discussion, so opening a new one to see if there has been any progress since then.

Thanks, in anticipation 💚

Platform

• Server
• Web
• Mobile

[bierwald]

I would appreciate implementing the Hailo AI HAT/Kit. This would enable a lot of small installations machine learning capabilities at a moderate invest.

Implementing this seems pretty obvious to me.

[mwesten]

According to the Product sheet it should be compatible with TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX.

[mertalev]

There were some compatibility issues when compiling our model catalog to Hailo format when I last played around with this, but that was almost a year ago. They might have better OOTB support for these models these days.

[zytegalaxy]

@mertalev I am new this world but I think I can figure out the acceleration part but I know very little about what models immich use and how this can be easily integrated to run with immich machine learning docker. It would be great if you offer some pointers/insights on the work you did with hailo last time and other suggestions.

I think hailo would give ML ops a significant boost compared to Mini PCs (at least 100x speed-up). This way maybe you can crack up indexing and use it to get as much context possible from each image, without worrying much about available resources. This would allow users to do semantic search.

This way you can also do things other products can't or won't because they have to pay for cloud compute resource but since immich users run on their own devices, they don't care about their electricity bill going up which is very insignificant financially for individual users.

[mbay0r]

Hailo 8 support would be nice, using a Hailo 8 pcie Card.