HEIC Support

[CEbbinghaus]

Describe the problem

Cinny cannot render HEIC images sent from IPhone and some Android phones, Making it difficult to interact with people using heic to get better quality at a lower size. Asking them to convert to jpg first is also not an acceptable solution since it adds significant overhead.

Describe the solution you'd like

Cinny to add heic support, perhaps through a JS library that converts it to a PNG/JPG on the fly. Anything is better than a blank frame

Alternatives considered

Not be able to view the images

Additional context

Example of how HEIC gets rendered on desktop. Note this is on Windows 11 via webview2 and the heic addon installed.

[notramo]

I recommend to avoid HEIC support.
It's a proprietary codec.

Browsers don't support it, because Google and Mozilla would need to pay huge sums to the license holders.
It could be added with WASM decoding, but if it's allowed to creep into the Matrix ecosystem, Element would probably need to pay, as the number of Element users exceed the free threshold.
Apple have supported it, because it was better than the alternative codecs at the time. There was no JPEG-XL back then. However, they are gradually switching to JPEG-XL nowadays. I guess they will keep paying HEIC fees for backwards compatibility. But I don't think a proprietary platform should influence which standards to support in an open ecosystem like Matrix.

JPEG-XL is a much better alternative. It's similar in quality/compression (some aspects are superior), and it's also an open, royalty-free format.
https://en.wikipedia.org/wiki/JPEG_XL

It's already compatible with Apple devices, and gaining support in Chrome and Firefox.
This means that no code changes will be required in Cinny, because the browsers are going to support it by default, without any additional codec needed.

[u9000-Nine]

Would it make sense to prevent users from uploading HEIC files?

[notramo]

I don't think that HEIC would require different approach than other nonfree formats.

However, I like the idea of adding an optional conversion dialog that appears for nonfree formats, and maybe for free formats that are space-inefficient, e.g. PNG, TIFF.

Offer conversion:

• HEIC - nonfree
• PNG, GIF, TIFF - free, but inefficient; JXL has better lossless mode
• JPG - free, and quite efficient, but JXL can compress it ~15-20% further losslessly

Do not offer conversion:

• AVIF - free, and quite efficient
• WEBP - free, and quite efficient

However, it would require a significant size WASM blob, which should be lazy-loaded.

[CEbbinghaus]

This may be my ignorance, but last I remember HEIC is royalty free as long as the software isn't making money from its infusion. Which Is why windows doesn't support it by default but rather, only as a free addon.

This let's them skip paying royalties for windows. But still provide HEIC support.

So while element pro would need to pay royalties. Element clients published for free wouldn't afaict.

More than happy with automatic conversion to a well supported. Lossless format. JPEG-XL sounds like a great candidate. Just needs browser support. Are we able to polyfill browser support to allow us to switch early?

[u9000-Nine]

What about OpenCollective donations and Github sponsorships?