Add additional image related types to the library

[ripytide]

I've been thinking about the rust image ecosystem a bit recently, and how this library works great for sharing pixel types between libraries. But, that's not the only information that often needs to be passed. Additional information could include:

• Pixel types (implemented)
• Colorspace Information types (not implemented)
• AlphaType types (not implemented)
• Image types (not implemented)
• Other Image Metadata (see Creating a new PixelImage type. image-rs/image#2262)

I think it still fits with the original purpose of this library, (as a common denominator compatibility crate), to also provide types for these other image-related types.

I've been working on what I think this might look like over at https://github.com/ripytide/pixmeta.

This would solve issues like the one you most recently raised: #140

What do you think @kornelski?

As an addendum I've also got a proof-of-concept of what a colorspace conversion library would look like which would be use the colorspaces provided to implement conversions from any pixel with any colorspace to any other pixel with any other colorspace. It's over at https://github.com/ripytide/pixconv, since I know it's another tricky piece of the rust image type puzzle, and having a working copy could help influence decisions made lower down like in this crate.

Another possible downstream crate we might want to think about implementing too so we know our library is properly structured is an ICC serializer/deserializer which would depend on the colorspace types.

[kornelski]

I think representing whole images is definitely out of scope of this crate. It's a massive problem in itself, way bigger than this crate.

There isn't a single thing that's an "image". It's a whole spectrum between needing to represent any image file with 100% feature richness (with any color space, HDR, gamut, metadata) all down to imgref and v_frame that may be used in performance-critical low-level code.

I've worked on an image resizer tool, and struggled to come up with enough synonyms for "image" to handle all the variations. There's a kind of image you need to round-trip a file, which means multiple animation frames (with their own timing and metadata), ICC files, metadata chunks in multiple formats (including file-format-specific ones), thumbnails, possibly even file permissions, xattrs, and original file size (to save only if smaller). But that notion of an image is too complex and bloated for APIs that want to modify pixels, so there's lighter pixels-with-colorspace-info kind of image, but you can't reasonably slice and subslice that, so there's also a lower-level notion of a bitmap where you've hopefully sorted out your working colorspace already. Then that bitmap may be in owned or borrowed flavor, and those may be either a generic T: Pixel containers, or a dynamic enum of pixel layouts. Also that enum always has too few or too many variants (you may want only opaque ones, or only 8bit, or only rgb without yuv, or only interleaved not planar, and so on).

Even at the very bottom there are lots of ways of representing pixels in memory. It creeps into layouts with stride (pixel-aligned and unaligned, even non-byte-aligned!), block padding (accessible or inaccessible), planar formats (with subsampling, and blocks and padding), less-than-8-bit pixels, tiled layouts (which they'll want to iterate in parallel without unsafe or UB), and uninitialized regions.

You know well it's already tricky to make a one-size-fits all RGBA pixel. It gets exponentially worse for bigger containers!

So I want to draw a line at representing a single pixel here. A Rust-native slice of pixels is the max this crate should have opinions about, and bigger storage and whole-image metadata should be left to other crates.

[kornelski]

For conversions, the problem is bigger than 1 pixel, since you want it SIMD-ified. I've tried to make it a 1-pixel problem with the old yuv crate design, but I had to abandon that approach. It's just non-competitive, and also incomplete without handling chroma upsampling at the same time.

In one project, I've once mixed-up cr with cb channels, so I thought I'd be clever and use Cb<u8, Rec709> for the channels. Ending up with Yuv<Luma<u8, Rec709>, Cb<u8, Rec709>, Cr<u8, Rec709>> created so much type-system boilerplate that in retrospect just dealing with the bugs would have been less effort.

In the ecosystem there is a need for enums for YUV-related metadata that comes from MPEG specs, since everyone has their own enum like https://docs.rs/ravif/latest/ravif/enum.MatrixCoefficients.html

[kornelski]

What do you mean for alpha types? Premultiplied?

What other colorspaces? I'm afraid that HDR is around the corner, and it's a hairy problem that ties pixels to metadata and weird hacks like hdr maps.

[ripytide]

I agree it's out of scope of this library, but not out of scope of the overarching goals. I was going to suggest creating some sort of cargo workspace so each was a separate crate but managed in a single repo.

I get that image layouts come in infinite flavors all with their own perfectly logical applications. But that by itself does not preclude the value that can be had for having a common denominator rust image type that crates can pass over crate boundaries. In fact this type already exists in the rust ecosystem, kind of, in the image crate ImageBuffer. But there are lots of issues with that type that I don't believe the image crate is in a position to fix, mainly due to backwards compatibility. Hence, I think it's worth the attempt at making the new 13th competing standard.

By all means, the image type would often suck for any performance-related application, and using an application-specific format will likely always be a faster option, but that's not possible over community crate boundaries since every downstream user could have a different application endpoint.

Regarding 1-pixel conversions not being able to take advantage of SIMD, that's why in the proof of concept there are two trait conversion functions, one for pixel-to-pixel and one for image-to-image:

pub trait HasColorSpace {
    type ColorSpace;
}

pub trait PixelConvert<P>: HasColorSpace + Sized
where
    P: HasColorSpace,
{
    #[must_use]
    fn pixel_convert(
        pixel: P,
        source_colorspace: P::ColorSpace,
        destination_colorspace: Self::ColorSpace,
    ) -> Self;

    #[must_use]
    fn pixel_convert_image(
        image: Pixels<P>,
        source_colorspace: P::ColorSpace,
        destination_colorspace: Self::ColorSpace,
    ) -> Pixels<Self>;
}

So that each could be implemented in different ways, such as using SIMD for image-to-image conversions.

Yeah I agree there is a limit on how much type boilerplate is feasible, so far, my best experiences have come from separating concerns from using the pixel types for component layout and size Rgb vs Bgra. And a separate external colorspace type for everything else, as can be seen in the PixelConvert trait methods. I feel this strikes a reasonable balance that can still be used in all use-cases from run-time generic to compile-time specific.

In the ecosystem there is a need for enums for YUV-related metadata that comes from MPEG specs, since everyone has their own enum like https://docs.rs/ravif/latest/ravif/enum.MatrixCoefficients.html

Yep this would be the primary goal of the YuvColorspace type which would be a component of the larger Colorspace type.

What do you mean for alpha types? Premultiplied?

Yes, essentially this:

pub struct RgbColorSpace {
    pub primaries: Primaries,
    pub transfer: Transfer,
    pub whitepoint: Whitepoint,
    pub luminance: Luminance,
}
pub struct RgbaColorSpace {
    pub primaries: Primaries,
    pub transfer: Transfer,
    pub whitepoint: Whitepoint,
    pub luminance: Luminance,
    pub alpha_type: AlphaType
}
enum AlphaType {
    Straight,
    Premultiplied,
}

What other colorspaces? I'm afraid that HDR is around the corner, and it's a hairy problem that ties pixels to metadata and weird hacks like hdr maps.

All of them ideally, I was mainly using Wikipedia's list of colorspaces page as guidance. Any HDR10 pixel types would have to be done in u16's though as non-byte aligned pixels are out of scope, which is also the case with this library.

The goal of the colorspace types library would not to be to get everything perfect right from the get-go but to provide a central location where it could be worked on incrementally with each contributor addressing their own needs and the benefits shared. The best kind of open-source imo.

[kornelski]

There's discussion in the image crate about a similar thing: image-rs/image#2478

[kornelski]

Adding definitions for Primaries, Whitepoint, etc. could be okay, since it's hopefully relatively simple. It's just a bit odd that Rgb pixel doesn't use them, so they're just taken for a ride in this crate.

[awxkee]

pub struct RgbColorSpace {
pub primaries: Primaries,
pub transfer: Transfer,
pub whitepoint: Whitepoint,
pub luminance: Luminance,
}
pub struct RgbaColorSpace {
pub primaries: Primaries,
pub transfer: Transfer,
pub whitepoint: Whitepoint,
pub luminance: Luminance,
pub alpha_type: AlphaType
}
enum AlphaType {
Straight,
Premultiplied,
}

This assumes that every RGB color model is a color space based on XYZ, which is not actually true.
There are many vendor shipped RGB color models that are LUT based and do not have direct links to XYZ, or at least vendor didn't defined them or they are using exotic transfer functions. Such as Arri Wide Gamut: Arri Log C3/C4, Sony S-Log, S-Gamut3.Cine, V-Log and others. Also there are some RGB color models as Kirk Yrg which based on LMS connecting on CIE LMS 2006 instead of CIE XYZ 1931.

[ripytide]

There's discussion in the image crate about a similar thing: image-rs/image#2478

Unless they are creating a new individual crate for the sole purpose of an image type, I don't think it will be good enough. They are still using their own pixel types and traits which are much inferior to the types and traits in this library in my biased opinion.

Adding definitions for Primaries, Whitepoint, etc. could be okay, since it's hopefully relatively simple. It's just a bit odd that Rgb pixel doesn't use them, so they're just taken for a ride in this crate.

In a separate crate though, for separation of concerns in case a user only wants the colorspace types and not the pixel types or vice versa.

@awxkee I'm know next to nothing about color-spaces unfortunately, since it seems there is not currently a single resource online I can find that contains all the necessary information about each and every colorspace and variation and also how to convert between them all. This is sort of the problem I am suggesting we fix. Once we have all the information about all the colorspaces we can then go about creating rust algebraic data-types that best fit all those color-spaces.

I am aware there are probably a million bad assumptions and bugs in the current prototype. It's the concept that I'm worried about.

If there are RGB color spaces not based on XYZ is it still possible to convert between them? Or are there some colorspaces so completely disjoint that conversion is not possible?

[awxkee]

I'm know next to nothing about color-spaces unfortunately, since it seems there is not currently a single resource online I can find that contains all the necessary information about each and every colorspace and variation and also how to convert between them all. This is sort of the problem I am suggesting we fix. Once we have all the information about all the colorspaces we can then go about creating rust algebraic data-types that best fit all those color-spaces.

There is no single list with colorspaces. Speaking shortly you need to implement almost full lcms2 for that, where your color model looks something like this one.

There are common points where RGB derived from:
1 - CIE XYZ 1931 based.
2 - Vendor shipped some "custom" ICC profile. Some software as libjxl ships tonemapping profiles this way. This is in fact also CIE XYZ 1931, but you need to make it there first.
3 - Vendor shipped CIE XYZ 1931 based color space with some exotic transfer function as Arri Log C4. This is common thing on professional video.
4 - Vendor shipped .cube LUT files which is very common for color grading, professional cameras etc.
5 - Some custom based RGB -> RGB connection based on HALD images. This is uncommon way to ship RGB based color space though.
6 - RGB that actually represents Yrg which in practice RGB color model, but not CIE XYZ 1931 based RGB color space. And to convert between them you need to convert at first CIE XYZ 1931 into CIE XYZ 2006.

Or are there some colorspaces so completely disjoint that conversion is not possible?

Vendor may ship .cube LUT or ICC profile with LUT Device -> sRGB Rec.709 or sRGB Rec.709 -> Device only, this completely disjoints colorspaces after conversion.

If there are RGB color spaces not based on XYZ is it still possible to convert between them?

Yes, this is possible either by interpolating a LUT directly or by using a LUT with a connection space.

Direct LUT interpolation is common in color grading and camera workflows because vendors ship LUTs that are RGB-based. To perform a conversion, you simply interpolate a 3D LUT. To convert back, you need a second LUT provided by the vendor that performs the inverse transformation. LUT with a connection space is common workflow for ICC profiles.

[ripytide]

Interesting, I took a brief look at your color library https://github.com/awxkee/moxcms and it looks like you implement your own pixel types, like Rgb here. What the reasons are that you have chosen not to use this library for those pixel types? It sounds like your opinions could have a lot of beneficial value to improving this library such that you could choose to use it. Apologies for going a bit of topic.

Regarding color-space conversions, it sounds like LUTs are like custom colorspaces made for a specific device, and if the manufacturer gives the necessary LUTs for forwards and reverse transformations then color conversion is possible. So a LUT is sort of a colorspace in itself?
It looks like you are already working on a color management system in rust, so we may be able to collaborate on this as a project, if you're interested.

My assumption going in was to make a big enum of all the colorspaces I could think of and their parameters. Like Rgb, Yuv, Xyz, Lab, etc.. and then start with a forwards/backwards conversion from all these basic colorspaces into a central colorspace like Xyz 1931 and then that would allow any colorspace to be converted into any other colorspace via Xyz. Then as time went on we could manually implement common colorspace conversions directly skipping Xyz like Rgb <-> Yuv etc.. to increase performance. Though it seems intractable to do this for every possible colorspace combination due to the exponential growth.

If there exist colorspaces that can't be converted to Xyz even via any other colorspaces then that could be tricky but I'm dubious such a colorspace exists as it would seem very user-unfriendly. My understanding was that Xyz 1931 contained all perceptible human colors.

[awxkee]

Interesting, I took a brief look at your color library https://github.com/awxkee/moxcms and it looks like you implement your own pixel types, like Rgb here. What the reasons are that you have chosen not to use this library for those pixel types? It sounds like your opinions could have a lot of beneficial value to improving this library such that you could choose to use it. Apologies for going a bit of topic.

My motivation is:

1. I mostly work directly with raw pixels, and I use Rgb types primarily for point-wise math operations like sqrt, cbrt, euclidean_distance, pow, which are often missing. I have no motivation to implement a large number of traits for external types just to do math, or to bind those external types to my own types like Luv/Lab, etc.
2. A large part of the library uses SIMD, and implementing such wide range of pixel types like Rgb,Bgr,Bgra etc is still require big amount of efforts. Binding this for external types won't make it easier. Likely binding to external types will make it harder.
3. Dependency management: after adding external dependency, the problems of external library become your problems. I believe the best way to solve any problems is not to add them explicitely, especially if they could be easily avoided.

Regarding color-space conversions, it sounds like LUTs are like custom colorspaces made for a specific device, and if the manufacturer gives the necessary LUTs for forwards and reverse transformations then color conversion is possible. So a LUT is sort of a colorspace in itself?

Cube LUT is just array of numbers that defines only one conversion path from one color space to another one. For example Sony S-Log to sRGB Rec.709. Thus, such LUT contains numbers that already in sRGB Rec.709.

If there exist colorspaces that can't be converted to Xyz even via any other colorspaces then that could be tricky but I'm dubious such a colorspace exists as it would seem very user-unfriendly. My understanding was that Xyz 1931 contained all perceptible human colors.

XYZ 1931 is bigger than human vision. But you need to know how to map your colors from your color space to it.

My assumption going in was to make a big enum of all the colorspaces I could think of and their parameters. Like Rgb, Yuv, Xyz, Lab, etc.. and then start with a forwards/backwards conversion from all these basic colorspaces into a central colorspace like Xyz 1931 and then that would allow any colorspace to be converted into any other colorspace via Xyz. Then as time went on we could manually implement common colorspace conversions directly skipping Xyz like Rgb <-> Yuv etc.. to increase performance. Though it seems intractable to do this for every possible colorspace combination due to the exponential growth.

You need to implement only for YUV <-> RGB a hundred of manual conversions. You could easily check this looks like.

But the main point is valid. This is how a CMS works.

When a LUT already contains an RGB-to-RGB conversion, things are much simpler because you don't need to involve XYZ.

Software that performs such arbitrary operations usually doesn't exist, because it's not particularly useful.
The YUV color model is very different from others and comes with its own issues ( subsampling, upsampling, plenty of models, different bit depths ) which are totally unrelated to other color models.
In practice, if you're doing color grading in UCS, JzAzBz, or similar models, you typically aren't working with YUV.
OTOH, if you're working with YUV, you're probably not doing color grading at the same point.
As well if you're drawing color picker with Okhsl you likely don't bother about JzAzBz, UCS and others.

It looks like you are already working on a color management system in rust, so we may be able to collaborate on this as a project, if you're interested.

I can give some advices, but I don't actually want to reimplement everything once again :)