How does ImageSpec handle images with non-byte-sized channels

[MarkCallow]

How does ImageSpec handle images with non-byte-sized channels, e.g, R5G6B5 or R9G9B9E5 format? format and channelFormats use TypeDesc which doesn't seem to have any types less than 1-byte, unless I've missed something. Also there are only channel_bytes functions in ImageSpec.

[lgritz]

That's right. ImageSpec, and all in-memory operations in OIIO, make several simplifying assumptions, including that pixel data types are exactly 1, 2, 4, or 8 bytes per channel (and are either int, uint, or float, of those width).

Remember that the core goal of OIIO is to allow apps to read and write images in a fairly generic way without regard to the disk file formats, and without apps needing a tangle of spaghetti code to handle every possible data representation that any format might possibly support, and especially for apps to need to change every time we add support for a new file format (which may come with a unique data representation).

So we would consider something like R5G6B5 to be like a compression or encoding method -- something specific to the way it's stored in the file. The file format reader/writer is expected to treat it as if it was 3 channels of uint8, and handle the uint8[3] <-> R5G6B5 internally and hide it from the application that's using the OIIO API. The format reader may also have a custom metadata field that reveals what bit depth or encoding is used in the file itself, and as well, the writer may honor that metadata to request that the file be written as R5G6B5 or whatever. (I think our BMP reader, and maybe others, are already examples of handling encodings in this manner, and I think also TIFF doesn't combine channels but it does allow channels to have 1, 2, 4, 10, 12, 14 bit representations, which it reports as either 8 or 16 and alters the values accordingly as they go in and out of the OIIO APIs.)

That said, the one place this approach (which was OIIO's original design over 13 years ago) falls down is if you WANT to directly get the compressed data -- for example, to directly transfer it to a GPU that supports on-card compressed texture formats. This is something we'll have to grapple with at some point, I suppose. We've kicked that can down the road for a long time because the core user base of OIIO is film VFX, and they don't use any of these lossy or low bit depth formats.

[MarkCallow]

This seems reasonable for most use cases. However I am working on using OIIO as the way to read images for my KTX v2 creator toktx. The input images will either be encoded to some block-compressed format or else they will be written to the output file as is (if they match a Vulkan VkFormat). In the former case exactly byte-size data is not a problem. In the latter I'll want to get directly to the image data whatever it is. One of the supported OIIO inputs I am hoping to take advantage of is dds to make a dds2ktx tool. In this case I will putting the untouched image data into a new container.

I had thought that passing TypeDesc::UNKNOWN to read_image, et al, gave you the native image but it sounds like it must get you the image as described by the ImageSpec. Perhaps the solution is, e.g., TypeDesc::NATIVE' which DOES tell read_image`, et al, to retrieve unmodified image data.

EDIT: I also need information about the actual format so the app can choose the proper VkFormat. How widespread are the custom metadata fields you mention that provide this information? Could they be made mandatory when channel widths are not a whole number of bytes? Does each format reader/writer use a different name for the metadata? If so it is inconvenient. Also I need more than channel widths. I need to know other things such as whether a shared exponent is being used, e.g. R9G9B9E5.