KHR_gaussian_splatting_wide_gamut_color

[weegeekps]

As mentioned in the base KHR_gaussian_splatting extension pull request, this extension adds support for several wide-gamut colors and their associated encoding methods to KHR_gaussian_splatting.

Some initial open questions:

• We still need to decide on how we want to handle the questions about which function to use at runtime for blending/composition with PQ and HLG.
• What additional language around tone mapping do we need?

[NorbertNopper-Huawei]

In general, I like splitting up the extension. However, to avoid future fragmentation and/or duplication, I recommend to define it more generic, as defining a color encoding is required not just for gaussian splatting.

[lexaknyazev]

WCG does not imply HDR. Could we be more specific about the intended use cases here?

[lexaknyazev]

Saying "wide color gamut" (here and throughout) would be more standard than "wide-gamut color", which is often used as an adjective.

[ZehuiLin-Huawei]

To include HDR and WCG in this extension, naming as color_space should be better.

[lexaknyazev]

glTF strings are case-sensitive. Could we align lower/upper case usage here? Not a very strong opinion but I'd prefer making these enums all-lowercase (or at least capitalizing Linear).

[NorbertNopper-Huawei]

As this has been discussed.
There are folks, who want to use RAW image data.
E.g. this paper is describing it:
https://aryan-garg.github.io/hdrsplat/

[ZehuiLin-Huawei]

To include HDR and WCG in this extension, naming as color_space should be better.

[ZehuiLin-Huawei]

It would be clearer if we explicitly state what color primiaries and transfer characteristics are being used for each color space. For example, describing BT.2020-PQ as "BT.2020 color primaires and SMPTE ST2084 Perceptual Quantizer (PQ) transfer function".

PQ is defined for HDR, while BT.2020 defines SDR and WCG. BT.2100 extends BT.2020 to HDR, but using the same color primaires.

[ZehuiLin-Huawei]

It would be helpful to add informative language somewhere explaining that the splats are intended for alpha blending in the specified linear or non-linear color space. This behavior differs from traditional meshes where people should first convert colors to linear color space before performing any other color operations.

[ZehuiLin-Huawei]

While gaussian splatting extension is in primitives level, I feel the colorSpace property should be put in meshes level, as multiple gaussian splatting "primitives" in a "mesh" probably share the same color space.

[NorbertNopper-Huawei]

It has to be on inside the primitive, as e.g. another primitive in the same mesh is e.g. common, triangle based glTF asset.

[ZehuiLin-Huawei]

Okay. If we are going to add color space for other kinds of primitives, then in meshes would be better.

[doug-walker]

Hi, I work for Autodesk and am serving as the chief architect on the OpenColorIO project, which is a color management system that is used in applications from Autodesk, Adobe, Foundry, Unreal, SideFX, V-Ray, and many more. We have a collaboration going with senior representatives from the OpenUSD and MaterialX projects to establish good color interop across the Academy Software Foundation (ASWF) projects and OpenUSD. I was wondering if we could interest you in adopting our color space naming?

We have published a recommendation for scene-referred color spaces used in computer graphics. And we have another recommendation for display-referred color spaces that is in progress here. We are providing open source OpenColorIO implementations of all of the color spaces to try and make them very clearly defined.

The key item is a compact name that we are now calling the "color interop ID", and that is the string I would propose you adopt. This has recently been added to OpenEXR as a standard attribute and there is work being done to implement that in OpenImageIO, Blender, and other software.

The splats schema that is in progress for OpenUSD will use the OpenUSD ColorSpaceAPI, which utilizes the interop IDs and these color space definitions.

Sharing common naming and precise color space definitions would help promote good interop between all of these projects.

[NorbertNopper-Huawei]

Hi, I work for Autodesk and am serving as the chief architect on the OpenColorIO project, which is a color management system that is used in applications from Autodesk, Adobe, Foundry, Unreal, SideFX, V-Ray, and many more. We have a collaboration going with senior representatives from the OpenUSD and MaterialX projects to establish good color interop across the Academy Software Foundation (ASWF) projects and OpenUSD. I was wondering if we could interest you in adopting our color space naming?

We have published a recommendation for scene-referred color spaces used in computer graphics. And we have another recommendation for display-referred color spaces that is in progress here. We are providing open source OpenColorIO implementations of all of the color spaces to try and make them very clearly defined.

The key item is a compact name that we are now calling the "color interop ID", and that is the string I would propose you adopt. This has recently been added to OpenEXR as a standard attribute and there is work being done to implement that in OpenImageIO, Blender, and other software.

The splats schema that is in progress for OpenUSD will use the OpenUSD ColorSpaceAPI, which utilizes the interop IDs and these color space definitions.

Sharing common naming and precise color space definitions would help promote good interop between all of these projects.

I can bring this up today in a meeting we have, especially for Linear Rec.709 (sRGB) and sRGB Encoded Rec.709 (sRGB) as it does also have an effect on the base extension: #2490

For curiosity, there are these image file formats:
https://en.wikipedia.org/wiki/AVIF and https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format
How would the compact name for e.g. Rec.2020 and PQ encoded look like? pq_rec2020_scene?

[NorbertNopper-Huawei]

@doug-walker We consider to use the values from here also in the base extension:
https://github.com/AcademySoftwareFoundation/ColorInterop/blob/main/Recommendations/01_TextureAssetColorSpaces/TextureAssetColorSpaces.md#summary-table--overview-of-the-recommendations
Reviewed in a smaller group on Tuesday and then presented to 3D Formats on Wednesday.

[doug-walker]

@NorbertNopper-Huawei, thank you for considering our proposal!

How would the compact name for e.g. Rec.2020 and PQ encoded look like? pq_rec2020_scene?

Yes, it could look like that, if it were included in one of the Color Interop Forum recommendations. Otherwise, it would need a namespace prefix to indicate it was an external definition. But I would not recommend adding a scene-referred version of PQ. As described in ISO 22028-5, it is, in theory, possible to use a scene-referred version of PQ (or HLG). However, in practice, the display-referred version is far more commonly used and a renderer would not want to apply additional tone-mapping to such a signal.

When defining the Color Interop Forum color spaces we included the ISO 22028-1 image state (i.e., scene-referred vs. display-referred) as part of the color space encoding definitions. This is an important detail for the renderer to know: If it's display-referred it has already been tone-mapped, whereas if it is scene-referred the tone-mapping is yet to be done. This is obviously something that will very significantly change the look of the final result, so it's important to communicate the intent of the file author.

It seems like most of the color spaces in this PR and the base schema are display-referred (tone-mapping has already been done), so the mapping to the Color Interop Forum ID strings would be as follows:

BT.709-sRGB: srgb_rec709_display
BT.2100-PQ: pq_rec2020_display
BT.2100-HLG: hlg_rec2020_display
Display-P3: srgb_p3d65_display (or srgbe_p3d65_display for the HDR version)

There is currently no CIF ID for "BT.2020-ITU" because Rec.2020 video never became a popular format.

For the two linear options, if the intent is that they are scene-referred, the CIF IDs would be:
BT.709-linear: lin_rec709_scene
BT.2020-linear: lin_rec2020_scene

If it would be helpful for me to attend one of your meetings to answer questions, please let me know.

[NorbertNopper-Huawei]

Okay, thanks I got it.

[NorbertNopper-Huawei]

I do not know, if finally the naming will be changed in the glTF extensions. At minimum, there should be the mapping possible. As soon as we proceed on this extension, it would make sense that you join the discussions.

[NorbertNopper-Huawei]

One problem I do see is, that the list is not yet complete and all the _display ones are not listed yet.

[NorbertNopper-Huawei]

If it would be helpful for me to attend one of your meetings to answer questions, please let me know.

@weegeekps Could you please invite him for Tuesday, mainly to clarify the values for the colorSpace #2490 (comment)

From my side, I was not considering the image states.

[doug-walker]

As discussed in the meeting today, I followed up on my action-item and added a display-referred linear Rec.709 color space encoding to the draft ASWF "Color Space Encodings for Displays" recommendation. The interop ID for this is "lin_rec709_display". The initial feedback on the ASWF Slack has been positive, so I think we should be able to include that.

[NorbertNopper-Huawei]

From a Vulkan perspective, it does make sense to match most of the given color spaces:
https://docs.vulkan.org/refpages/latest/refpages/source/VkColorSpaceKHR.html