Check for invalid layers in XR_ENABLE_API_LAYERS environment variable

[Rectus]

I came across this issue on Reddit:
https://www.reddit.com/r/pchelp/comments/1l8xhu0/openxr_runtime_issues/

I'm suspecting it was caused by the XR_ENABLE_API_LAYERS environment variable being set to an invalid layer name, causing the loader to pass it on to the runtime. By adding the variable I could replicate the symptom of xrCreateInstance returning XR_ERROR_API_LAYER_NOT_PRESENT with OpenXR-Explorer.

This is a completely opaque error for most users, and will prevent loading of all OpenXR applications.

I'm suggesting adding checks for XR_ERROR_API_LAYER_NOT_PRESENT and XR_API_LAYER_PATH. Aside from the above issue, they both have a huge impact on layer behavior.

[fredemmott]

I'm not sure how this would fit into the UI which is very much centered around a list of configured layers, but generally supportive of checking for the environment variables.

[add call to xrCreateInstance, and] adding checks for XR_ERROR_API_LAYER_NOT_PRESENT

This is really a separate issue, and one I'm unlikely to address. The tool does not currently invoke any code from the loader/runtime/any layers. Doing this in a way that is resilient to misconfiguration would require subprocesses and IPC, making things a fair bit more complicated.

It's also crossing over into a scope that I feel might be best left to OpenXR Explorer.

[fredemmott]

I'm not sure how this would fit into the UI which is very much centered around a list of configured layers

Eh, this isn't really true, it could be added to the 'All layers' list without being tied to a specific layer. There's a pre-existing UI problem in that 'all layers' can't be re-entered after selecting a layer, but that's not a blocker.

[Rectus]

Whoops, I meant to type XR_ENABLE_API_LAYERS instead of XR_ERROR_API_LAYER_NOT_PRESENT, as in checking the global environment variables.

[fredemmott]

It's a little awkward that for explicit API layers, there's definitely a JSON path (which might be invalid, but there is one), and there might not be an API layer name (e.g. if the JSON path doesn't exist, or is invalid), whereas for implicit the opposite is also possible.

Probably going to add a key field to the APILayer struct, setting that to mJSONPath.string() for explicit, and to the name (with an empty json path) if there is an invalid name in the environment variable.

The mechanism of XR_ENABLE_API_LAYERS is a bit surprising to me; I would have expected instead to make the setup more similar to explicit layers, but make the enable_environment manifest value mandatory for explicit layers

[Rectus]

Yeah, the behavior seems extreme. I'm guessing it's designed as something you switch on locally when debugging, rather than a feature for end users.

[fredemmott]

I'm suspecting it was caused by the XR_ENABLE_API_LAYERS environment variable being set to an invalid layer name, causing the loader to pass it on to the runtime.

FWIW, I think I'm seeing something similar from the opposite end in #33 without users setting the environment variable: a vendor runtime appears to be calling setenv in the game/app process when xrEnumerateInstanceExtensions() is called. I was a little surprised that xrEnumerateInstanceExtensions() loads the runtime DLL but xrEnumerateApiLayerProperties() does not.

End result is that on this runtime, calling xrEnumerateInstanceExtensions() changes the result of xrEnumerateApiLayerProperties()

[Rectus]

Oh that's interesting. It seems that behavior would enable a runtime to force loading an API layer. From reading the spec, xrEnumerateInstanceExtensionProperties would load the DLLs of all found API layers as well for chaining the call, and a layer could also use setenv to force load itself.

[fredemmott]

Funnily enough when called by an app, neither of the xrEnumerate* functions actually make the enumerate call into the runtime or layers at all - it's all done by the loader, based on the JSON files/registry/environment variables.

This had led to quite a few 'only broken when called by API layers' bugs, as those do go through the normal openxr layered call chain, but normal app/game calls - including the majority of the conformance test suite - do not.

It seems that behavior would enable a runtime to force loading an API layer.

Yep, definitely. This is intentional, at least for enable_environment in implicit layers:

[Rectus]

Funnily enough when called by an app, neither of the xrEnumerate* functions actually make the enumerate call into the runtime or layers at all - it's all done by the loader, based on the JSON files/registry/environment variables.

That's gonna lead to some fun bugs if someone makes a custom loader that does chain the call, or the khronos one gets updated.