Add the VK_EXT_present_timing extension #1364
Conversation
|
|
|
Hi I have created an WIP/RFC proposal for a Wayland protocol which is modeled after this extension (here). It aims to provide enhanced presentation timing requests and events for Wayland, and to provide support for this VK extension in particular. Given the early stages of the public discussion both at the Wayland protocol and the Vulkan extension level, (possibly significant) changes should be expected. Thanks! |
| [open,refpage='VkRelativePresentTimeEXT',desc='Specifying the minimum duration an image should be presented',type='structs'] | ||
| -- | ||
|
|
||
| The sname:VkRelativePresentTimeEXT structure is defined as: |
There was a problem hiding this comment.
Maybe it would be a good idea to state explicitly the motivation for a separate relative timing target.
Applications submitting a single frame at a time wouldn't have any use for this struct I think (apart from maybe a slightly simpler interface), because they could create a suitable absolute target instead (last actualPresentTime + minPresentDuration).
However, applications queueing multiple frames at once don't know in advance when the presentation engine will choose to display their image. VkRelativePresentTimeEXT allows the application to specify a target relative to the previous actual present time, without having to manually adjust the absolute timing (n+1) depending on feedback n.
There was a problem hiding this comment.
This would be great for stuff like presenting at a stable 30fps on a 60hz display.
There was a problem hiding this comment.
I think spec language should refrain from making recommendations or suggestions about usage and intent. I know this PR has quite a few of these, which I might clean up, but I'd rather not add more here. The appendix, which could probably be fleshed out a bit, is a better place for these remarks. Alternatively, one of the large existing .NOTE sections may also be appropriate; I'll look into adding a comment to that effect somewhere.
| visual anomalies. | ||
| Adjustments to a larger IPD because of late images should happen quickly, | ||
| but adjustments to a smaller IPD should only happen if the | ||
| pname:optimalPresentTime member of the |
There was a problem hiding this comment.
What does same mean here? Exactly the same value, or +/- some delta?
|
|
||
| The sname:VkPastPresentationTimingEXT structure is defined as: | ||
|
|
||
| include::{generated}/api/structs/VkPastPresentationTimingEXT.txt[] |
There was a problem hiding this comment.
I'm missing the "headroom" field from GOOGLE_display_timing. That feedback is useful for low-latency situations where you'd be able to time the main loop such that there is little time before GPU is done rendering and display unit scans out.
I suppose calibrated timestamps might work around that niche, but that would rely on swapchain and GPU working on the same timebase, and it wouldn't be able to take into account any compositing work happening outside the knowledge of Vulkan.
cubanismo
force-pushed
the
VK_EXT_present_timing
branch
from
a72d9af to
6fcaf1b
Compare
| If the presentation engine's pname:refreshDuration is a fixed value, | ||
| the application's image present duration (IPD) must be a multiple of | ||
| pname:refreshDuration. |
There was a problem hiding this comment.
This language may be valid for other presentation modes, but is not valid for immediate present mode.
| Image Present Duration:: | ||
| The amount of time the application intends for each | ||
| newly-presented image to be visible to the user. | ||
| This value should: be a multiple of the refresh cycle duration. |
There was a problem hiding this comment.
This language may be valid for other presentation modes, but not for immediate present mode.
appendices/glossary.txt
Outdated
| Image Present Rate:: | ||
| The number of newly-presented images the application intends to present | ||
| each second (a.k.a. frame rate). | ||
| This value should: be a multiple of the refresh rate. |
There was a problem hiding this comment.
I think this language is not what is intended.
|
How this extension interacts with immediate present mode? Can we also make it possible to query vertical blanking duration, which is a fraction of refresh cycle duration? |
| 2) Do we return min/max Values for Refresh Duration for VRR? | ||
|
|
||
| *PROPOSED*: return only the minimum value of refreshDuration for a VRR. | ||
|
|
There was a problem hiding this comment.
Please reconsider this. I would like to ask for providing (at least as an optionally reported value) both minimum and maximum value for refreshDuration for VRR. One could always have special values like 0 to signal that specific info is not available/not yet available/not applicable to a given display setup, e.g., if a driver vendor doesn't want to implement additional info fields in initial implementations of the extension. Ideally it would be great if even more detailed info would be provided, e.g.,
-
minimum duration (== maximum refresh rate), corresponding to what the display can do at current video mode/display link bandwidth/other limitations.
-
maximum duration (== minimum refresh rate) of what the display can physically do without tricks like low framerate compensation (lfc). Typical values for current G-Sync or FreeSync/DP-adaptive sync displays are around 30 Hz - 48 Hz as far as i know.
-
maximum duration (== minimum refresh rate) of what can be done on the display + gpu combo + driver via tricks like low framerate compensation, panel self-refresh etc., e.g., the solutions implemented by NVidia and AMD atm. for going below ~ 30 Hz refresh rate / above present intervals of 33 msecs.
-
Ideally for VRR modes we'd even have some reporting of the "good quality range" in which duration can be varied quickly without causing strong perceptible visual artifacts like significant flicker of the image.
In my experience, some (many? most?) VRR panels do show a perceptible dimming of the image if the time-gap between successive presents is substantially increased beyond minimum duration, e.g., if min duration would be 6.944 msecs (on a max 144 Hz display), and max duration supported by the display itself would be 33.333 msecs (on a min 30 Hz display), then switching between presents with a spacing of 7 msecs and 17 msecs and back to 7 msecs may work without any user-perceptible flicker, but making bigger jumps from 7 msecs to 33 msecs and back will cause very perceptible and irritating flicker.
Similar, going beyond 33.333 msecs interval between presents would work as gpu's or panels would trigger an automatic refresh cycle (once maximum front-porch duration of the vblank is exceeded), but then block the app from presenting for a period of time (ie. at least one active scanout duration). Similar exceeding the physical maximum duration will trigger low framerate compensation which changes the timing characteristics wrt. flicker and blocking the application from presenting.
I am the core developer and maintainer of the popular open-source toolkit Psychtoolbox-3 which is used for basic research in neuroscience, vision-science, cognitive science, psychology and related basic medical research applications related to visual perception, eye care etc. The toolkit is also used for basic perception research in industry for future VR/AR applications and display technologies. Cfe. this VESA press release for how the software will be used for research into new HDR displays, display stream compression methods, VR and AR.. Without going into too many details here, be assured that precise visual presentation timing and precise/trustworthy visual presentation timestamping matters critically for many research applications. VRR is one of the most useful tools developed in the last decade for these paradigms, to allow fine-grained timing and framerate control. But we need presentation to be high quality, e.g., minimizing flicker when transitioning between different presentation intervals between presents. So having as much info about the timing properties as possible is very valuable to achieve that goal.
Since January 2020 Psychtoolbox implements a special VRR based presentation mode for research applications, which tries to make as good use as possible of AMD FreeSync with Displayport adaptive sync etc. to give fine-grained presentation timing control. This is currently done under Linux with classic native X11/GLX + OpenGL on top of AMD's open-source display drivers. A demo and test of VRR scheduling is part of Psychtoolbox under this link VRRTest.m.
I contributed some bug fixes and enhancements to AMD's Linux kernel driver to make this more robust and trustworthy for apps like mine. It works reasonably well, given the constraint of doing all scheduling of bufferswaps/flips in the user-space application, but one annoyance of the current approach is that my algorithm needs to know both minimum and maximum physical refresh duration of the display at a minimum. Ideally also the other properties mentioned above. Lacking api support mean users have to figure out minimum refresh rates themselves, e.g., by finding their displays specs online or in printed manuals, or by manually decoding EDID data or accessing root-only-readable privileged files, deep into the Linux debug filesystem, just to tell the application what these values are. Other useful properties like the ones mentioned above need to be guessed via heuristics. That's a quite tedious and error prone user experience that could be easily avoided in future presentation timing extensions like this one, by providing rather more details about display system timing characteristics rather than less.
Since recently we also support HDR display for research, using Vulkan's HDR extensions. However, reasonable timing can only be achieved under Vulkan atm. via rather gross hacks - and often not at all. Therefore my hope is that this extension can contribute to great progress for applications like mine wrt. research use of VRR + HDR, as well as for VR/AR.
Some of the issues requiring more detailed info about presentation engine properties could be solved in the Vulkan drivers or display drivers themselves, and i intend to contribute to the Linux FOSS drivers related to this extension where it is helpful, once the extension is finalized. But assuming not all problems can be solved on all target platforms perfectly, it will be neccessary for applications like mine to adapt or compensate for shortcomings or limitations of drivers, so more information and control is always good for use cases like research.
| * pname:pNext is `NULL` or a pointer to an extension-specific structure. | ||
| * pname:timeDomain is a elink:VkTimeDomainEXT value representing the time | ||
| domain that should be used with the swapchain. | ||
|
|
There was a problem hiding this comment.
One thing that i would like to see - not sure if this would be the right place, but given it is related to presentation timing - would be a way for the application to optionally request or forbid the use of VRR mode, similar to the fullscreen_exclusive extension, where there could be three settings "Enable VRR", "Disable VRR" and "Let driver decide between VRR or FRR". Again, research applications would appreciate control about VRR vs. FRR on VRR capable systems, as VRR is very well suited and beneficial for some research paradigms, but detrimental to others. My toolkit currently implements such a switch on Linux+X11+Mesa via some hacks to manipulate X-Atoms "behind the back" of the Mesa graphics library, based on detailed knowledge about how this works on XOrg's X11 + Mesa DRI3/Present implementation. It works, but it is ugly and only works on FOSS drivers on Linux, not on other drivers or operating systems. A proper on/off/auto switch as part of this Vulkan extension could make this so much more sane, future-proof and portable.
| zero; otherwise it is ename:VK_FALSE if the presentation engine is | ||
| operating as a FRR display, or ename:VK_TRUE if the presentation | ||
| engine is operating as a VRR display. | ||
|
|
There was a problem hiding this comment.
Would this be a good place to also report if the swapchain's associated presentation engine is in principle capable of VRR, via a VkBool variableRefreshSupported? I'd like to have a way to find out if VRR is in principle supported on a swapchain's display (ie. driver + gpu + display support it and it is enabled by compositor/user). "variableRefresh" describes if VRR mode is active at time of query.
E.g., if connected display or gpu or display server/driver is incapable of VRR or if VRR is disabled by the user in some display setting, then both variableRefreshSupported and variableRefresh would be VK_FALSE. But if the setup is capable of VRR and just at this specific time of query VRR mode is not active, maybe because the window is not a fullscreen window or partially obscured, then variableRefreshSuppported would be VK_TRUE, but variableRefresh would be VK_FALSE.
This would be very useful for research applications but presumably also games and other apps to provide users with more accurate feedback/warnings or some troubleshooting instructions if the given use case requires VRR and VRR is not active, e.g., to figure out if it is because of obscured window / desktop composition kicking in, or instead because of unsuitable display/gpu or user error (=user disabled the required VRR feature in some display setting GUI or config file). My toolkit does this on Linux/X11, involving lots of non-portable hacks, but a proper and portable reporting mechanism in this extension would be splendid.
Update code examples accordingly
- Removed VK_PRESENT_STAGE_IMAGE_LATCHED_BIT_EXT as we couldn't agree on its semantics. Replaced with VK_PRESENT_STAGE_REQUEST_DEQUEUED_BIT_EXT which should be clearer, as "dequeue"'ing a request is already mentioned in the spec. - Swapchains that require presents to be made before being able to return timing properties should now return VK_NOT_READY from vkGetSwapchainTimingPropertiesEXT - Renamed "variableRefreshDelay" to "refreshInterval". Spec wording is still a bit fuzzy, but this allows to express scenarios where the vblank interval is decoupled from the display's true refresh rate, like Android's ARR. - VkPastPresentationTimingEXT now also reports the targetTime that was provided by the app. This is a usability thing that avoids the need to keep track of previous present times in the app. - Removed targetPresentStage: it didn't make sense from API perspective. Presentation engine now makes a "best effort" approach to hit VK_PRESENT_STAGE_IMAGE_FIRST_PIXEL_VISIBLE_BIT_EXT with the target present time. This is unfortunately the best we can do at the moment due to multiple platforms restrictions. As a result, presentAtNearestRefreshCycle is now merely a hint instead of a requirement. - Added description for the remaining issues in the proposal - Move vkGetSwapchain*PropertiesEXT optional counter parameter at the end. This is closer to the overall API style and fixes issues with vulkan.hpp generation. - Reference VK_KHR_present_id2 instead of VK_KHR_present_id - Cleanup VkPresentTimingInfo Remove the VkPresentTimeEXT union, as it does not fit the style of Vulkan Remove boolean members and replace them with a new flags type. Cleanup some spec language and VUs.
Fixed a few inconsistencies in member names, parameter orders, etc.
When using a VK_TIME_DOMAIN_PRESENT_STAGE_LOCAL_EXT time domain to specify absolute present times, it was impossible to specify which stage to reference.
Explicitly state that targetTime may also be a duration in a few places.
…stage results. Also remove the by-present-id requests and add proper VUs for presentStageCount.
The extension did not preempt a createInfo bit back then, and it hasn't come up frequently. I will reserve bit 9 in a separate PR.
nvlduc
force-pushed
the
VK_EXT_present_timing
branch
from
ba85c8d to
e4307c5
Compare
nvlduc
force-pushed
the
VK_EXT_present_timing
branch
from
e4307c5 to
7dcb614
Compare
nvlduc
changed the title
20 participants
This extension allows an application that uses the VK_KHR_swapchain extension to obtain information about the presentation engine's display, to obtain timing information about each present operation, and to schedule a present to happen at a specific
time. Applications can use this to minimize various visual anomalies (e.g., stuttering).
See the proposal doc for more details.
Status
Because of the volume of comments, it has become difficult to follow the current status of the proposal.
Here is a list of issues that still need to be addressed, in no particular order:
VkSurfaceKHRcapabilities in multiple places, e.g. at swapchain creation.VK_PRESENT_STAGE_IMAGE_HANDOFF_BIT_EXTis too controversial. Need to rename or re-spec it.VkPastPresentationTimingEXT'stimingPropertiesChangedandtimeDomainChangedare not the rightway to handle properties updates. For example, time domain could just be reported as part of the timing result instead of invalidating results, with some fallback mechanism.
VkPastPresentationTimingEXTdoes not provide enough details for dropped present requests.vkSetSwapchainTimingEXTor createInfo, rather than per-present inVkPresentTimesInfoEXT.This list is kept updated to follow current discussions.
Contributing
Because of the nature of the problem space this specification covers, the Vulkan System Integration working group has decided to move development of this specification to a public forum to make it easier to gather feedback and collaborate with developers across the various graphics ecosystems. However, to preserve the option of promoting the resulting extension to a Khronos ratified extension in the future, some guidelines (In addition to the usual contributing guidelines) must be followed when providing feedback or making contributions to the specification language.
If you have high-level feedback that does not include sensitive IP or implementation details, feel free to comment on the merge requested directly.
If you have specific wording changes or functionality you wish to add to the specification, please provide it as a pull request targeting the VK_EXT_present_timing branch of the KhronosGroup/Vulkan-Docs repository. This will require accepting the Khronos CLA before the pull request can be merged.
This is a relatively new process for Khronos, so feel free to provide meta-suggestions on how best to facilitate dialogue and foster engagement as well.