Adding keyboard controls to vsg::Trackball, feedback on keyboard bindings sought

[robertosfield]

Hi All,

Today I started experimenting with adding keyboard controls to vsg::Trackall that make it possible to move the eye/look point with the keys, I've created a TrackballKeys branch for this:

https://github.com/vsg-dev/VulkanSceneGraph/tree/TrackballKeys

So far I've used the left, right, up and down keys and the shift and alt modifiers. What I've checked in so far is really more of a proof of concept of how it might work rather than final implementation. Currently with no modified the arrow keys pan, but if you press alt at the same time they control turn left/right and move forward/back. The shift key reduces the speed of movement to 20%.

I have decided to next experiment with motion per frame rather in just at the time of key press. The time the key has been pressed would be used as as way of controlling speed, so it'll behavior like an initial acceleration then up to constant speed and when you let go constant deceleration back to zero. This is bit more awkward to implement but I'm thinking it will feel more natural.

I'm not happy with my experiment with overloading the arrow keys with different behavior based on the modifier key and now feel that just putting these controls on other keys would be appropriate. The key binding used will something developers will be able to set themselves, as they can for the existing keyboard bindings in vsg::Trackball for the move to viewpoint support, but I'm curious what folks feel like the defaults should be.

If there is a dominant keyboard binding conventions used by applications or big games please enumerate them in this thread so we can all get an idea of what users might already be used to. I expect there will be lots of combinations used, but with a wider picture we might be able to pick bindings that will be intuitive for as many users as we can support.

[timoore]

On Fri, May 5, 2023 at 7:39 PM Robert Osfield ***@***.***> wrote: Hi All, Today I started experimenting with adding keyboard controls to vsg::Trackall that make it possible to move the eye/look point with the keys, I've created a TrackballKeys branch for this: https://github.com/vsg-dev/VulkanSceneGraph/tree/TrackballKeys Yay. If there is a dominant keyboard binding conventions used by applications or big games please enumerate them in this thread so we can all get an idea of what users might already be used to. I expect there will be lots of combinations used, but with a wider picture we might be able to pick bindings that will be intuitive for as many users as we can support. A lot of games use w s a d for forward/back/left/right, or some variation

of that. The VI movement keys are another options. I'd love to see a manipulator that rotates the view while keeping the eye point fixed, but that could be another manipulator.

…

— Reply to this email directly, view it on GitHub <#801>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAQAIUK7PFOII5AHUBJDTXEU3MLANCNFSM6AAAAAAXXMZEB4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[robertosfield]

One possibility would be to have a convenience method for setting certain common mappings i,e, VI style etc.

W.r.t moving around the eye, I've done this in the branch already, pressing Alt + left arrow, Alt + right arrow turns the camera left or right.

[kannode]

Robert, Thank you for doing this. I have not looked at the branch yet. One set of keys that I always use to tether to an object O, and have Camera C look at O. Use spherical coordinates (r,phi,theta) https://en.wikipedia.org/wiki/Spherical_coordinate_system A spherical orbit around the i, o for zoom in zoom out i.e., modify 'r' w, s to pitch up and down by camera phi a, d to pan left and right by modifying theta. Suresh.

…

On Fri, May 5, 2023 at 1:39 PM Robert Osfield ***@***.***> wrote: Hi All, Today I started experimenting with adding keyboard controls to vsg::Trackall that make it possible to move the eye/look point with the keys, I've created a TrackballKeys branch for this: https://github.com/vsg-dev/VulkanSceneGraph/tree/TrackballKeys So far I've used the left, right, up and down keys and the shift and alt modifiers. What I've checked in so far is really more of a proof of concept of how it might work rather than final implementation. Currently with no modified the arrow keys pan, but if you press alt at the same time they control turn left/right and move forward/back. The shift key reduces the speed of movement to 20%. I have decided to next experiment with motion per frame rather in just at the time of key press. The time the key has been pressed would be used as as way of controlling speed, so it'll behavior like an initial acceleration then up to constant speed and when you let go constant deceleration back to zero. This is bit more awkward to implement but I'm thinking it will feel more natural. I'm not happy with my experiment with overloading the arrow keys with different behavior based on the modifier key and now feel that just putting these controls on other keys would be appropriate. The key binding used will something developers will be able to set themselves, as they can for the existing keyboard bindings in vsg::Trackball for the move to viewpoint support, but I'm curious what folks feel like the defaults should be. If there is a dominant keyboard binding conventions used by applications or big games please enumerate them in this thread so we can all get an idea of what users might already be used to. I expect there will be lots of combinations used, but with a wider picture we might be able to pick bindings that will be intuitive for as many users as we can support. — Reply to this email directly, view it on GitHub <#801>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHWFHIJW4TRZXHJCZEKX3WDXEU3MRANCNFSM6AAAAAAXXMZEB4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- S <https://www.nodein.com/>uresh K. Kannan, Ph.D., Chief Scientist, Nodein https://www.nodein.com

[robertosfield]

I have progressed a bit further with my experiment with adding keyboard controls to vsg::Trackball, the code can be found in the TrackballKeys branch:

https://github.com/vsg-dev/VulkanSceneGraph/tree/TrackballKeys

I have now implemented a form of keyboard state tracking that can be queried whether a key is pressed or not, this is then used by the Trackball::accept(FrameEvent&) to decided how to move the Camera's LookAt. Moving from responding directly to KeyPress events to testing whether a key is pressed makes the movement far smoother.

However, the keyboard state tracking it does add quite a bit more complexity to Trackball which is not core to features the class provides, so it would be a natural step to move the keyboard state tracking out into it's own class, which the Trackball could then use. To make this class more robust I'll also need to implement a focus in/focus out so we can handle the case where the focus moves from the VSG window to another window on the desktop and start getting the events while the key is still pressed. When when this happens the keyboard state tracking doesn't know that a key has been released so no longer in a pressed state.

I will add the focus in/out to the XCB support but will need others in the community to help out with implementing on Windows/macOS etc.

If folks want to test out the new functionality in the TrackballKeys branch the key bindings are presently hardwired to:

Key	Action
Left Arrow	pan left
Right Arrow	pan right
Up Arrow	pan up
Down Arrow	pan down
a	turn left
d	turn right
w	move forward
s	move backward

In the final rev these keys will be user configurable. Also consider the present code as experimental/proof of concept.

As well as the implementing a vsg::Keyboard state class I want to implement a very basic inertia to the rate of movement so it feels a little more natural rather than having the eye point accelerate instantaneously to a new speed like it does with the present code.

I am happy to take suggestions on how to refine this functionality, if there are 3rd party examples worthy of checking out let me know.

[robertosfield]

I have added support for pitching the camera up/down and have changed the keyboard bindings to broadly what @kannode suggested:

Key	Action
Left Arrow	pan left
Right Arrow	pan right
Up Arrow	pan up
Down Arrow	pan down
a	turn left
d	turn right
w	look up
s	look down
o	move forward
i	move backward

These will need to be user controllable rather than hardwired, but for now I'm happy to test with hardwired bindings.

[kannode]

Thank you @robertosfield. I will take a look at it as soon as I can.

[robertosfield]

I have created a vsg::FocusInEvent and vsg::FocusOutEvent classes with support in Xcb_Window.cpp and Win32_Window.cpp so that we can respond to changes in the keyboard focus. I don't have a Windows system so have only tested under Linux, I took an educated guess at what the Win32 code should be based on online docs and what the OSG codebase does so I think there is reasonable change it'll work.

I don't have macOS, iOS or Android systems or expertise so haven't attempted adding support for FocusIn/FocusOut events. I would appreciate help from the community to add support on these platforms.

I have used the FocusOutEvent in the new vsg::Keyboard class so that it automatically clears the keys it's tracking to avoid it having key held state when the window doesn't have keyboard focus. This prevents the Trackball from mistakenly applying movement based on keys that were pressed before the keyboard focus was lost.

I have also moved the new vsg::Keyboard class out into it's own header/source file.

The functionality is now starting to feel that it's getting close to solid enough to merge with master. I still have to make the keyboard bindings user definable and clean up the code but it's certainly far enough along now for users to test out the TrackballKeys branch.

[robertosfield]

I have made the keys user settable:

20f7b54

[robertosfield]

I have now implemented the inertial effect so the the the longer you keep a key pressed the greater the speed of movement or rotation and when you release the key that movement/rotation will continue decelerating at the same pace as the initial acceleration.

This avoids the jarring sudden start/stops as well as enabling short key presses to nudge the viewpoint so you can achieve finer precision when setting up the viewpoint.

To implement that I've added bool methods to the vec2,vec3, plane and quat classes so you can conveniently check to see if a vector is non zero. This means you use these types in a similar way to the standard numeric and bool types when a non zero value is true, and zero is false. The commit 68fe636 add this functionality.

The inertial effect needs to know the time since the key was first pressed and the time since it was released so I've added a Keyboard::times(..) method that returns a std::pair<double, double> with these two times. The Trackball illustrates how this is used: b30c90d

I think I'll add a roll into the keyboard controls as well, but overall I'm now pretty happy with the behavior so will merge with VSG master in the next hour or so.

[robertosfield]

I have implemented roll using the keys: f08409d

Key	Action
a	turn left
d	turn right
q	roll left
e	roll right
w	look up
s	look down
w	look up
s	look down
Left Arrow	move left
Right Arrow	move right
Up Arrow	move up
Down Arrow	move down
o	move forward
i	move backward

[robertosfield]

I have merged the changes with VSG master:

#816

[robertosfield]

One area I'd like feedback from the community is testing on Windows and macOS as I don't have access to these systems. In particular the vsg::Keyboard tracks key presses and key releases, and if a key repeat is happening it implements detection of the dummy KeyEeleaseEvent that precedes the repeated KeyPressEvent and essentially treats these two as a repeat if the time of the release and key press are the same. This works reliably on xcb but I don't know how key repeats are done on Win32 and macOS so this scheme might fail if they handle the key repeat in a different way.

[kannode]

Which vsg example is best to try the trackball

On Wed, May 17, 2023 at 7:57 AM Robert Osfield ***@***.***> wrote: One area I'd like feedback from the community is testing on Windows and macOS as I don't have access to these systems. In particular the vsg::Keyboard tracks key presses and key releases, and if a key repeat is happening it implements detection of the dummy KeyEeleaseEvent that precedes the repeated KeyPressEvent and essentially treats these two as a repeat if the time of the release and key press are the same. This works reliably on xcb but I don't know how key repeats are done on Win32 and macOS so this scheme might fail if they handle the key repeat in a different way. — Reply to this email directly, view it on GitHub <#801 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHWFHIL7FYVD5QDQ2ESOAR3XGS4I7ANCNFSM6AAAAAAXXMZEB4> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

-- Relevant Theory

[robertosfield]

On Thu, 18 May 2023 at 11:57, Suresh Kannan ***@***.***> wrote: Which vsg example is best to try the trackball

I have just been using vsgviewer, vsgwindows and vsgmultiviews for testing, but that's more for making sure the handling of keyboard focus works fine. Making it work OK for a range of models is another aspect. The speed of movement is scaled by the distance between the eye and center of Camera's LookAt settings, if you zoom with the mouse in this will reduce distance and hence the speed of movement.

…

Message ID: ***@***.*** com>

[jamierobertson1]

It behaves a bit strangely on windows. It initially accelerates while key pressed for approximately 1s, then stops moving. Cant immediately see why it stops but I've checked and there are no false KeyReleaseEvents.

Like the idea though.

[robertosfield]

Could you put some debug output into the Keyboard::apply(KeyPress&) and apply(KeyRelease) code to see events are being generated when you hold down the key. Does the KeyEvent::repeatCount get incremented?

[jamierobertson1]

I get the following when I hold down a key, and a single apply(KeyReleaseEvent) only when I actually release the key.

Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyPressEvent& keyPress)repeatCount: 1
key reusing first press 0ms
Keyboard::apply(KeyReleaseEvent& keyRelease)
key provisionally released 33.5694ms

[jamierobertson1]

I think for windows, the only way to identify repeated keypress events is by comparing keypress events to their previous state. The "repeatCount" here is the number of missed repeats between successive processing of keydown events (so not a value that increments as long as keys are held down).

Perhaps adding something like this in Keyboard::apply to check whether the preceeding event was also a key down event (and therefore the key is held down) would work without causing adverse effects to other platforms?

`       if (keyHistory.timeOfKeyRelease == keyPress.time || keyHistory.timeOfKeyRelease < keyHistory.timeOfLastKeyPress)
        {
            keyHistory.timeOfKeyRelease = keyHistory.timeOfFirstKeyPress;
            keyHistory.timeOfLastKeyPress = keyPress.time;
        }`

[vsg-dev]

Is it line 28 of Keyboard.cpp that you have modified by adding the || keyHistory.timeOfKeyRelease < keyHistory.timeOfLastKeyPress ?

Have you tested it out and found it to work reliably? If so could you generate a PR for this change.

I am wondering about whether it would be would adding in the number of keyRepeats into Keyboard::KeyHistory so we can better track this for apps that want an easier way of insulating themselves from different Windowing platforms doing different things.

[jamierobertson1]

Yes, thats the bit. It works reliably on windows. It's not really necessary update the keyHistory.timeOfKeyRelease here for windows as there are no dummy key release events, but it's already set to the first key press time elsewhere so should just be setting it to the same value as it was already.