Feature: virtual keyboard

[dynamo-foundation]

If you are submitting a bug report, please respect the below template :

OS platform / Browser

Any / mobile focused

melonJS version

10

Bug description

Feature to generate pre-made virtual keyboard primarily for use on mobile platform. This would allow developers to control the size and location of the keyboard, the layout and even which keys are available. Adding one or more input forms to the HTML is cumbersome and difficult to manage, particularly on mobile. For example, if you have a login screen, user registration screen, player maintenance screen, etc. there is no easy way to manage all of those input boxes. It would be a much cleaner design to support the ability to take input from a virtual keyboard generated from keys which are buttons responding to touch inputs.

Steps to reproduce the bug

Not a bug

Console log Extract

N/A

How to enable/use the console log

• Webkit (Chrome, Safari) : https://developer.chrome.com/devtools
• Opera : http://www.opera.com/dragonfly/documentation/
• FireFox : http://getfirebug.com/enable
• MS (Edge) : https://docs.microsoft.com/en-us/microsoft-edge/f12-devtools-guide/debugger

[parasyte]

IMHO this would not be cleaner than using HTML input fields. Internationalization is the real challenge here. An English QWERTY virtual keyboard is not the only thing needed. I don't know how many keyboard layouts exist, but it is probably hundreds (if you want full coverage or you are targeting a market with a relatively uncommon keyboard layout).

And actually there are even non-QWERTY English keyboard layouts. The most popular example is probably Dvorak.

Then there are IMEs which are necessary for complex scripts like CJK. An IME is its own virtual input interface on top of a virtual keyboard.

You should also consider assistive technologies like screen readers. These apps work well with HTML input forms. Removing the form means removing support for screen readers, reducing the size of your potential audience. Another kind of assistive technology is a braille keyboard. These also work with HTML input forms.

There are special input methods that most virtual keyboards support, like swipe. These use either word lists or some predictive AI layer similar to auto-complete or auto-correct. All of these are supported by the system virtual keyboard and you get them "for free" with HTML input forms.

Because it hides a lot of the complexity, I think it is easy to fall into the trap of believing that rolling your own solution must be cleaner. But the truth is you will end up going down a rabbit hole of issues that the industry has been working on for 60 years.

I can liken it to font rendering. Displaying text that supports the ASCII character set is simple. When you need to support languages other than English, you start looking into Unicode and realize that the complexity of the entire specification is above and beyond what it seems it should be. Compared to the simplified English-only solution, the rest of the character sets are ridiculous. And that's just the scope of per-character issues like combining characters and grapheme clusters. It doesn't even include problems you will face like font parsing. Or text layout. Bidirectional text handling comes to mind as a particularly challenging example. There are a ton of JavaScript virtual keyboard implementations that fall into this category, like https://virtual-keyboard.js.org

All of these issues arise from the fact that human languages are non-trivial. They all have ambiguities and special case rules and unique properties. Handling the majority of popular languages used in the world is not easy or clean, regardless of whether you are taking text input or drawing it on the screen.

In short, you can get something working very quickly if you don't mind making it impossible to localize your game for other markets. Deferring to the OS for most things is probably a very good idea, because they have actually addressed issues that you as a game developer do not need to be concerned with. (Think of things like thread scheduling, virtual memory, and file system formatting.) The best way to currently get access to the OS virtual keyboard in an HTML app is to use HTML input forms. You should focus on abstracting problems of managing input fields instead of reinventing them completely.

The good news is that some of these things may eventually change in the future. Getting more direct access to the system virtual keyboard from JavaScript could be a real game changer for games. As far as I know, the most recent efforts have been on internals, like an IME API proposal last updated in 2016 that is supported by fewer than 1% of browsers in the wild according to https://caniuse.com/ime

And finally there is a subset of accessibility which is specific to mobile devices: https://developer.mozilla.org/en-US/docs/Learn/Accessibility/Mobile#user_input (This whole page is a good reference, but I'm linking to the most relevant part.)

While I disagree with your claim that rolling your own keyboard would be cleaner than using HTML, I can appreciate your intention. Just beware that you ignore accessibility at your own peril.

[obiot]

I would tend to agree that adding a virtual keyboard is a a project in itself, especially indeed if you need to take account localisation options and what not....

I'm not going to close this one though, because this is still a very interesting request or at least discussion, so I'll move it to discussion !