Support wlr-virtual-{pointer,keyboard} for headless wlroots / sway session

[falk4243]

There's issue #137, but the thread is quite diluted and not a feature request, so I figured I'd post one here in the hopes that someone (not a dev myself) working on Sunshine might find this an interesting idea ...

Use-case: sway lends itself really well to setup a headless Sunshine session that allows for streaming games in the background while continuing to use the host for other tasks in another session. For example, I have a sway session running on tty6 and xfce on tty7. This works really well as long as a game controller can be used, which keeps its focus on the sway session even if xfce is brought up on the host. Unfortunately, physical and virtual keyboard and mouse input is only possible as long as the sway session is in focus, otherwise all events are sent to the xfce / X11 session.

This is OK for my purposes 90% of the time, but I couldn't help wondering if there isn't a way to do this properly and had a little chat with "emersion" (working on sway and wlroots) who mentioned that as long as evdev is being used, this would probably not be possible and recommended supporting wlr-virtual-keyboard and wlr-virtual-pointer in order to send virtual input to the sway session in the background.

wayvnc supports this protocol and it works quite nicely, though only on wlroots / sway. Still, that's probably a big step forward as firing up a headless sway session is quickly done and comes with little overhead compared to a full blown DE. Also, launching sway on its own tty or from within another X11 or wayland session is quite painless so that this could become the go-to solution for people wanting to setup a poor man's multiseat system without the (administrative) overhead of virtualization or containers.

Since Sunshine does support multiple ways for screen capture as well, would it be possible to support the aforementioned protocol(s) in addition to evdev to enable this use-case?

[falk4243]

Alright, after a long silence and my setup changing quite a bit, I have come full circle with the above request while trying to run a fully accelerated cloud desktop based on a Incus / LXD container and sway / wlroots. This works really well for the audio and video part by just passing the GPU to the container, but Sunshine's virtual input devices (keyboard, mouse) won't show in libinput just the same as described above. Game controllers can be used as before, but I'd really like to use m/k to control the desktop via Moonlight.

There's a hack how to make this work: "Overlay" Moonlight with a transparent TigerVNC window and use that for input via wlr-virtual-keyboard and wlr-virtual-pointer, which does register correctly with sway. This combination makes for a super simple and fully accelerated cloud desktop (video, 3D, without physical display) running in a container.

The question is as before, would it be possible to implement both protocols in Sunshine as an alternative to the existing virtual devices via uinput / udev (which does not work in the container)? If this is somewhat feasible, I wouldn't mind compensating you for your time if the result makes it into Sunshine as it would be really nice to have this working directly without having to use input via VNC as a crutch. Anybody interested in picking this little project up?

[cgutman]

When Sunshine is running in that environment, does /dev/uinput work at all or does it fail to create the virtual devices when we try to use it? We have some fallback code for X11 to inject events using XTest. Hopefully it would suffice to add another fallback for Wayland when uinput is unavailable, otherwise we'll need to create a configuration option for it.

[falk4243]

Thank you for your reply and /dev/uinput works when passed through to the container as a "unix-char" device. With this, the necessary/dev/input/eventX devices are created on the host, but even passing those through to the container is of no use since libinput (and consequently sway) cannot find them. Only the controller works (accessed by games directly), as well as testing mouse and keyboard with "evtest" within the container - so close and yet so far ...

[chaitan3]

Like @falk4243 mentioned, sway cannot find the input devices that are created by sunshine even if the devices are visible inside the container as the method by which sway detects if devices are attached depends on seats and udev and the coordination between them. In a headless environment all of these don't exist.

It would be good to add direct support for wayland virtual input backends like libei or wlr-virtual-keyboard rather than relying on uinput in sunshine.

libei has broader support: Gnome, KDE (planned) and seems to have support for controllers in addition to mouse/keyboard. Whereas, wlr-virtual-* is restricted to wlroots based compositors: sway, hyprland, Xfce (planned)?