support for rm2fb

[raisjn]

when running rm2fb (as a package from https://github.com/toltec-dev/toltec), the behavior is the following:

• start rmview - connect and the screen displays properly
• the pen trail shows up correctly
• no further updates to the display are shown

when running rm2fb, there are two binaries with the paper plugin: remarkable-shutdown (hosting rm2fb server) and xochitl (which has been re-configured to send its updates to the server). is it possible that rmview is using xochitl (instead of remarkable-shutdown?)

PS: what logs / debugging information should i attach? all I have so far is the output of rmview which doesn't contain much information.

[raisjn]

from @SyntaxBlitz on discord

I'm having some trouble on rM2.. it connects, shows the current screen on my computer, then nothing can write to the framebuffer anymore until I restart the device. clear-full-screen messages are still making it through (when I open a notebook and swipe left/right, or when I press the button to sleep), but nothing is actually drawn. just frozen on the initial screen (on the desktop client and on the actual device). pen location makes it through to the desktop client but no screen updates

i believe the issue is rm2fb because i uninstalled rm2fb, reboot the device and rmview works as expected

[pl-semiotics]

Thank you for the report! I will try to take a look.

It is particularly interesting because you use QMetaObject::invokeMethod, which is precisely the way in which I find sendUpdate (in the sense that it grabs the appropriate metaobject and eventually resolves to a call to the qt_static_metacall function that I emulate to find the address), so we certainly ought to be finding the same function.

when running rm2fb, there are two binaries with the paper plugin: remarkable-shutdown (hosting rm2fb server) and xochitl (which has been re-configured to send its updates to the server). is it possible that rmview is using xochitl (instead of remarkable-shutdown?)

The library finds out which process is controlling the framebuffer by looking for who has /dev/fb0 open. If you let xochitl open /dev/fb0 there may be some issue, but since this happens in the EPFramebuffer initialization routines I was assuming that you patch it prior to that.

In case you don't want to build a copy yourself (I do apologize for the build infrastructure at the moment! If you would like to but have any issues please let me know and I'll try to improve it), here is a version where the fb logic can be overridden for debugging. Could you try running it with LIBQSGEPAPER_FBCON_PID=<pid of rm2fb>?

PS: what logs / debugging information should i attach? all I have so far is the output of rmview which doesn't contain much information.

Unfortunately, most debugging is via gdb and snooping on the Unix socket used to send information around at this point, so if the process didn't die with an informative error there's not a lot of logs per se.

If you could send me the specific xochitl and remarkable-shutdown binaries you used (and any other shared objects needed to run them), it would make debugging quite a bit easier.

from @SyntaxBlitz on discord

I'm having some trouble on rM2.. it connects, shows the current screen on my computer, then nothing can write to the framebuffer anymore until I restart the device. clear-full-screen messages are still making it through (when I open a notebook and swipe left/right, or when I press the button to sleep), but nothing is actually drawn. just frozen on the initial screen (on the desktop
client and on the actual device). pen location makes it through to the desktop client but no screen updates

I suppose this might be caused by my hooking of sendUpdate if I ended up hooking xochitl and broke the frida interception (e.g. if it does something pc-relative, since it gets relocated by my own interception).

Thanks!

[raisjn]

I tried the supplied binary and supplied the PID of remarkable-shutdown process, still no luck.

for rm2fb: can download the server and client.so files from: https://github.com/ddvk/remarkable2-framebuffer/releases/tag/v0.0.2

the short is: LD_PRELOAD=/path/to/server.so.1.0.0 /usr/bin/remarkable-shutdown to run server, to run client: LD_PRELOAD=/path/to/client.so.1.0.0 /usr/bin/xochitl (make sure to give full paths to all binaries).

i tested on 2.5 version of xochitl:

md5s are

• fec600ccae7743dd4e5d8046427244c0 /usr/bin/remarkable-shutdown
• aaa77163501618b0cfae54843505df88 /usr/bin/xochitl

[pl-semiotics]

Oh, dear, I think I see the issue. You have replaced the framebuffer that the EPFramebuffer draws from with your own shared memory. Then I have replaced it again with my own shared memory. Your clients write into your shared memory which is no longer mapped in the process, but EPFramebuffer and my server read from my shared memory which is never updated. The damage tracking is working fine, it's just that the repainted areas are being drawn from the old backing store.

I apologize for the delay---I had been looking just at your swtfb.cpp where you actually send updates, and did not realize that server/main.cpp is also doing things.

I don't know what a great fix that wouldn't involve an annoying amount of special-casing is. Since you are already setting up shared memory and already intercepting all surface damage, I think the best bet (unless you have any better ideas) is probably to provide a vnc backend that gets the information directly from your server without having to deal with the injection (I think you can easily make an LD_PRELOAD'able .so which adds this---it just needs a call to register_backend in its .ctors---and the vncs.c would need to be made a little bit smarter about allowing the user to choose a backend when multiple are available---I'd love a patch for this ;)). I would also be amenable to modifying libqsgepaper-interact to work with rm2fb-taken-over'd processes, but I'd need you to provide some means of identifying that the process has been modified by your library, such as magic numbers somewhere convenient (before/after the QImage of the framebuffer/its vtable, most probably). Let me know how you'd like to proceed.

(I do appreciate your zero-copy approach. Given how much work the software tcon thread is doing, I'm becoming more convinced that one or two extra copies would be fairly harmless, though, which is why I'm thinking of Wayland---I rather like the idea of having a standard protocol and a fairly decent Xorg DDX that already has support for XDamage reporting :). With appropriate buffer sizing optimizations and perhaps protocol changes, I think that wl_shm can probably get down to one extra copy, anyway, unless you're doing fancy things with subsurfaces (and perhaps sometimes even then). It's a bit annoying that the big wayland compositor libraries like wlroots are all so tightly tied to EGL/linux-dmabuf, though.)

[pl-semiotics]

(Oh, this is a bit tangential to the issue at hand, but as to compositor design---IIRC wayland recently came up with a fullscreen unredirect protocol extension to let fullscreen clients draw directly to the scanout buffer---so potentially that would provide the same zero-copy-ness).

[raisjn]

my preference here is to avoid modifying rm2fb because there's already some set of people using it. oth - since its been installed via a package manager, it should be easy enough to roll out updates. in order of preference (based on least amount of modification)

1. the rm2fb process can be identified because it's opened fb0 and its cmdline will contain remarkable-shutdown. until rM team removes that binary, its the one we plan on using.

2. we can also add an environment variable to the binary (or otherwise add some magic values somewhere)

3. for the vnc backend option, maybe one method would be to add another message queue in rm2fb where updates will be sent to, then in this repo, we read from that queue and the shared memory in a new backend type.

[pl-semiotics]

my preference here is to avoid modifying rm2fb because there's already some set of people using it. oth - since its been installed via a package manager, it should be easy enough to roll out updates. in order of preference (based on least amount of modification)

That makes sense.

1. the rm2fb process can be identified because it's opened fb0 and its cmdline will contain `remarkable-shutdown`. until rM team removes that binary, its the one we plan on using.

I'd rather not rely on this, simply because remarkable-shutdown may be used for other things as well. So on the principle of being the least likely to accidentally trash something if run in an unknown environment, I'd prefer to have a bit of a stronger indication.

However, I could also imagine letting the user indicate that they are using rm2fb in which case the normal "looking for who is on /dev/fb0" process works.

2. we can also add an environment variable to the binary (or otherwise add some magic values somewhere)

Magic values around the framebuffer itself are probably the most reasonable thing to special-case support for in libqsgepaper-snoop.a. But see my comment below.

3. for the vnc backend option, maybe one method would be to add another message queue in rm2fb where updates will be sent to, then in this repo, we read from that queue and the shared memory in a new backend type.

That makes sense. It occurs to me that since you already have the framebuffer mapped, though, that perhaps the simplest thing to do is to add a config parameter to libqsgepaper-snoop.a that allows disabling grabbing the framebuffer, since I understand that you already have some way of allowing other processes to acquire the framebuffer. Then we could write another vnc backend that gets the damage updates via patching sendUpdate ourselves through my library, but gets the framebuffer from you.

If there are easily-locatable magic values in the patched process, then this can be combined with the normal rm2 backend and libqsgepaper-snoop.a can just decide dynamically whether to steal the framebuffer or to nab a file descriptor for it from wherever you are stashing it. Or, if you decide to prefer to avoid that, it can be an option for the user via backend selection as described above.

[raisjn]

However, I could also imagine letting the user indicate that they are using rm2fb in which case the normal "looking for who is on /dev/fb0" process works.

That makes sense. It occurs to me that since you already have the framebuffer mapped, though, that perhaps the simplest thing to do is to add a config parameter to libqsgepaper-snoop.a that allows disabling grabbing the framebuffer, since I understand that you already have some way of allowing other processes to acquire the framebuffer. Then we could write another vnc backend that gets the damage updates via patching sendUpdate ourselves through my library, but gets the framebuffer from you.

this sounds good to me. the framebuffer is just opened as shared mem in /dev/shm/swtfb.01. i think rmview can detect if rm2fb is running (systemctl is-active rm2fb) and set the right config param to tell the snooper to read the framebuffer from the correct shared mem (or even accept the location as a config param)

btw: i like how you obtain the framebuffer without using magic bytes, nice work!

[MrShoenel]

Hey, any update on this? If I understand this correctly, then launchers like remux and oxide require rm2fb (I am using remux on the rM2). If I do uninstall all of these, rmview works nicely, but not with them installed.

[raisjn]

no updates, but looking at the code, i think adding a new backend is more appropriate. we will update rm2fb to emit dirty events, then we can have an rm2fb backend

[pl-semiotics]

I apologize for the lack of updates---I've been a bit busy and not had time to finish doing RM2 things, unfortunately.

I just got back to looking at this today. It looks like the sendUpdate hooking that libqsgepaper-snoop does right now will still work, and in fact even the address calculation for the fb address should work---we just need to turn off the "map a new shared memory framebuffer over the one used by the process" code. This is pretty easy---I just implemented the most important bit in libqsgepaper-snoop this morning. This binary should work with rm2fb if you run it as:

LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb> /path/to/downloaded/server/binary

(If you have something else touching the mxs framebuffer, you can try making sure that it gets the right process via LIBQSGEPAPER_FBCON_PID=<pid of rm2fb server>)

It would presumably also work to grab the shared memory via the normal shmem mechanisms, rather than pulling it out of your process via a Unix socket, but this is a slightly smaller code change, seems to work with rm2fb, and should support other options as well.

I may also write a bit of code in backend-qsg.c to speak with systemd and get the appropriate pid and fd.

[raisjn]

awesome!

It would presumably also work to grab the shared memory via the normal shmem mechanisms, rather than pulling it out of your process via a Unix socket, but this is a slightly smaller code change, seems to work with rm2fb, and should support other options as well.

how are people supposed to know the fd to pass to LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb>? it seems more reasonable to just use open() (or shm_open) on /dev/shm/swtfb.01

[pl-semiotics]

how are people supposed to know the fd to pass to LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb>? it seems more reasonable to just use open() (or shm_open) on /dev/shm/swtfb.01

I was going to just grab it from reading /proc/<pid>/fd/ in backend-qsg (since one can get the pid from systemd over dbus), but that is indeed an issue. I can add support for using some other open()'d file in the client process instead, but since I don't know of any other consumers of the API (or specifically, such consumers that want support for this) I thought this might be easier. (Since one needs to speak to systemd/get the pid of rm2fb anyway to know whether or not to run this code path.)

[raisjn]

ok, that makes sense - it is just some code that needs to connect the fd. i thought this was going to be something everyone has to do.

[pl-semiotics]

Ah, yes, that would have been a problem. The reason I haven't pushed the libqsgepaper code for release yet is that I was going to write that this evening---I just wanted to update that the tiny modification worked.

(I'll also need someone to test the wiring-fds-up code on a real device, since my emulator of course doesn't use systemd to launch the rm2fb.)