Advice needed: how to send events back from an emulation implementation

[whot]

I've been working on trying to get clipboard support in here but I keep running into a wall trying to understand the architecture and how to send events back from an emulation client (currently working on on the remote desktop part).

What I have right now is an extended LibeiEmulation that connects to the clipboard portal and gets notifications from that. I have a wire protocol to send clipboard bits left and right and (I think) they work.

What I'm stuck on is a simple thing: I need to send an event back from LibeiEmulation - in particular i need to send a "SelectionOwnerChanged" event (sent by the portal when I copy something) which contains mime-types and whatnot. And I need to send a "TransferRequest" event (sent when I paste something). And I cannot for the life of me figure out how I can send something back "up".

So basically what I have is: LibeiEmulation has a spawned task that receives the SelectionOwnerChanged/TransferRequest signal from the portal. I need to send the data from those signals to the remote end. The remote end will (eventually) reply with some protocol message at which point I can send data.

But right now I'm stuck at the "how do I actually notify the remote end from the emulation code". Any hints would be much appreciated, right now I'm stuck and I've stared at this for too long.

[feschber]

Yeah, this is something, the input-emulation was absolutely not designed to do (yet)... (as I hinted at in #387)
Currently it's a simple one way channel to send events.

I'm not sure if it would be better to separate this entirely from the input-emulation crate. Reason being that some other emulation backends most certainly won't be handling clipboard events at all and some clipboard functionality is probably also going to be handled by the input-capture, e.g. layer-shell (which makes more sense in my mind tbh).

Since we already have the restore tokens now, maybe it's easiest to add an entirely separate crate clipboard-provider, and implement it in there as a new xdg-rdp client.

But we can also extend the input-emulation - I drafted a little patch:

diff --git a/input-emulation/src/lib.rs b/input-emulation/src/lib.rs
index 930695f..fd87fbc 100644
--- a/input-emulation/src/lib.rs
+++ b/input-emulation/src/lib.rs
@@ -2,9 +2,10 @@ use async_trait::async_trait;
 use std::{
     collections::{HashMap, HashSet},
     fmt::Display,
+    future,
 };
 
-use input_event::{Event, KeyboardEvent};
+use input_event::{ClipboardEvent, Event, KeyboardEvent};
 
 pub use self::error::{EmulationCreationError, EmulationError, InputEmulationError};
 
@@ -225,10 +226,18 @@ impl InputEmulation {
             pressed_keys.insert(key)
         }
     }
+
+    pub async fn clipboard_event(&mut self) -> ClipboardEvent {
+        self.emulation.clipboard_event().await
+    }
 }
 
 #[async_trait]
 trait Emulation: Send {
+    async fn clipboard_event(&mut self) -> ClipboardEvent {
+        future::pending().await
+    }
+
     async fn consume(
         &mut self,
         event: Event,
diff --git a/input-emulation/src/libei.rs b/input-emulation/src/libei.rs
index 4e60c98..ee4ae0a 100644
--- a/input-emulation/src/libei.rs
+++ b/input-emulation/src/libei.rs
@@ -7,7 +7,7 @@ use std::{
         Arc, Mutex, RwLock,
         atomic::{AtomicBool, Ordering},
     },
-    time::{SystemTime, UNIX_EPOCH},
+    time::{Duration, SystemTime, UNIX_EPOCH},
 };
 use tokio::task::JoinHandle;
 
@@ -26,7 +26,7 @@ use reis::{
     tokio::EiConvertEventStream,
 };
 
-use input_event::{Event, KeyboardEvent, PointerEvent};
+use input_event::{ClipboardEvent, Event, KeyboardEvent, PointerEvent};
 
 use crate::error::EmulationError;
 
@@ -260,6 +260,11 @@ impl Emulation for LibeiEmulation<'_> {
         let _ = self.session.close().await;
         self.ei_task.abort();
     }
+    async fn clipboard_event(&mut self) -> ClipboardEvent {
+        // TODO receive from a channel here
+        tokio::time::sleep(Duration::from_secs(10)).await;
+        ClipboardEvent::TransferRequest
+    }
 }
 
 async fn ei_task(
diff --git a/input-event/src/lib.rs b/input-event/src/lib.rs
index 1d8c9ff..58c4a7a 100644
--- a/input-event/src/lib.rs
+++ b/input-event/src/lib.rs
@@ -13,6 +13,12 @@ pub const BTN_MIDDLE: u32 = 0x112;
 pub const BTN_BACK: u32 = 0x113;
 pub const BTN_FORWARD: u32 = 0x114;
 
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub enum ClipboardEvent {
+    SelectionOwner,
+    TransferRequest,
+}
+
 #[derive(Debug, PartialEq, Clone, Copy)]
 pub enum PointerEvent {
     /// relative motion event
diff --git a/src/emulation.rs b/src/emulation.rs
index 853267c..8752d4e 100644
--- a/src/emulation.rs
+++ b/src/emulation.rs
@@ -1,7 +1,7 @@
 use crate::listen::{LanMouseListener, ListenEvent, ListenerCreationError};
 use futures::StreamExt;
 use input_emulation::{EmulationHandle, InputEmulation, InputEmulationError};
-use input_event::Event;
+use input_event::{ClipboardEvent, Event};
 use lan_mouse_proto::{Position, ProtoEvent};
 use local_channel::mpsc::{Receiver, Sender, channel};
 use std::{
@@ -52,6 +52,8 @@ pub(crate) enum EmulationEvent {
     EmulationEnabled,
     /// capture should be released
     ReleaseNotify,
+    /// clipboard content
+    Clipboard(ClipboardEvent),
 }
 
 enum EmulationRequest {
@@ -378,6 +380,9 @@ impl EmulationTask {
                     ProxyRequest::Terminate => break Ok(()),
                     ProxyRequest::Reenable => continue,
                 },
+                cb_event = emulation.clipboard_event() => {
+                    self.event_tx.send(EmulationEvent::Clipboard(cb_event)).expect("channel closed");
+                },
             }
         }
     }
diff --git a/src/service.rs b/src/service.rs
index ff7c22f..d796992 100644
--- a/src/service.rs
+++ b/src/service.rs
@@ -263,6 +263,7 @@ impl Service {
 
     fn handle_emulation_event(&mut self, event: EmulationEvent) {
         match event {
+            EmulationEvent::Clipboard(e) => log::info!("{e:?}"),
             EmulationEvent::ConnectionAttempt { fingerprint } => {
                 self.notify_frontend(FrontendEvent::ConnectionAttempt { fingerprint });
             }

[feschber]

Thinking about this a bit more, I think we should definitely first extend input-emulation the way you started before we overcomplicate things for no reason.

I can still extract this into some other crate in the future, if I ever feel the need to do so.

[whot]

Yeah, going the simpler path seems to be the better choice for now :) I'm wondering: since we need both emulation and capture on both ends anyway and capture is definitely not a one-way street - we could integrate clipboard into the input capture session and go from there? At least with portals this should work just fine. The only requirement here is of course to be able to send events to and from the capture session but I think this is already possible (without checking, I've mainly focused on the remote desktop code so far).

Oh, and just ftr, my very much WIP branch is here and it has bits and pieces in place (for emulation), so it's the connection between portal code and core code that's missing.

[nbolton]

https://github.com/feschber/lan-mouse/issues/github.com/whot/lan-mouse/tree/wip/inputcapture-clipboard

Heads up: link is broken
https://github.com/whot/lan-mouse/tree/wip/inputcapture-clipboard

[feschber]

Would

I'm wondering: since we need both emulation and capture on both ends anyway and capture is definitely not a one-way street - we could integrate clipboard into the input capture session and go from there? At least with portals this should work just fine.

Are you suggesting to spawn a remote desktop session within the input-capture?

Logically I'd say it would make sense for the input-capture to generate clipboard content (i.e. receive from the compositor) and the input-emulation to consume clipboard content (i.e. send it to the compositor).

However, since the remote desktop portal seems to do both of these things, we can't really put it into either of those.
Thus, I would almost prefer to put it into a separate crate that does all of the clipboard stuff independently.

Btw, are there any plans to support drag & drop in xdg-desktop-portal?
Because that would have to be integrated with the input-capture portal in some capacity...

fixed link
just a heads up, in case you haven't noticed: The network protocol is unreliable atm.
For key and pointer events we simply don't care about that (worst case, a key gets stuck until the user presses it again).

For larger clipboard contents there probably should be a mechanism to re-request clipboard contents until the whole thing is received, or potentially a reliable data channel of some sort that handles this.

[whot]

Link is fixed, thanks @nbolton

Are you suggesting to spawn a remote desktop session within the input-capture?

no, on the dbus side the clipboard portal sort-of attaches itself to a session, so we can simply attach to the input-capture session and receive and write out clipboard from that. And we leave the remote session as-is so we don't need to rewrite that.

Btw, are there any plans to support drag & drop in xdg-desktop-portal?
Because that would have to be integrated with the input-capture portal in some capacity...

You mean drag-and-drop across hosts? I don't have an answer for that, would have to dig through the portal sources/PRs and see what is currently being worked on (this is a a side-gig for me, mainly :)

just a heads up, in case you haven't noticed: The network protocol is unreliable atm.

oh. I did not notice... the approach you say seems fine, but can (hopefully) be tacked on once we get something working :)

[feschber]

no, on the dbus side the clipboard portal sort-of attaches itself to a session, so we can simply attach to the input-capture session and receive and write out clipboard from that. And we leave the remote session as-is so we don't need to rewrite that.

Interesting. That would require changes in ashpd as well, right?

[whot]

Interesting. That would require changes in ashpd as well, right?

yes, but I have those ready locally.

[whot]

ftr bilelmoussaoui/ashpd#364 is the last remaining bit on the ashpd side, assuming the portal MR gets merged, the clipboard support for inputcapture is already in the portal and ashpd git (technically in 0.13.0 but a bug prevents it from working)