3 issues encountered with GTk3 apps using the wayland GDK backend

[RJVB]

Continued from issue #13 ...

I've built Owl and also managed to build GTk3 with the Wayland backend. I notice 2 things:

1. GTk3 uses wl_pointer_add_listener() to get pointer events, notably clicks. Some clicks do come through but for the most part it is as if there are no mouse events being transmitted. At least the button click handler doesn't appear to be called at all.

2. When I try to open a menu in any GTk3 application the compositor aborts under wl_invoke_closure() because there's a NULL listener for "opcode 1" in xdg_wm_base (whatever that means). When I skip the abort and simply don't do the callback the compositor survives but the client is terminated with an error instead of a crash - which I find much preferable.

3. After testing this menu-related failure a few time I witnessed this error (for a 2nd time):

2025-03-06 02:13:30.214 Owl[38330:d07] Running compositor on Wayland display wayland-0
2025-03-06 02:13:30.215 Owl[38330:d07] Checked in with the bootstrap server as io.github.bugaevc.Owl
listener function for opcode 1 of xdg_wm_base is NULL
error in client communication (pid 38334)
listener function for opcode 1 of xdg_wm_base is NULL
error in client communication (pid 38379)
listener function for opcode 1 of xdg_wm_base is NULL
error in client communication (pid 38408)
Buffer address requested when its parent pool has an external reference and a deferred resize pending.
2025-03-06 02:14:59.978 Owl[38330:d07] Unknown shm format
Owl(38330,0x7fff746e2310) malloc: *** mach_vm_map(size=18392700977636450304) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

with backtrace

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libwayland-server.0.dylib     	0x00000001048ed2eb wl_shm_buffer_begin_access + 11 (wayland-shm.c:637)
1   io.github.bugaevc.Owl         	0x00000001048a52c7 -[OwlShmBuffer invalidate] + 167 (OwlShmBuffer.m:74)
2   libffi.8.dylib                	0x0000000104925052 ffi_call_unix64 + 82
3   libffi.8.dylib                	0x0000000104929ba7 ffi_call_int.llvm.BC09A206 + 807
4   libwayland-server.0.dylib     	0x00000001048efc14 wl_closure_invoke + 596
5   libwayland-server.0.dylib     	0x00000001048eae4b wl_client_connection_data + 523 (wayland-server.c:445)
6   libwayland-server.0.dylib     	0x00000001048ede8b wl_event_loop_dispatch + 123
7   io.github.bugaevc.Owl         	0x00000001048aaf71 socketCallback + 17 (OwlRunLoopSource.m:49)
8   com.apple.CoreFoundation      	0x00007fff8ed04057 __CFSocketPerformV0 + 855

Not an Owl issue but for completeness: using OpenGL will probably require building Mesa with both EGL and Wayland support. Or would it require going through Apple's OpenGL instead?? Anyone got an idea about that?

[0xZaph]

I ran into a possibly similar issue when testing out AttoUI and its button implementation and the problem was smth to do with OwlSurface.m:

if (damageAll) {
        [self setNeedsDisplay: YES];
    } else {
        for (NSValue *value in [_currentState damage]) {
            NSRect rect = [value rectValue];
            rect.origin.y = [newBuffer size].height - rect.size.height - rect.origin.y;
            [self setNeedsDisplayInRect: rect];
        }
    }

iirc. If I always updated it then the button would work (at least visually) so maybe the problem is to do with how it's being checked? Maybe you could try just doing

[self setNeedsDisplay: YES];

On a similar note, how did you manage to get gtk3 to build w the wayland backend, I keep on running into libEpoxy not having the EGL headers on mac. Is it possible to do with llvmpipe and mesa?

[RJVB]

On Thursday March 06 2025 13:14:27 Zaphh wrote: On a similar note, how did you manage to get gtk3 to build w the wayland backend, I keep on running into libEpoxy not having the EGL headers on mac. Is it possible to do with llvmpipe and mesa?

Yes, you need to build Mesa with EGL support, and then libepoxy idem. And then it's just a matter of telling the GTk3 build system that the wayland backend is possible, and adding #defines for the mouse buttons.

[RJVB]

After some more tinkering (and understanding I did something wrong at the GTk level) I can now confirm that button clicks do arrive in GTk, with the proper codes for BTN_LEFT and BTN_RIGHT (I don't have a BTN_MIDDLE defined). That's almost a pity because it means that they're being ignored for some reason which could be in Wayland or GTk. The issue with the submenus also applies to tooltips and thus probably to any (sub)window being opened.

[bugaevc]

I'm sorry that I don't have the time right now to look into this 😐. But, here's a patch that has been lying on my local machine uncommitted, perhaps it will help with something:

diff --git a/Sources/Compositor/OwlSurface.m b/Sources/Compositor/OwlSurface.m
index 0774a05..7ce425f 100644
--- a/Sources/Compositor/OwlSurface.m
+++ b/Sources/Compositor/OwlSurface.m
@@ -249,6 +249,19 @@ static void surface_set_buffer_transform_handler(
     // TODO
 }
 
+static void surface_damage_buffer_handler(
+    struct wl_client *client,
+    struct wl_resource *resource,
+    int x,
+    int y,
+    int width,
+    int height
+) {
+    OwlSurface *self = wl_resource_get_user_data(resource);
+    NSRect rect = NSMakeRect(x, y, width, height);
+    [self->_pendingState addDamage: rect];
+}
+
 static void surface_frame_handler(
     struct wl_client *client,
     struct wl_resource *resource,
@@ -275,7 +288,8 @@ static const struct wl_surface_interface surface_interface = {
     .set_opaque_region = surface_set_opaque_region_handler,
     .set_input_region = surface_set_input_region_handler,
     .set_buffer_scale = surface_set_buffer_scale_handler,
-    .set_buffer_transform = surface_set_buffer_transform_handler
+    .set_buffer_transform = surface_set_buffer_transform_handler,
+    .damage_buffer = surface_damage_buffer_handler
     // TODO
 };
 
diff --git a/Sources/Shell/OwlXdgToplevel.m b/Sources/Shell/OwlXdgToplevel.m
index ecd2f43..aa75912 100644
--- a/Sources/Shell/OwlXdgToplevel.m
+++ b/Sources/Shell/OwlXdgToplevel.m
@@ -118,6 +118,36 @@ static void xdg_toplevel_unset_maximized_handler(
     [self->_window unmaximize];
 }
 
+static void xdg_toplevel_set_fullscreen_handler(
+    struct wl_client *client,
+    struct wl_resource *resource,
+    struct wl_resource *output_resource
+) {
+    OwlXdgToplevel *self = wl_resource_get_user_data(resource);
+    // TODO: output_resource
+    NSWindow *window = [self->_window window];
+    if (self->_fullscreen) {
+        return;
+    }
+    if ([window respondsToSelector: @selector(toggleFullScreen:)]) {
+        [window toggleFullScreen: self];
+    }
+}
+
+static void xdg_toplevel_unset_fullscreen_handler(
+    struct wl_client *client,
+    struct wl_resource *resource
+) {
+    OwlXdgToplevel *self = wl_resource_get_user_data(resource);
+    NSWindow *window = [self->_window window];
+    if (!self->_fullscreen) {
+        return;
+    }
+    if ([window respondsToSelector: @selector(toggleFullScreen:)]) {
+        [window toggleFullScreen: self];
+    }
+}
+
 static const struct xdg_toplevel_interface xdg_toplevel_impl = {
     .destroy = xdg_toplevel_destroy_handler,
     .set_parent = xdg_toplevel_set_parent_handler,
@@ -128,7 +158,9 @@ static const struct xdg_toplevel_interface xdg_toplevel_impl = {
     .set_min_size = xdg_toplevel_set_min_size_handler,
     .set_minimized = xdg_toplevel_set_minimized_handler,
     .set_maximized = xdg_toplevel_set_maximized_handler,
-    .unset_maximized = xdg_toplevel_unset_maximized_handler
+    .unset_maximized = xdg_toplevel_unset_maximized_handler,
+    .set_fullscreen = xdg_toplevel_set_fullscreen_handler,
+    .unset_fullscreen = xdg_toplevel_unset_fullscreen_handler
     // TODO
 };
 
@@ -240,7 +272,7 @@ static const struct xdg_toplevel_interface xdg_toplevel_impl = {
     [self sendConfigureWithSize: s];
 }
 
-- (BOOL) windowShouldClose: (NSWindow *) w {
+- (BOOL) windowShouldClose: (id) sender {
     xdg_toplevel_send_close(_resource);
     [[OwlServer sharedServer] flushClientsLater];
     return NO;
@@ -261,6 +293,18 @@ static const struct xdg_toplevel_interface xdg_toplevel_impl = {
     [self sendConfigureWithSize: NSZeroSize];
 }
 
+- (void) windowDidEnterFullScreen: (NSNotification *)notification {
+    _fullscreen = YES;
+    // TODO: This should probably send the new size,
+    // unless windowDidResize: already does.
+    [self sendConfigureWithSize: NSZeroSize];
+}
+
+- (void) windowDidExitFullScreen: (NSNotification *)notification {
+    _fullscreen = YES;
+    [self sendConfigureWithSize: NSZeroSize];
+}
+
 - (void) windowDidBecomeKey: (NSNotification *) notification {
     [[self keyboard] sendEnterSurface: _surface];
     [[self dataDevice] focused];
diff --git a/Sources/Shell/OwlZxdgToplevelV6.m b/Sources/Shell/OwlZxdgToplevelV6.m
index 359b675..708d27c 100644
--- a/Sources/Shell/OwlZxdgToplevelV6.m
+++ b/Sources/Shell/OwlZxdgToplevelV6.m
@@ -240,7 +240,7 @@ static const struct zxdg_toplevel_v6_interface xdg_toplevel_v6_impl = {
     [self sendConfigureWithSize: s];
 }
 
-- (BOOL) windowShouldClose: (NSWindow *) w {
+- (BOOL) windowShouldClose: (id) sender {
     zxdg_toplevel_v6_send_close(_resource);
     [[OwlServer sharedServer] flushClientsLater];
     return NO;

[RJVB]

On Friday March 07 2025 07:57:34 Sergey Bugaev wrote: I'm sorry that I don't have the time right now to look into this 😐. But, here's a patch that has been lying on my local machine uncommitted, perhaps it will help with something:

Sorry to say that neither suggestion had any effect. I'm getting a sh'load of failed GDK_IS_MONITOR assertions (presumable because the GdkMonitor pointer is NULL). That might have something to do with the problem too?

[bugaevc]

That could certainly be an issue. We should make Owl advertise monitors (OwlOutput, OwlXdgOutput) based on NSScreen.

[RJVB]

That could certainly be an issue. We should make Owl advertise monitors (`OwlOutput`, `OwlXdgOutput`) based on `NSScreen`.

Looking forward to it! Looks a bit beyond my current skills but I can evidently help test it on a 2-monitor set-up. Could the lack of detecting a monitor/screen explain the NULL listener symptom when trying to open a sub/popup window?

[bugaevc]

No, that's xdg positioner, a more complex thing to implement. Should be very doable too though.

[RJVB]

On Friday March 07 2025 20:44:56 Sergey Bugaev wrote: That could certainly be an issue.

Lol, yes, it kinda prevents running Qt applications via waypipe ;) ``` Running on a compositor with no screens is not supported s1556: 0.355842 [mainloop.c:885] application has closed s1556: 0.355892 [mainloop.c:1464] Exiting main loop (0, 2, 2), attempting close message ```

[FlyGoat]

Re proper OpenGL (ES) support, we'll need to hook up an EGL platform implementation, it doesn't have to be mesa.

IMO the best option would be ANGLE as it comes with Metal translation support which is hardware accelerated. We will need to implement an OS support layer for Vulkan backend based on owl_iosurface_v1 though.

[RJVB]

On Wednesday March 12 2025 05:40:07 Jiaxun Yang wrote: Re proper OpenGL (ES) support, we'll need to hook up an EGL platform implementation, it doesn't have to be mesa.

Have at it, I'd say! Whatever solution is chosen, it needs to be able to accept EGL calls from Mesa's libEGL - unless you're suggesting to write an alternative implementation for use by non-X11 applications? In that case an additional solution will have to be found to make the library co-exist with Mesa's libEGL.dylib !

[FlyGoat]

On Wednesday March 12 2025 05:40:07 Jiaxun Yang wrote:
Re proper OpenGL (ES) support, we'll need to hook up an EGL platform implementation, it doesn't have to be mesa.
Have at it, I'd say! Whatever solution is chosen, it needs to be able to accept EGL calls from Mesa's libEGL - unless you're suggesting to write an alternative implementation for use by non-X11 applications? In that case an additional solution will have to be found to make the library co-exist with Mesa's libEGL.dylib !

Yes, I was thinking Wayland only applications. Most UI toolkits do support Wayland + GL ES those days.

For XWayland, if I'm understanding correctly, you don't need to link against EGL for GLX indirect rendering. XWayland's glamor can take advantage of GL ES for X composition.

XQuartz didn't go beyond GLX Indirect rendering either, since the whole DRI2 stack is not designed to work with AGL or IOSurface.

[RJVB]

On Wednesday March 12 2025 06:16:34 Jiaxun Yang wrote: Yes, I was thinking Wayland only applications. Most UI toolkits do support Wayland + GL ES those days.

It'll still have to be co-installable with Mesa's libEGL - for X11 applications that require it.

For XWayland, if I'm understanding correctly, you don't need to link against EGL for GLX indirect rendering. XWayland's glamor can take advantage of GL ES for X composition.

No, that shouldn't be necessary. XWayland is just a simplified XOrg that doesn't do all the platform-specific stuff itself but instead issues Wayland commands. Not unlike how XQuartz hooks into Cocoa/Quartz/and/whatever APIs.

XQuartz didn't go beyond GLX Indirect rendering either, since the whole DRI2 stack is not designed to work with AGL or IOSurface.

No. GLX direct rendering does work though (locally), and remote EGL also works using Mesa Gallium (llvmpipe and/or swrast). I've been trying to get Mesa on Mac to do Wayland but so far I've failed. I don't know if that's because I tried it from GTk3 apps which access GL or EGL through libepoxy (and thus somehow require a valid X11 connection) or because of some other reason.