Server: Don't disable Wayland; Use Display :1024

dcommander · dcommander · commit b0552dcf4ced · 2025-10-07T14:01:17.000-04:00
On modern Linux distributions, GDM runs Xwayland on Display :1024 if WaylandEnable=true (the default), and it runs Xorg on Display :0 if WaylandEnable=false. Referring to aa5ff87 and #94, in 2019 I observed that it was necessary on RHEL 8 and Fedora 27-28 to add/uncomment WaylandEnable=false in order for VirtualGL's greeter autostart script to work. However, that is apparently no longer necessary. In 2025, using Display :1024 seems to work fine on RHEL 7-10, Ubuntu 16-24, Fedora 40-42, and openSUSE 15. (In fact, it is necessary on RHEL 10, since GDM no longer allows Wayland to be disabled. That is what prompted this change.)
diff --git a/ChangeLog.md b/ChangeLog.md
@@ -12,6 +12,12 @@ in the VirtualGL User's Guide for more information.
 enabling quad-buffered stereo, even if the requirements for quad-buffered
 stereo had been met.
 
+3. On Wayland-enabled Linux systems running GDM, `vglserver_config` no longer
+disables the ability to log in locally with a Wayland session.  Instead,
+`vglrun` automatically sets `VGL_DISPLAY` to `:1024` if `VGL_DISPLAY` is unset,
+X Display :1024 exists, and X Display :0 does not exist.  This allows VirtualGL
+to use the GDM greeter's Xwayland instance as a 3D X server.
+
 
 3.1.3
 =====
diff --git a/doc/advancedconfig.txt b/doc/advancedconfig.txt
@@ -119,21 +119,26 @@ previous method:
 | Summary | __''{d}''__ = the X display/screen or EGL device to use for 3D \
 	rendering |
 | Image Transports | All |
-| Default Value | '':0'' |
+| Default Value | '':0'' or '':1024'' |
 #OPT: hiCol=first
 
 	Description :: If the VirtualGL server has multiple GPUs, then you can use
 	this option to specify which GPU should be used for 3D rendering.  For
 	instance, setting ''VGL_DISPLAY'' to (or invoking ''vglrun -d'' with)
 	'':0.1'' would cause VirtualGL to use the GLX back end and redirect all of
 	the OpenGL rendering from the 3D application to a GPU attached to Screen 1 on
-	X display :0.  Setting ''VGL_DISPLAY'' to (or invoking ''vglrun -d'' with)
+	X Display :0.  Setting ''VGL_DISPLAY'' to (or invoking ''vglrun -d'' with)
 	a DRI device path (such as ''/dev/dri/card0'') or an EGL device ID (such as
 	''egl0'') would cause VirtualGL to use the EGL back end and redirect all of
 	the OpenGL rendering from the 3D application to the specified EGL device.
 	''/opt/VirtualGL/bin/eglinfo -e'' lists all valid EGL device IDs and their
 	associated DRI device paths.
 
+	!!! ''vglrun'' automatically sets ''VGL_DISPLAY'' to '':1024'' if
+	''VGL_DISPLAY'' is unset, X Display :1024 exists, and X Display :0 does not
+	exist.  This allows VirtualGL to use the GDM greeter's Xwayland instance as a
+	3D X server.
+
 | Environment Variable | {pcode: VGL_EGLLIB = __{l}__ } |
 | Summary | __''{l}''__ = the location of an alternate EGL library |
 | Image Transports | All |
diff --git a/doc/index.html b/doc/index.html
@@ -3,7 +3,7 @@
 <head>
 <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
 <meta name="language" content="en">
-<meta name="date" content="2025-07-01T12:15:57">
+<meta name="date" content="2025-10-07T13:56:21">
 <meta name="generator" content="deplate.rb 0.8.5">
 <title>User&rsquo;s Guide for VirtualGL 3.1.4</title>
 <link rel="start" href="index.html" title="Frontpage">
@@ -921,10 +921,6 @@ <h2 id="hd006001">6.1&nbsp;Initial System Configuration</h2>
 and 3D X server at boot time.  This is the default with most modern 
 Linux and Unix distributions.</p>
 
-<div class="important"><p class="important">
-On Wayland-enabled Linux machines running GDM, configuring the machine for use with the GLX back end will disable the ability to log in locally with a Wayland session.  In general, logging in locally once the machine has been configured for use with the GLX back end is discouraged, as this could disrupt VirtualGL&rsquo;s connection to the 3D X server and thus cause OpenGL applications running with VirtualGL to abort or freeze.
-</p></div>
-
 <p>3. Accelerated OpenGL is working properly in the 3D X server.</p>
 
 <ul class="Itemize">
@@ -969,8 +965,10 @@ <h2 id="hd006002">6.2&nbsp;GLX Back End: Granting Access to the 3D X Server</h2>
 need to use the GLX back end. Unless you fully trust the users to whom 
 you are granting this access, then you should avoid logging in locally 
 to the 3D X server (particularly as root) unless absolutely necessary.  
-(Logging in locally to the 3D X server is discouraged anyhow, for 
-reasons explained in the previous section.)</p>
+(In general, logging in locally once the machine has been configured for 
+use with the GLX back end is discouraged, as this could disrupt 
+VirtualGL&rsquo;s connection to the 3D X server and thus cause OpenGL 
+applications running with VirtualGL to abort or freeze.)</p>
 
 <p>This section will explain how to configure a VirtualGL server such that 
 selected users can use the GLX back end, even if the server is sitting 
@@ -1166,27 +1164,26 @@ <h3 id="hd006002001">6.2.1&nbsp;Sanity Check</h3>
 <dl class="Description">
     <dt class="Description-1 Description">If you restricted 3D X server access to <code>vglusers</code></dt>
     <dd class="Description-1 Description">
-<pre class="verbatim">
-xauth&nbsp;merge&nbsp;/etc/opt/VirtualGL/vgl_xauth_key
-xdpyinfo&nbsp;-display&nbsp;:0
-/opt/VirtualGL/bin/glxinfo&nbsp;-display&nbsp;:0&nbsp;-c
-</pre>
-
     </dd>
+</dl>
+
+<pre class="verbatim">xauth merge /etc/opt/VirtualGL/vgl_xauth_key
+xdpyinfo -display :<em>n</em>
+/opt/VirtualGL/bin/glxinfo -display :<em>n</em> -c</pre>
+<dl class="Description">
     <dt class="Description-1 Description">If you did not restrict 3D X server access</dt>
     <dd class="Description-1 Description">
-<pre class="verbatim">
-xdpyinfo&nbsp;-display&nbsp;:0
-/opt/VirtualGL/bin/glxinfo&nbsp;-display&nbsp;:0&nbsp;-c
-</pre>
-
     </dd>
 </dl>
 
-<p>Both commands should output a list of visuals and should complete with 
-no errors.  If you chose to disable the <code>XTEST</code> extension, 
-then check the output of the <code>xdpyinfo</code> command to verify 
-that <code>XTEST</code> does not show up in the list of extensions.</p>
+<pre class="verbatim">xdpyinfo -display :<em>n</em>
+/opt/VirtualGL/bin/glxinfo -display :<em>n</em> -c</pre>
+<p>(where <em><code>n</code></em> is the display number of the 3D X 
+server.)  Both commands should output a list of visuals and should 
+complete with no errors.  If you chose to disable the <code>XTEST</code> 
+extension, then check the output of the <code>xdpyinfo</code> command to 
+verify that <code>XTEST</code> does not show up in the list of 
+extensions.</p>
 
 <p>You should also examine the output of the <code>glxinfo</code> command 
 to ensure that at least one of the visuals is 24-bit or 32-bit TrueColor 
@@ -3103,7 +3100,7 @@ <h2 id="hd0018001">18.1&nbsp;Faker Settings</h2>
   </tr>
   <tr class="standard">
     <td class="high standard">Default Value</td>
-    <td class="standard"><code>:0</code></td>
+    <td class="standard"><code>:0</code> or <code>:1024</code></td>
   </tr>
 </table>
 </div>
@@ -3118,13 +3115,16 @@ <h2 id="hd0018001">18.1&nbsp;Faker Settings</h2>
         <code>vglrun&nbsp;-d</code> with) <code>:0.1</code> would cause 
         VirtualGL to use the GLX back end and redirect all of the OpenGL 
         rendering from the 3D application to a GPU attached to Screen 1 on X 
-        display :0.  Setting <code>VGL_DISPLAY</code> to (or invoking 
+        Display :0.  Setting <code>VGL_DISPLAY</code> to (or invoking 
         <code>vglrun&nbsp;-d</code> with) a DRI device path (such as 
         <code>/dev/dri/card0</code>) or an EGL device ID (such as 
         <code>egl0</code>) would cause VirtualGL to use the EGL back end and 
         redirect all of the OpenGL rendering from the 3D application to the 
         specified EGL device. <code>/opt/VirtualGL/bin/eglinfo&nbsp;-e</code> 
         lists all valid EGL device IDs and their associated DRI device paths.
+        <div class="important"><p class="important">
+        <code>vglrun</code> automatically sets <code>VGL_DISPLAY</code> to <code>:1024</code> if <code>VGL_DISPLAY</code> is unset, X Display :1024 exists, and X Display :0 does not exist.  This allows VirtualGL to use the GDM greeter&rsquo;s Xwayland instance as a 3D X server.
+        </p></div>
     </dd>
 </dl>
 
diff --git a/doc/unixconfig.txt b/doc/unixconfig.txt
@@ -21,13 +21,6 @@ LightDM) installed and has been configured to start the display manager and 3D
 X server at boot time.  This is the default with most modern Linux and Unix
 distributions.
 
-	!!! On Wayland-enabled Linux machines running GDM, configuring the machine
-	for use with the GLX back end will disable the ability to log in locally with
-	a Wayland session.  In general, logging in locally once the machine has been
-	configured for use with the GLX back end is discouraged, as this could
-	disrupt VirtualGL's connection to the 3D X server and thus cause OpenGL
-	applications running with VirtualGL to abort or freeze.
-
 3. Accelerated OpenGL is working properly in the 3D X server.
 	* If the 3D X server is not headless, then this can be verified by logging
 	into the X server locally and running ''glxinfo''.
@@ -60,8 +53,10 @@ disallow access to the GPU.  Thus, it is necessary to grant full, trusted
 access to the 3D X server for any users that will need to use the GLX back end.
 Unless you fully trust the users to whom you are granting this access, then
 you should avoid logging in locally to the 3D X server (particularly as root)
-unless absolutely necessary.  (Logging in locally to the 3D X server is
-discouraged anyhow, for reasons explained in the previous section.)
+unless absolutely necessary.  (In general, logging in locally once the machine
+has been configured for use with the GLX back end is discouraged, as this could
+disrupt VirtualGL's connection to the 3D X server and thus cause OpenGL
+applications running with VirtualGL to abort or freeze.)
 
 This section will explain how to configure a VirtualGL server such that
 selected users can use the GLX back end, even if the server is sitting at the
@@ -206,22 +201,23 @@ end, log out of the server, log back into the server using SSH, and execute the
 following commands in the SSH session:
 
 	If you restricted 3D X server access to ''vglusers'' :: {:}
-	#Verb <<---
-	xauth merge /etc/opt/VirtualGL/vgl_xauth_key
-	xdpyinfo -display :0
-	/opt/VirtualGL/bin/glxinfo -display :0 -c
-	---
+#Pverb <<---
+xauth merge /etc/opt/VirtualGL/vgl_xauth_key
+xdpyinfo -display :__n__
+/opt/VirtualGL/bin/glxinfo -display :__n__ -c
+---
 
 	If you did not restrict 3D X server access :: {:}
-	#Verb <<---
-	xdpyinfo -display :0
-	/opt/VirtualGL/bin/glxinfo -display :0 -c
-	---
-
-Both commands should output a list of visuals and should complete with no
-errors.  If you chose to disable the ''XTEST'' extension, then check the output
-of the ''xdpyinfo'' command to verify that ''XTEST'' does not show up in the
-list of extensions.
+#Pverb <<---
+xdpyinfo -display :__n__
+/opt/VirtualGL/bin/glxinfo -display :__n__ -c
+---
+
+(where __''n''__ is the display number of the 3D X server.)  Both commands
+should output a list of visuals and should complete with no errors.  If you
+chose to disable the ''XTEST'' extension, then check the output of the
+''xdpyinfo'' command to verify that ''XTEST'' does not show up in the list of
+extensions.
 
 You should also examine the output of the ''glxinfo'' command to ensure that at
 least one of the visuals is 24-bit or 32-bit TrueColor and has Pbuffer support.
diff --git a/server/vglrun.in b/server/vglrun.in
@@ -2,7 +2,7 @@
 
 # Copyright (C)2004 Landmark Graphics Corporation
 # Copyright (C)2005 Sun Microsystems, Inc.
-# Copyright (C)2009-2016, 2018-2019, 2021, 2024 D. R. Commander
+# Copyright (C)2009-2016, 2018-2019, 2021, 2024-2025 D. R. Commander
 #
 # This library is free software and may be redistributed and/or modified under
 # the terms of the wxWindows Library License, Version 3.1 or (at your option)
@@ -41,7 +41,7 @@ usage()
 	echo "            library."
 	echo
 	echo "-d <d>    : <d> = the X display/screen or EGL device to use for 3D rendering"
-	echo "           [default = :0]"
+	echo "           [default = :0 or :1024]"
 	echo
 	echo "-fps <f>  : Limit image transport frame rate to <f> frames/sec"
 	echo
@@ -193,12 +193,20 @@ else
 		if [ ! -z "$SSH_IP" -a -z "$VGL_CLIENT" -a -z "$RRCLIENT" ]; then
 			VGL_CLIENT=$SSH_IP
 			export VGL_CLIENT
-			echo "[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to"
+			echo "[VGL] NOTICE: Automatically setting the VGL_CLIENT environment variable to"
 			echo "[VGL]    $SSH_IP, the IP address of your SSH client."
 		fi
 	fi
 fi
 
+if [ -z "$VGL_DISPLAY" -a ! -e /tmp/.X11-unix/X0 -a \
+	-e /tmp/.X11-unix/X1024 ]; then
+	echo "[VGL] NOTICE: Automatically setting the VGL_DISPLAY environment variable to"
+	echo "[VGL]    :1024, the X display number of the GDM greeter's Xwayland instance."
+	VGL_DISPLAY=:1024
+	export VGL_DISPLAY
+fi
+
 # This is so we can launch both 32-bit and 64-bit executables using the same
 # script.
 if [ -x "$MYPATH/.vglrun.vars64" ]; then
diff --git a/server/vglserver_config b/server/vglserver_config
@@ -20,7 +20,6 @@ RMMODNEEDED=0
 NVIDIADRMNEEDED=0
 UNATTENDED=0
 SELINUX=0
-WAYLAND=0
 BACKEND=glx
 
 GID=
@@ -281,22 +280,6 @@ disallowgdmtcp()
 	commentline $1 "DisallowTCP.*=.*"
 }
 
-disablewaylandgdm()
-{
-	backup $1
-	echo ... Disabling Wayland in $1 ...
-	grep -q "WaylandEnable.*=.*false" $1 && (
-		uncommentline $1 "WaylandEnable.*=.*false"
-	) || (
-		grep -q "WaylandEnable" $1 && (
-			replaceline $1 WaylandEnable "WaylandEnable=false"
-		) || (
-			removeline $1 WaylandEnable
-			addtoconf $1 "daemon" "daemon" WaylandEnable=false
-		)
-	)
-}
-
 setdripermissions()
 {
 	if [ -z "$1" ]; then
@@ -579,37 +562,6 @@ checkselinux()
 	fi
 }
 
-checkwayland()
-{
-	which Xwayland >/dev/null 2>&1 || return 0
-	if [ -f /etc/systemd/system/display-manager.service ]; then
-		DM=`grep ExecStart=\/usr /etc/systemd/system/display-manager.service | sed 's/.*\///g'`
-		if [ ! "$DM" = "gdm" -a ! "$DM" = "gdm3" ]; then
-			return 0
-		fi
-	elif [ -f /etc/alternatives/default-displaymanager ]; then
-		DM=`grep DISPLAYMANAGER=\/usr /etc/alternatives/default-displaymanager | sed 's/.*\///g'`
-		if [ ! "$DM" = "gdm" ]; then
-			return 0
-		fi
-	else
-		return 0
-	fi
-	WAYLAND=1
-	echo
-	echo WARNING: Configuring this server for use with VirtualGL will disable the
-	echo ability to log in locally with a Wayland session.
-	if [ $UNATTENDED = 0 ]; then
-		ynprompt "Continue?"
-		if [ ! "$?" = "1" ]; then
-			return 1
-		fi
-	else
-		echo
-	fi
-	return 0
-}
-
 unconfigglx()
 {
 	if [ -d /etc/opt/VirtualGL ]; then
@@ -793,8 +745,6 @@ configegl()
 
 configglx()
 {
-	checkwayland || return
-
 	if [ $UNATTENDED = 0 ]; then
 		ynprompt "Restrict 3D X server access to vglusers group (recommended)?"
 		if [ $? = 1 ]; then VGLUSERSONLY=1; else VGLUSERSONLY=0; fi
@@ -916,8 +866,8 @@ EOF
 		/etc/X11/gdm/custom.conf \
 		/etc/gdm3/custom.conf \
 		/etc/gdm3/daemon.conf; do
-		if [ -f $file -a ! -h $file -a $WAYLAND = 1 ]; then
-			disablewaylandgdm $file
+		if [ -f $file -a ! -h $file ]; then
+			commentline $file "WaylandEnable.*=.*false"
 		fi
 	done
 
