Builded on macOS Monterey and can't connect to it

[gbogomolov001]

Hello.
Colud you please advice me. For the last 2 weeks I'm trying to access macOS from Linux using RDP protocol, FreeRDP/Remmina RDP client on the Linux side and neutrinolabs tools on the macOS side. Dont ask me why :( just need to do that for some security reasons.
There are a lot of errors on macOS side, and they are different each time. So I decided to start from the beginnigg, from the point that maybe I should figure out firstable which version of xrdp/xorgxrdp tools is fully applicable to my macOS version.
macOS Monterey 12.7.4 (can install 12.2.1 or 12.7.6 depends on xrdp/xorgxrdp support). Updating to newer macOS version is not applicable.

P.S. If there is sombody here who has been able to install and configure this xrdp tools on macOS successfully, and it is really working - please contact me.

[gbogomolov001]

Dear @matt335672 please help. 3 weeks have passed since I'm trying to access my macOS via xrdp tool.
I've read tons of different topics, articles and other google-related stuff.
Got an error:

[20250714-23:09:08] [ERROR] libxrdp_force_read: bad header length 0
[20250714-23:09:08] [INFO ] connected ok
[20250714-23:09:08] [ERROR] [ITU-T X.224] Connection Sequence: CR-TPDU (Connection Request) failed
[20250714-23:09:08] [DEBUG] status from xrdp_mm_connect() : 0
[20250714-23:09:08] [ERROR] xrdp_sec_incoming: xrdp_iso_incoming failed
[20250714-23:09:08] [DEBUG] Login state change request WMLS_CONNECT_IN_PROGRESS -> WMLS_CLEANUP
[20250714-23:09:08] [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed
[20250714-23:09:08] [DEBUG] xrdp_wm_login_mode_changed: login_mode is 4
[20250714-23:09:08] [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed
[20250714-23:09:08] [DEBUG] Login state change request WMLS_CLEANUP -> WMLS_INACTIVE
[20250714-23:09:08] [DEBUG] Closed socket 11 (AF_INET 127.0.0.1:3389)
[20250714-23:09:08] [ERROR] lib_data_in: bad size
[20250714-23:09:08] [DEBUG] Closed socket 11 (AF_INET 192.168.0.17:3389)
[20250714-23:09:08] [DEBUG] xrdp_mm_module_cleanup
[20250714-23:09:08] [DEBUG] Closed socket 17 (AF_INET 127.0.0.1:49153)

Hardware/OS/soft configs.

Target machines:
macOS 12.6 Monterey, MacBook Pro 16 M1 2021 (current log above is from this mac)
macOS 12.7.6 Monterey, MacBook Air 13 Intel 2018
one more Air 2018 on Intel and Mac mini 2018 on Intel too. All of it has different macOS Monterey releases, from 12.2.1 to 12.7.6.

RDP-client machines:
Windows 10 with default mstsc
Also have Linux machine with FreeRDP client, KRDC client, Remmina client.

xrdp version:

xrdp 0.9.19
A Remote Desktop Protocol Server.
Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
See https://github.com/neutrinolabs/xrdp for more information.

Configure options:
PKG_CONFIG_PATH=/opt/homebrew/opt/openssl/lib/pkgconfig
CC=/usr/bin/clang
CXX=/usr/bin/clang++
--host=arm-linux
--with-cflags=-mfpu=neon
host_alias=arm-linux
CPPFLAGS=-I/usr/local/opt/openssl/include

Compiled with OpenSSL 3.5.1 1 Jul 2025

xrdp-sesman version:

xrdp-sesman 0.9.19
The xrdp session manager
Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
See https://github.com/neutrinolabs/xrdp for more information.

Configure options:
PKG_CONFIG_PATH=/opt/homebrew/opt/openssl/lib/pkgconfig
CC=/usr/bin/clang
CXX=/usr/bin/clang++
--host=arm-linux
--with-cflags=-mfpu=neon
host_alias=arm-linux
CPPFLAGS=-I/usr/local/opt/openssl/include

In fact, it is XorgxRDP 0.2.18 and xRDP 0.9.19 from this topic started by @metalefty
Why did I choose this version? Cause I need to run this exactly on macOS Monterey, and the mentioned post tells us (as I understand from the edit history, @tenpaiyomi successfully connected):

[Update: April 2022] With regards to the above, using the commands to compile and build, I was successfully able to RDP into a Mac laptop from a Windows machine while getting the proper screen, without having to log out. YMMV.

So, I tried to repeat all the steps described several times. All the steps below done on MacBook Pro M1, but applies for Apple Silicon and Intel both (maybe some config files may have different paths but nothing more seriuos - tried this on Intel too but without writing down the details).
At the stage sudo make install LIBTOOL="glibtool" for xrdp - got an error:

make[3]: Nothing to be done for `install-exec-am'.
mkdir -p '/etc/xrdp'
/usr/bin/install -c -m 644 km-00000406.ini km-00000407.ini km-00000409.ini km-0000040a.ini km-0000040b.ini km-0000040c.ini km-00000410.ini km-00000411.ini km-00000412.ini km-00000414.ini km-00000415.ini km-00000416.ini km-00000419.ini km-0000041d.ini km-00000807.ini km-00000809.ini km-0000080a.ini km-0000080c.ini km-00000813.ini km-00000816.ini km-0000100c.ini km-00010409.ini km-19360409.ini '/etc/xrdp'
/Applications/Xcode.app/Contents/Developer/usr/bin/make install-data-hook
if [ -f /etc/init.d/xrdp ]; then
sed -i 's|BASE|/usr/local|' /etc/init.d/xrdp;
fi
sed: 1: "/etc/init.d/xrdp": command i expects \ followed by text
make[4]: *** [install-data-hook] Error 1
make[3]: *** [install-data-am] Error 2
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1

Solved by editing the line sed -i 's to sed -i ‘’ -e ’s in /Users/user/Documents/xrdp-0.9.19/instfiles/makefile
To do that, u need to copy makefile somewhere like desktop, edit it, and do logout-login, cause if u will try to replace it and do make again immediately after editing the file - the system will report you that the file is busy. So, u have to do logout-login, then replace the original file with a new one and do sudo make install LIBTOOL="glibtool" again.

Also, before the stage sudo make install for xorgxrdp, I had to do make CFLAGS="-I/opt/X11/include -I/opt/X11/include/X11" again, cause after logout-login all variables were lost.
And one more thing before starting xdrp and xrdp-sesman. To get the result, described in mentioned topic:

❯ which xrdp
/usr/local/sbin/xrdp

I had to do the folowing:
Open up Terminal.
Run the following command: sudo nano /etc/paths
Enter your password, when prompted.
Go to the bottom of the file, and add the path /usr/local/sbin
Hit control-x to quit.
Enter “Y” to save the modified buffer.
To test it, in new terminal window, type: echo $PATH

Also, in macOS Sharing settings enabled Remote Login and Remote Management for all users, the last is configured as "All may request permission to control screen" and "VNC viewers may control screen with password" as suggested here:

Enable VNC aka ARD
And now don't forget to start VNC to which your XRDP instance will be connected.
You can do that using GUI settings of mac.
Or simply run this to activate access with the password 12345678 (you can change it):
Code:
$ /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -clientopts -setvnclegacy -vnclegacy yes -clientopts -setvncpw -vncpw 12345678 -restart -agent -privs -all

So VNC will be enabled at port 5900 (you can try some custom vnc to change it if you want).

I don't quite understand why this guy recommended this, unless xrdp works as a proxy to macOS built-in VNC server. Anyway, configured as advised.

OOOOKAY, finally all of this were installed and seems to be workable.
Trying to connect - get a "turquoise" xrdp login window, select xorg connection type, then entered my mac username and password (this user is an admin) and then... A black screen flashes for a millisecond and the log shows tons of errors mentioned at the beginning of this topic. Then I started googling and it took me a days and nights of research but I'm still at the same point.

But for someone everything works, so what am I doing wrong?

Here is the log files:

xrdp-sesman.log
xrdp.log

Here is xrdp.ini:

Custom experimental changes:
security_layer=negotiate - tried to set all of it - tls and rdp and negotiate
crypt_level=high - tried all types
certificate=/etc/xrdp/cert.pem - decided to specify the path instead of default empty
key_file=/etc/xrdp/key.pem - decided to specify the path instead of default empty
ssl_protocols=TLSv1.2, TLSv1.3 - tried tls 1.2 only
max_bpp=16 - I found here that there may be some coloring problems when connecting from Windows mstsc using default 36 value

[Globals]
; xrdp.ini file version number
ini_version=1

; fork a new process for each incoming connection
fork=true

; ports to listen on, number alone means listen on all interfaces
; 0.0.0.0 or :: if ipv6 is configured
; space between multiple occurrences
; ALL specified interfaces must be UP when xrdp starts, otherwise xrdp will fail to start
;
; Examples:
;   port=3389
;   port=unix://./tmp/xrdp.socket
;   port=tcp://.:3389                           127.0.0.1:3389
;   port=tcp://:3389                            *:3389
;   port=tcp://<any ipv4 format addr>:3389      192.168.1.1:3389
;   port=tcp6://.:3389                          ::1:3389
;   port=tcp6://:3389                           *:3389
;   port=tcp6://{<any ipv6 format addr>}:3389   {FC00:0:0:0:0:0:0:1}:3389
;   port=vsock://<cid>:<port>
port=3389

; 'port' above should be connected to with vsock instead of tcp
; use this only with number alone in port above
; prefer use vsock://<cid>:<port> above
use_vsock=false

; regulate if the listening socket use socket option tcp_nodelay
; no buffering will be performed in the TCP stack
tcp_nodelay=true

; regulate if the listening socket use socket option keepalive
; if the network connection disappear without close messages the connection will be closed
tcp_keepalive=true

; set tcp send/recv buffer (for experts)
#tcp_send_buffer_bytes=32768
#tcp_recv_buffer_bytes=32768

; security layer can be 'tls', 'rdp' or 'negotiate'
; for client compatible layer
security_layer=negotiate

; minimum security level allowed for client for classic RDP encryption
; use tls_ciphers to configure TLS encryption
; can be 'none', 'low', 'medium', 'high', 'fips'
crypt_level=high

; X.509 certificate and private key
; openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 365
certificate=/etc/xrdp/cert.pem
key_file=/etc/xrdp/key.pem

; set SSL protocols
; can be comma separated list of 'SSLv3', 'TLSv1', 'TLSv1.1', 'TLSv1.2', 'TLSv1.3'
ssl_protocols=TLSv1.2, TLSv1.3
; set TLS cipher suites
#tls_ciphers=HIGH

; concats the domain name to the user if set for authentication with the separator
; for example when the server is multi homed with SSSd
#domain_user_separator=@

; The following options will override the keyboard layout settings.
; These options are for DEBUG and are not recommended for regular use.
#xrdp.override_keyboard_type=0x04
#xrdp.override_keyboard_subtype=0x01
#xrdp.override_keylayout=0x00000409

; Section name to use for automatic login if the client sends username
; and password. If empty, the domain name sent by the client is used.
; If empty and no domain name is given, the first suitable section in
; this file will be used.
autorun=

allow_channels=true
allow_multimon=true
bitmap_cache=true
bitmap_compression=true
bulk_compression=true
#hidelogwindow=true
max_bpp=16
new_cursors=true
; fastpath - can be 'input', 'output', 'both', 'none'
use_fastpath=both
; when true, userid/password *must* be passed on cmd line
#require_credentials=true
; when true, the userid will be used to try to authenticate
#enable_token_login=true
; You can set the PAM error text in a gateway setup (MAX 256 chars)
#pamerrortxt=change your password according to policy at http://url

;
; colors used by windows in RGB format
;
blue=009cb5
grey=dedede
#black=000000
#dark_grey=808080
#blue=08246b
#dark_blue=08246b
#white=ffffff
#red=ff0000
#green=00ff00
#background=626c72

;
; configure login screen
;

; Login Screen Window Title
#ls_title=My Login Title

; top level window background color in RGB format
ls_top_window_bg_color=009cb5

; width and height of login screen
;
; The default height allows for about 5 fields to be comfortably displayed
; above the buttons at the bottom. To display more fields, make <ls_height>
; larger, and also increase <ls_btn_ok_y_pos> and <ls_btn_cancel_y_pos>
; below
;
ls_width=350
ls_height=430

; login screen background color in RGB format
ls_bg_color=dedede

; optional background image filename. BMP format is always supported,
; but other formats will be supported if xrdp is build with imlib2
; The transform can be one of the following:-
;     none  : No transformation. Image is placed in bottom-right corner
;             of the screen.
;     scale : Image is scaled to the screen size. The image aspect
;             ratio is not preserved.
;     zoom  : Image is scaled to the screen size. The image aspect
;             ratio is preserved by clipping the image.
#ls_background_image=
#ls_background_transform=none

; logo
; full path to file or file in shared folder. BMP format is always supported,
; but other formats will be supported if xrdp is build with imlib2
; For transform values, see 'ls_background_transform'. The logo width and
; logo height are ignored for a transform of 'none'.
ls_logo_filename=
#ls_logo_transform=none
#ls_logo_width=240
#ls_logo_height=140
ls_logo_x_pos=55
ls_logo_y_pos=50

; for positioning labels such as username, password etc
ls_label_x_pos=30
ls_label_width=65

; for positioning text and combo boxes next to above labels
ls_input_x_pos=110
ls_input_width=210

; y pos for first label and combo box
ls_input_y_pos=220

; OK button
ls_btn_ok_x_pos=142
ls_btn_ok_y_pos=370
ls_btn_ok_width=85
ls_btn_ok_height=30

; Cancel button
ls_btn_cancel_x_pos=237
ls_btn_cancel_y_pos=370
ls_btn_cancel_width=85
ls_btn_cancel_height=30

[Logging]
; Note: Log levels can be any of: core, error, warning, info, debug, or trace
LogFile=xrdp.log
LogLevel=DEBUG
EnableSyslog=true
#SyslogLevel=INFO
#EnableConsole=false
#ConsoleLevel=INFO
#EnableProcessId=false

[LoggingPerLogger]
; Note: per logger configuration is only used if xrdp is built with
; --enable-devel-logging
#xrdp.c=INFO
#main()=INFO

[Channels]
; Channel names not listed here will be blocked by XRDP.
; You can block any channel by setting its value to false.
; IMPORTANT! All channels are not supported in all use
; cases even if you set all values to true.
; You can override these settings on each session type
; These settings are only used if allow_channels=true
rdpdr=true
rdpsnd=true
drdynvc=true
cliprdr=true
rail=true
xrdpvr=true
tcutils=true

; for debugging xrdp, in section xrdp1, change port=-1 to this:
#port=/tmp/.xrdp/xrdp_display_10


;
; Session types
;

; Some session types such as Xorg, X11rdp and Xvnc start a display server.
; Startup command-line parameters for the display server are configured
; in sesman.ini. See and configure also sesman.ini.
[Xorg]
name=Xorg
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=3389
code=20

[Xvnc]
name=Xvnc
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=-1
#xserverbpp=24
#delay_ms=2000
; Disable requested encodings to support buggy VNC servers
; (1 = ExtendedDesktopSize)
#disabled_encodings_mask=0
; Use this to connect to a chansrv instance created outside of sesman
; (e.g. as part of an x11vnc console session). Replace '0' with the
; display number of the session
#chansrvport=DISPLAY(0)

; Generic VNC Proxy
; Tailor this to specific hosts and VNC instances by specifying an ip
; and port and setting a suitable name.
[vnc-any]
name=vnc-any
lib=libvnc.so
ip=ask
port=ask5900
username=na
password=ask
#pamusername=asksame
#pampassword=asksame
#pamsessionmng=127.0.0.1
#delay_ms=2000

; Generic RDP proxy using NeutrinoRDP
; Tailor this to specific hosts by specifying an ip and port and setting
; a suitable name.
[neutrinordp-any]
#name=neutrinordp-any
; To use this section, you should build xrdp with configure option
; --enable-neutrinordp.
##lib=libxrdpneutrinordp.so
#ip=ask
#port=ask3389
#username=ask
#password=ask
; Uncomment the following lines to enable PAM authentication for proxy
; connections.
#pamusername=ask
#pampassword=ask
#pamsessionmng=127.0.0.1
; Currently NeutrinoRDP doesn't support dynamic resizing. Uncomment
; this line if you're using a client which does.
#enable_dynamic_resizing=false
; By default, performance settings requested by the RDP client are ignored
; and chosen by NeutrinoRDP. Uncomment this line to allow the user to
; select performance settings in the RDP client.
#perf.allow_client_experiencesettings=true
; Override any experience setting by uncommenting one or more of the
; following lines.
#perf.wallpaper=false
#perf.font_smoothing=false
#perf.desktop_composition=false
#perf.full_window_drag=false
#perf.menu_anims=false
#perf.themes=false
#perf.cursor_blink=false
; By default NeutrinoRDP supports cursor shadows. If this is giving
; you problems (e.g. cursor is a black rectangle) try disabling cursor
; shadows by uncommenting the following line.
#perf.cursor_shadow=false
; By default, NeutrinoRDP uses the keyboard layout of the remote RDP Server.
; If you want to tell the remote the keyboard layout of the RDP Client,
; by uncommenting the following line.
#neutrinordp.allow_client_keyboardLayout=true
; The following options will override the remote keyboard layout settings.
; These options are for DEBUG and are not recommended for regular use.
#neutrinordp.override_keyboardLayout_mask=0x0000FFFF
#neutrinordp.override_kbd_type=0x04
#neutrinordp.override_kbd_subtype=0x01
#neutrinordp.override_kbd_fn_keys=12
#neutrinordp.override_kbd_layout=0x00000409

; You can override the common channel settings for each session type
#channel.rdpdr=true
#channel.rdpsnd=true
#channel.drdynvc=true
#channel.cliprdr=true
#channel.rail=true
#channel.xrdpvr=true

Here is sesman.ini:

Custom experimental changes:
ListenAddress=0.0.0.0 - set as an experiment following #2336 - no effect
param=/opt/X11/bin/xorg - the default param=Xorg as I understand equals to param=/usr/bin/Xorg and this path does not exist on macOS, so I found where it is and specified the actual path

;; See `man 5 sesman.ini` for details

[Globals]
ListenAddress=0.0.0.0
ListenPort=3350
EnableUserWindowManager=true
; Give in relative path to user's home directory
UserWindowManager=startwm.sh
; Give in full path or relative path to /etc/xrdp
DefaultWindowManager=startwm.sh
; Give in full path or relative path to /etc/xrdp
ReconnectScript=reconnectwm.sh

[Security]
AllowRootLogin=true
MaxLoginRetry=4
TerminalServerUsers=tsusers
TerminalServerAdmins=tsadmins
; When AlwaysGroupCheck=false access will be permitted
; if the group TerminalServerUsers is not defined.
AlwaysGroupCheck=false
; When RestrictOutboundClipboard=all clipboard from the
; server is not pushed to the client.
; In addition, you can control text/file/image transfer restrictions
; respectively. It also accepts comma separated list such as text,file,image.
; To keep compatibility, some aliases are also available:
;   true: an alias of all
;   false: an alias of none
;   yes: an alias of all
RestrictOutboundClipboard=none
; When RestrictInboundClipboard=all clipboard from the
; client is not pushed to the server.
; In addition, you can control text/file/image transfer restrictions
; respectively. It also accepts comma separated list such as text,file,image.
; To keep compatibility, some aliases are also available:
;   true: an alias of all
;   false: an alias of none
;   yes: an alias of all
RestrictInboundClipboard=none

[Sessions]
;; X11DisplayOffset - x11 display number offset
; Type: integer
; Default: 10
X11DisplayOffset=10

;; MaxSessions - maximum number of connections to an xrdp server
; Type: integer
; Default: 0
MaxSessions=50

;; KillDisconnected - kill disconnected sessions
; Type: boolean
; Default: false
; if 1, true, or yes, every session will be killed within DisconnectedTimeLimit
; seconds after the user disconnects
KillDisconnected=false

;; DisconnectedTimeLimit (seconds) - wait before kill disconnected sessions
; Type: integer
; Default: 0
; if KillDisconnected is set to false, this value is ignored
DisconnectedTimeLimit=0

;; IdleTimeLimit (seconds) - wait before disconnect idle sessions
; Type: integer
; Default: 0
; Set to 0 to disable idle disconnection.
IdleTimeLimit=0

;; Policy - session allocation policy
; Type: enum [ "Default" | "UBD" | "UBI" | "UBC" | "UBDI" | "UBDC" ]
; "Default" session per <User,BitPerPixel>
; "UBD" session per <User,BitPerPixel,DisplaySize>
; "UBI" session per <User,BitPerPixel,IPAddr>
; "UBC" session per <User,BitPerPixel,Connection>
; "UBDI" session per <User,BitPerPixel,DisplaySize,IPAddr>
; "UBDC" session per <User,BitPerPixel,DisplaySize,Connection>
Policy=Default

[Logging]
; Note: Log levels can be any of: core, error, warning, info, debug, or trace
LogFile=xrdp-sesman.log
LogLevel=DEBUG
EnableSyslog=true
#SyslogLevel=INFO
#EnableConsole=false
#ConsoleLevel=INFO
#EnableProcessId=false

[LoggingPerLogger]
; Note: per logger configuration is only used if xrdp is built with
; --enable-devel-logging
#sesman.c=INFO
#main()=INFO

;
; Session definitions - startup command-line parameters for each session type
;

[Xorg]
; Specify the path of non-suid Xorg executable. It might differ depending
; on your distribution and version. Find out the appropreate path for your
; environment. The typical path is known as follows:
;
; Fedora 26 or later    :  param=/usr/libexec/Xorg
; Debian 9 or later     :  param=/usr/lib/xorg/Xorg
; Ubuntu 16.04 or later :  param=/usr/lib/xorg/Xorg
; Arch Linux            :  param=/usr/lib/Xorg
; CentOS 7              :  param=/usr/bin/Xorg or param=Xorg
; CentOS 8              :  param=/usr/libexec/Xorg
;
param=/opt/X11/bin/xorg
; Leave the rest paramaters as-is unless you understand what will happen.
param=-config
param=xrdp/xorg.conf
param=-noreset
param=-nolisten
param=tcp
param=-logfile
param=.xorgxrdp.%s.log

[Xvnc]
param=Xvnc
param=-bs
param=-nolisten
param=tcp
param=-localhost
param=-dpi
param=96

[Chansrv]
; drive redirection
; See sesman.ini(5) for the format of this parameter
#FuseMountName=/run/user/%u/thinclient_drives
#FuseMountName=/media/thinclient_drives/%U/thinclient_drives
FuseMountName=thinclient_drives
; this value allows only the user to acess their own mapped drives.
; Make this more permissive (e.g. 022) if required.
FileUmask=077
; Can be used to disable FUSE functionality - see sesman.ini(5)
#EnableFuseMount=false
; Uncomment this line only if you are using GNOME 3 versions 3.29.92
; and up, and you wish to cut-paste files between Nautilus and Windows. Do
; not use this setting for GNOME 4, or other file managers
#UseNautilus3FlistFormat=true

[ChansrvLogging]
; Note: one log file is created per display and the LogFile config value 
; is ignored. The channel server log file names follow the naming convention: 
; xrdp-chansrv.${DISPLAY}.log
;
; Note: Log levels can be any of: core, error, warning, info, debug, or trace
LogLevel=INFO
EnableSyslog=true
#SyslogLevel=INFO
#EnableConsole=false
#ConsoleLevel=INFO
#EnableProcessId=false

[ChansrvLoggingPerLogger]
; Note: per logger configuration is only used if xrdp is built with
; --enable-devel-logging
#chansrv.c=INFO
#main()=INFO

[SessionVariables]
PULSE_SCRIPT=/etc/xrdp/pulse/default.pa

Thank you for reading to the end.
I'm getting desperate because my knowledge is not enough to solve this problem on my own. Hope together we can solve this.

[matt335672]

@gbogomolov001 - I'm sorry, but I'm not in a position to help with this very much at all. Although some of the other xrdp developers have a Mac, I don't.

I suggest getting xrdp working as a proxy to the built-in VNC server as a start. I can probably help you with that, as it's a common enough config. However, I lack the knowledge to take things beyond that.

[gbogomolov001]

Thank you for the answer!

getting xrdp working as a proxy to the built-in VNC server as a start

Could you please advice me on that?

[matt335672]

The bare bones of it look like this.

Just build xrdp - you don't need xorgxrdp for that configuration. I'd suggest using the latest v0.10.x as the error reporting is a lot better than the version you're working with.

You'll need a section in xrdp.ini which looks like this:-

[localvnc]
name=localvnc
lib=libvnc.so
port=5900
username=ask
password=ask
# chansrvport=DISPLAY(0)
enable_dynamic_resizing=false
code=0

Leave everything else as defaults for now.

Start the xrdp service. I don't know how this is done on the Mac I'm afraid.

Log in to the Mac and start the built-in VNC server.

Connect from an RDP client. Select a session type of 'localvnc' and put the username and password for the VNC session into the login box.

There's a bit of guesswork in there, but that should get us started.

[gbogomolov001]

• Do I need to start xrpd-sesman also?
• This configuration means that the connection between client machine and target machine will be established using RDP protocol, and inside macos this RDP will proxy to VNC?

[matt335672]

1. You don't need sesman for this config. It's a very simple config.
2. That's right.

[matt335672]

That's an interesting one.

getgrouplist() is not a POSIX function. On Linux, FreeBSD and OpenBSD the 3rd parameter is a pointer to gid_t. POSIX defines gid_t as an 'integer type' which means it can be signed or unsigned. It seems on MacOS, gid_t is unsigned, but getgrouplist() expects a list of (signed) integers. Also on MacOS, getgroups() uses gid_t.

Here's a patch for you:-

-- a/common/os_calls.c
+++ b/common/os_calls.c
@@ -118,6 +118,13 @@ struct sockaddr_hvs
 #define INADDR_NONE ((unsigned long)-1)
 #endif
 
+// MacOS uses a different type for getgrouplist() than getgroups()
+#ifdef __APPLE__
+#    define GETGROUPLIST_T int
+#else
+#    define GETGROUPLIST_T GETGROUPS_T
+#endif
+
 /**
  * Type big enough to hold socket address information for any connecting type
  */
@@ -3729,6 +3736,7 @@ g_getgroup_info(const char *groupname, int *gid)
 
 /*****************************************************************************/
 #ifdef HAVE_GETGROUPLIST
+
 int
 g_check_user_in_group(const char *username, int gid, int *ok)
 {
@@ -3741,13 +3749,13 @@ g_check_user_in_group(const char *username, int gid, int *ok)
         // Some implementations of getgrouplist() (i.e. muslc) don't
         // allow ngroups to be <1 on entry
         int ngroups = 1;
-        GETGROUPS_T dummy;
+        GETGROUPLIST_T dummy;
         getgrouplist(username, pwd_1->pw_gid, &dummy, &ngroups);
 
         if (ngroups > 0) // Should always be true
         {
-            GETGROUPS_T *grouplist;
-            grouplist = (GETGROUPS_T *)malloc(ngroups * sizeof(grouplist[0]));
+            GETGROUPLIST_T *grouplist;
+            grouplist = (GETGROUPLIST_T *)malloc(ngroups * sizeof(grouplist[0]));
             if (grouplist != NULL)
             {
                 // Now get the actual groups. The number of groups returned
@@ -3763,7 +3771,7 @@ g_check_user_in_group(const char *username, int gid, int *ok)
                 int i;
                 for (i = 0 ; i < ngroups; ++i)
                 {
-                    if (grouplist[i] == (GETGROUPS_T)gid)
+                    if (grouplist[i] == (GETGROUPLIST_T)gid)
                     {
                         *ok = 1;
                         break;

Here's an explanation in case you're not used to reading patches:-

1. Edit os_calls.c and add the bit near the top of the file starting with the comment '// MacOS uses a different type...'
2. In the rest of the file replace GETGROUPS_T with GETGROUPLIST_T.

[gbogomolov001]

Thank you, this helped. But.
on sudo make install LIBTOOL="glibtool" getting an error:

0000816.ini km-0000100c.ini km-00010409.ini km-19360409.ini '/etc/xrdp'
/Library/Developer/CommandLineTools/usr/bin/make install-data-hook
if [ -f /etc/init.d/xrdp ]; then
sed -i 's|BASE|/usr/local|' /etc/init.d/xrdp;
fi
sed: 1: "/etc/init.d/xrdp": command i expects \ followed by text
make[4]: *** [install-data-hook] Error 1
make[3]: *** [install-data-am] Error 2
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1

And my fix from 0.9.19 installation like editing the line sed -i 's to sed -i ‘’ -e ’s in /Users/user/Documents/xrdp-0.9.19/instfiles/makefile does not help this time.

Update:
after editing the mentioned line, getting an error

/Library/Developer/CommandLineTools/usr/bin/make install-data-hook
if [ -f /etc/init.d/xrdp ]; then
sed -i '' -e 's|BASE|/usr/local|' /etc/init.d/xrdp;
fi
(cd /etc/xrdp &&
ln -s km-00010426.ini km-00020426.ini)
ln: km-00020426.ini: File exists
make[4]: *** [install-data-hook] Error 1
make[3]: *** [install-data-am] Error 2
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1

Update 2:
Solved by deleting km-00020426.ini from /etc/xrdp folder
xdrp installed, now i will try to connect to it.

[gbogomolov001]

There is no such section:

[localvnc]
name=localvnc
lib=libvnc.so
port=5900
username=ask
password=ask
# chansrvport=DISPLAY(0)
enable_dynamic_resizing=false
code=0

But this two vnc-similar exists:

[Xvnc]
name=Xvnc
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
; port is -1 (sesman controlled), numeric (TCP connection) or an
; absolute path (UDS connection).
port=-1
; For sesman-controlled Xvnc, the 'code' parameter can be used to switch
; the connection protocol:-
; 0 - Use a TCP connection
; 1 - Use a Unix Domain Sockets (UDS) connection
;     UDS connections are not supported by older VNC servers, but are
;     supported by TigerVNC. If you select this option, comment out
;     (or remove) the 'ip=' setting.
;
; UDS connections are recommended, if your X server supports them. They are
; more secure, and untroubled by firewalls.
;
; On FIPS-based systems, TCP CANNOT be used, as the classic algorithm used for
; VNC password files is no longer considered secure by FIPS
;
; The default value is 0 on non-FIPS systems, and 1 on FIPS-based systems.
#code=0
#xserverbpp=24
#delay_ms=2000
; Disable requested encodings to support buggy VNC servers
; (1 = ExtendedDesktopSize)
#disabled_encodings_mask=0

; Generic VNC Proxy
; Tailor this to specific hosts and VNC instances by specifying an ip
; and port and setting a suitable name.
[vnc-any]
name=vnc-any
lib=libvnc.so
ip=ask
port=ask5900
username=na
password=ask
#pamusername=asksame
#pampassword=asksame
#delay_ms=2000
; Use one of these to connect to a chansrv instance created outside of sesman
; (e.g. as part of an x11vnc console session). Replace 'n' with the
; display number of the session, and (if applicable) 'u' with the numeric
; UID of the session.
;
; If 'username' or 'pamusername' is set, you probably don't need to use
; the two parameter variant with 'u'.
#chansrvport=DISPLAY(n)
#chansrvport=DISPLAY(n,u)

[gbogomolov001]

Connected to [vnc-any], don't quite understand the difference btw this and [Xvnc], will be appreciate if you could explain.

Getting black screen. Seems like vnc server cannot resize the image...

[2025-07-15T19:48:22.440+0300] [ERROR] resize_server_to_client_layout: Asked to resize server, but not possible
[2025-07-15T19:48:22.440+0300] [INFO ] dynamic_monitor_process_queue: Clearing failed request to resize to: (w: 3456 x h: 2234)

Is that [vnc-any] a real proxy connection and through the internet are "flying" RDP protocol packets?

Full log is also attached (cut it a bit for ease of reading, cause there were too many similar debug and trace messages):
xrdplog.txt

[gbogomolov001]

Changed setting in System Preferences - Sharing on target mac. Turned off "Remote management", turned on "Screen Sharing" for All users. Now I can see 1/4 of my mac pro login screen, cause vnc cannot resize it. Also, I can move mouse, can enter the password. But after entering the password the user session logs in on target mac pro, bit on client device from I've connected - I still see this 1/4 of login screen and iI can't see the desktop. xrdp log - still same errors about resizing.

[matt335672]

The setup you've got now is a simple proxy to the console. Resize can't be supported, as the VNC server on the console doesn't expose that function.

The difference with [Xvnc] is that sesman isn't involved at the moment. You've managed to prove that there's nothing wrong with the xrdp bits though.

What I don't know is how you can get sesman to start a session on a Mac. I'm hoping some of the guides will make this clearer now you've got something working.

I'll take the changes you've got in the discussion above and get them merged into the main product.

[gbogomolov001]

Did not changed anything in /etc/xrdp/startwm.sh
Started sesman. Tried Xvnc. Getting:

025-07-16T19:17:43.899+0300] [DEBUG] Waiting for X server to start on display :10
[2025-07-16T19:17:43.899+0300] [DEBUG] Calling exec (excutable: /usr/local/libexec/xrdp/waitforx, arguments: /usr/local/libexec/xrdp/waitforx -d :10)
[2025-07-16T19:17:43.903+0300] [INFO ] Starting X server on display 10: Xvnc :10 -auth .Xauthority -geometry 2560x1600 -depth 16 -rfbauth /Users/user/.vnc/[email protected]:10 -bs -nolisten tcp -localhost -dpi 96
[2025-07-16T19:17:43.903+0300] [DEBUG] Calling exec (excutable: Xvnc, arguments: Xvnc :10 -auth .Xauthority -geometry 2560x1600 -depth 16 -rfbauth /Users/user/.vnc/[email protected]:10 -bs -nolisten tcp -localhost -dpi 96)
[2025-07-16T19:17:43.904+0300] [ERROR] Error calling exec (excutable: Xvnc, arguments: Xvnc :10 -auth .Xauthority -geometry 2560x1600 -depth 16 -rfbauth /Users/user/.vnc/[email protected]:10 -bs -nolisten tcp -localhost -dpi 96) returned errno: 2, description: No such file or directory
[2025-07-16T19:17:43.904+0300] [ERROR] A fatal error has occurred attempting to start the X server on display 10, aborting connection
[2025-07-16T19:17:43.996+0300] [DEBUG] waitforx: Opening display :10. Attempt 1 of 10
[2025-07-16T19:17:45.407+0300] [DEBUG] waitforx: Opening display :10. Attempt 2 of 10
[2025-07-16T19:17:46.711+0300] [DEBUG] waitforx: Opening display :10. Attempt 3 of 10
[2025-07-16T19:17:47.942+0300] [DEBUG] waitforx: Opening display :10. Attempt 4 of 10
[2025-07-16T19:17:48.135+0300] [DEBUG] waitforx: Opening display :10. Attempt 5 of 10
[2025-07-16T19:17:49.161+0300] [DEBUG] waitforx: Opening display :10. Attempt 6 of 10
[2025-07-16T19:17:50.187+0300] [DEBUG] waitforx: Opening display :10. Attempt 7 of 10
[2025-07-16T19:17:51.258+0300] [DEBUG] waitforx: Opening display :10. Attempt 8 of 10
[2025-07-16T19:17:52.298+0300] [DEBUG] waitforx: Opening display :10. Attempt 9 of 10
[2025-07-16T19:17:53.339+0300] [DEBUG] waitforx: Opening display :10. Attempt 10 of 10
[2025-07-16T19:17:54.359+0300] [ERROR] waitforx: Unable to open display :10
[2025-07-16T19:17:54.372+0300] [DEBUG] waiting for pid 656 to exit
[2025-07-16T19:17:54.374+0300] [ERROR] X server failed to start
[2025-07-16T19:17:54.377+0300] [DEBUG] Closed socket 8 (AF_UNIX)
[2025-07-16T19:17:54.387+0300] [INFO ] Session on display 10 has finished.
[2025-07-16T19:17:54.396+0300] [DEBUG] Closed socket 13 (AF_UNIX)
[2025-07-16T19:17:54.671+0300] [DEBUG] Closed socket 14 (AF_UNIX)\

Now trying to figure out what is this "display 10".
Full log attached if somebody is interested.
xrdp-sesman-log.txt

UPDATE.
Tried to logout from the user on target mac before remote connection. Getting:

[2025-07-16T20:01:19.478+0300] [ERROR] Can't get UID for user user

macOS admin username is "user". Used both passwords for VNC built-in macOS server and for account. No effects.

[gbogomolov001]

@matt335672 good day to you! Do u have any guesses regarding info above?

[matt335672]

@gbogomolov001 - I can't help you much past this point as I don't have a Mac. You'll need to look at the notes pages which you have access to to figure out what to do going forwards.

[gbogomolov001]

@matt335672 Yes, I understand that point that you don't have a macOS device. Could you please just explain me some things which will help me to understand what is this soft consist of?

1. Both of Xorg/Xvnc are using sesman, right?
2. xrdp-sesman is a part of xrdp component, or a part of xorgxrdp component?
3. Do I need at all xorgxrdp component to try Xvnc connection? Trying to figure out, do I need to update my current xorgxrdp 0.2.18 to try Xvnc, or not.
4. Does installation of xrdp/homebrew/xorgxrdp install Xvnc server also? Or I need to install Xvnc server separately?
5. Same question as 4 for Xorg.

[matt335672]

1. Yes. Having port=-1 in the config will use sesman
2. sesman comes with xrdp.
3. xorgxrdp has nothing to do with Xvnc. xorgxrdp is an addon to Xorg which lets you use Xorg with xrdp.
4. On Linux/FreeBSD, etc Xvnc is provided by a separate program of which there are many variants (tigervnc, tight vnc, free vnc, etc).
5. Xorg is entirely separate from xrdp. It used to be the main display server for Linux/FreeBSD etc, but it is slowly being replaced by Wayland.