User does not exist and VNC password failed

[proJM-Coding]

Hello! I am struggling with getting xrdp working for my server with Rocky Linux 9. I get the error User does not exist, or could not be authenticated but the user does 100% exist on my system. Here are the logs:

~/.xsession-errors:

Environment variable $XAUTHORITY not set, ignoring.

/var/log/xrdp.log:

[2024-10-02T18:15:18.968+0930] [WARN ] Ignoring obsolete SCP port value '3350'
[2024-10-02T18:15:18.971+0930] [INFO ] xrdp_wm_log_msg: sesman connect ok
[2024-10-02T18:15:18.973+0930] [INFO ] xrdp_wm_log_msg: Performing login request for mini-server-remote
[2024-10-02T18:15:20.779+0930] [INFO ] xrdp_wm_log_msg: User does not exist, or could not be authenticated
[2024-10-02T18:15:27.558+0930] [INFO ] Login screen monitor height is 1080 pixels over 173 mm (158 DPI)

/var/log/xrdp-sesman.log:

[2024-10-02T18:14:25.919+0930] [INFO ] starting xrdp-sesman with pid 1467
[2024-10-02T18:14:25.923+0930] [INFO ] Sesman now listening on /run/xrdp/sesman.socket
[2024-10-02T18:15:18.971+0930] [INFO ] Socket 13: connection accepted from AF_UNIX
[2024-10-02T18:15:18.976+0930] [INFO ] Received system login request from xrdp for user: mini-server-remote IP: ::ffff:192.168.1.4
[2024-10-02T18:15:18.983+0930] [INFO ] starting xrdp-sesexec with pid 5534
[2024-10-02T18:15:20.768+0930] [ERROR] pam_authenticate failed: Authentication failure
[2024-10-02T18:15:20.776+0930] [INFO ] AUTHFAIL: user=mini-server-remote ip=::ffff:192.168.1.4 time=1727858720
[2024-10-02T18:15:28.634+0930] [ERROR] Error reading libipm transport
[2024-10-02T18:15:28.642+0930] [INFO ] Received sys login status for mini-server-remote : not logged in
[2024-10-02T18:15:28.645+0930] [ERROR] sesexec_main_loop: trans_check_wait_objs failed for ECP transport

I tried making a new user and connecting to that and gave me VNC password failed. The logs for that are here:

~/.xsession-errors:

Environment variable $XAUTHORITY not set, ignoring.

/var/log/xrdp.log:

[2024-10-02T18:41:00.634+0930] [WARN ] Ignoring obsolete SCP port value '3350'
[2024-10-02T18:41:00.147+0930] [INFO ] xrdp_wm_log_msg: sesman connect ok
[2024-10-02T18:41:00.176+0930] [INFO ] xrdp_wm_log_msg: Performing login request for mini-server-user
[2024-10-02T18:41:00.558+0930] [INFO ] xrdp_wm_log_msg: login was successful - creating session
[2024-10-02T18:41:01.736+0930] [INFO ] xrdp_wm_log_msg: session is available on display 10 for user mini-server-user
[2024-10-02T18:41:01.738+0930] [INFO ] xrdp_wm_log_msg: Connecting to session
[2024-10-02T18:41:01.741+0930] [INFO ] loaded module 'libvnc.so' ok, interface size 4832, version 4
[2024-10-02T18:41:01.745+0930] [ERROR] DES-EDE3-CBC: error:0308010C:digital envelope routines::unsupported
[2024-10-02T18:41:01.748+0930] [ERROR] DES-EDE3-CBC: error:0308010C:digital envelope routines::unsupported
[2024-10-02T18:41:01.852+0930] [ERROR] VNC error before sending share flag
[2024-10-02T18:41:01.855+0930] [ERROR] VNC error before receiving server init
[2024-10-02T18:41:01.858+0930] [ERROR] VNC error before receiving pixel format
[2024-10-02T18:41:01.860+0930] [ERROR] VNC error before receiving name length
[2024-10-02T18:41:01.862+0930] [ERROR] VNC error before receiving name
[2024-10-02T18:41:01.865+0930] [ERROR] xrdp_wm_log_msg: Error connecting to user session
[2024-10-02T18:41:17.758+0930] [INFO ] Login screen monitor height is 1080 pixels over 173 mm (158 DPI)

/var/log/xrdp-sesman.log:

[2024-10-02T18:41:00.149+0930] [INFO ] Socket 13: connection accepted from AF_UNIX
[2024-10-02T18:41:00.203+0930] [INFO ] Received system login request from xrdp for user: mini-server-user IP: ::ffff:192.168.1.4
[2024-10-02T18:41:00.273+0930] [INFO ] starting xrdp-sesexec with pid 5516
[2024-10-02T18:41:00.508+0930] [INFO ] TerminalServerUsers group tsusers doesn't exist. Access granted for mini-server-user
[2024-10-02T18:41:00.532+0930] [INFO ] Access permitted for user: mini-server-user
[2024-10-02T18:41:00.557+0930] [INFO ] Received sys login status for mini-server-user : logged in
[2024-10-02T18:41:00.606+0930] [INFO ] Received request from xrdp to create a session for user mini-server-user
[2024-10-02T18:41:00.745+0930] [ERROR] DES-EDE3-CBC: error:0308010C:digital envelope routines::unsupported
[2024-10-02T18:41:00.752+0930] [INFO ] Starting X server on display 10: Xvnc :10 -auth .Xauthority -geometry 1920x1080 -depth 16 -rfbauth /home/mini-server-user/.vnc/[email protected]:10 -bs -nolisten tcp -localhost -dpi 96 
[2024-10-02T18:41:01.718+0930] [INFO ] X server :10 is working
[2024-10-02T18:41:01.726+0930] [INFO ] Starting window manager for display :10
[2024-10-02T18:41:01.728+0930] [INFO ] Starting the xrdp channel server for display :10
[2024-10-02T18:41:01.729+0930] [INFO ] Using the default window manager on display 10: /usr/libexec/xrdp/startwm-bash.sh
[2024-10-02T18:41:01.731+0930] [INFO ] Session in progress on display :10. Waiting until the window manager (pid 5564) exits to end the session

I installed XRDP by following the guide https://docs.rockylinux.org/desktop/gnome/rdp-server/. Thank you!

[matt335672]

Are you running FIPS?

[2024-10-02T18:41:00.745+0930] [ERROR] DES-EDE3-CBC: error:0308010C:digital envelope routines::unsupported

See #2518

[proJM-Coding]

Yes, I am running FIPS. Thanks for the comment I followed this workaround which works perfectly now.