Random "remote error: tls: illegal parameter"

[GammaPi]

The problem

While using teddysun's Docker container for Xray, I found a potential issue caused by its legacy OpenSSL version.

I used an nginx+websocket+tls configuration, and sometimes Xray completely fails to connect upon container start and continuously throws the following error:

[Error] [936000282] transport/internet/websocket: failed to dial to <SERVER_ADDRESS> > remote error: tls: illegal parameter

The problem may only occur after restarting the container a few times.

I used Wireshark to investigate and found that Xray is trying to handshake with X25519MLKEM768 cipher suites, which are not available in the OpenSSL version installed in the latest Xray container image as of September 10th, 2025.

To solve this issue, the openssl package should be upgraded to v3.5 or later. You can verify this by first installing OpenSSL in the container and then running a test command:

apk add openssl 

openssl s_client -connect <Server name> \
    -groups X25519MLKEM768 \
    -tls1_3

The above command will throw an illegal parameter error.

This problem is confusing for users.

To fix the problem

1. Switch to the official Xray image to use a newer version of OpenSSL. I reported the packaging problem to @teddysun
2. After switching the image, the error changed to "CurvePreferences includes unsupported curve". The problem seems related to TLS连接有一定概率会报错CurvePreferences includes unsupported curve #5031. So I have to switch the fingerprint to chrome.

[GammaPi]

Members in the Project X group reminded me that X25519MLKEM768 requires golang 1.24+ and golang does not use OpenSSL.
The problem might be caused by the compilation procedures in Teddy's image.
The solution is still switch to the official image. But the root-cause may not be OpenSSL version related.

[Meo597]

编译参数cgo enable 0后是不会依赖c库更不会依赖openssl的

然而teddysun的编译环境是黑盒，我们看不到
猜测大概率是编译环境的golang版本过低导致

你可以改用官方镜像ghcr.io/xtls/xray-core:latest

[teddysun]

编译脚本如下：
https://github.com/teddysun/across/blob/master/docker/xray/build_xray.sh

Golang 版本是 1.25.0

[GammaPi]

Thanks for your response. Then it's clear that I might made wrong assumptions while diagnosing the problem.

Another user also reported a similar error message. I'm not sure which image does this user use.

I will just leave the discussion here for experts to diagnose.