How to properly enable X25519MLKEM768 support?

[Amia33]

I've compiled latest openssl v3.5.0 and nginx v1.28.0, made a website working with X25519MLKEM768, but can't "steal" it as REALITY dest.

I used Linode tutorial to compile softwares.

Working /opt/nginx/nginx.conf snippet:

ssl_dhparam            /opt/nginx/dhparam.pem;
ssl_protocols          TLSv1.2 TLSv1.3;
ssl_ciphers            ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers on;

Not working /opt/nginx/nginx.conf snippet:

ssl_protocols          TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve         X25519MLKEM768;

Inbound snippet for server config:

{
      "port": 443,
      "protocol": "vless",
      "settings": {
        "clients": [
          {
            "id": "uuid",
            "email": "email",
            "flow": "xtls-rprx-vision"
          }
        ],
        "decryption": "none",
        "fallbacks": [
          {
            "dest": "@xhttp"
          }
        ]
      },
      "streamSettings": {
        "security": "reality",
        "realitySettings": {
          "show": false,
          "target": "8001",
          "xver": 1,
          "serverNames": ["domain"],
          "privateKey": "privkey",
          "shortIds": ["shortid"],
          "maxTimeDiff": 60000
        }
      },
      "tag": "reality_inbound",
      "sniffing": {
        "enabled": true,
        "destOverride": ["http", "tls", "quic"]
      }
    },
    {
      "listen": "@xhttp",
      "protocol": "vless",
      "settings": {
        "clients": [
          {
            "id": "uuid",
            "email": "email"
          }
        ],
        "decryption": "none"
      },
      "streamSettings": {
        "network": "xhttp",
        "xhttpSettings": {
          "host": "domain",
          "path": "/path",
          "mode": "auto"
        }
      },
      "tag": "xhttp_inbound",
      "sniffing": {
        "enabled": true,
        "destOverride": ["http", "tls", "quic"]
      }
    }

Since I only changed nginx config to break my implementation, I think it has something to do with nginx... Plz help!

[lxhao61]

Try to change the snippet of opt/nginx/nginx.conf to the following:​

ssl_protocols          TLSv1.2 TLSv1.3;
ssl_ciphers            ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve         X25519MLKEM768;

[Amia33]

Try to change the snippet of opt/nginx/nginx.conf to the following:​

ssl_protocols          TLSv1.2 TLSv1.3;
ssl_ciphers            ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve         X25519MLKEM768;

Nope... It still returned EOF error.

[lxhao61]

You didn’t compile with the system required by the Linode tutorial​, did you?
It’s recommended to compile with the path to the OpenSSL library source set (--with-openssl=path), which is not limited to any specific system.​

[Amia33]

You didn’t compile with the system required by the Linode tutorial​, did you? It’s recommended to compile with the path to the OpenSSL library source set (--with-openssl=path), which is not limited to any specific system.​

Let me check it... I explicitly followed the tutorial.
Yet the tutorial didn't set specific --with-openssl value
Let me compile NGINX again.

[Amia33]

You didn’t compile with the system required by the Linode tutorial​, did you? It’s recommended to compile with the path to the OpenSSL library source set (--with-openssl=path), which is not limited to any specific system.​

Nope... I specified openssl 3.5.0 src but it still failed with your new snippet.
I'm gonna put my logs here for your need.

Server-side log:

2025/06/06 17:24:36.938019 [Warning] core: Xray 25.5.16 started
2025/06/06 17:24:51.212799 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.214994 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.215958 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.218738 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.220002 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.220368 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.220636 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.226748 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.227944 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.271013 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.275367 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:24:51.275724 [Info] transport/internet/tcp: REALITY: processed invalid connection
2025/06/06 17:25:49.454413 [Debug] app/log: Logger closing

Client-side log:

2025/06/07 00:24:40.312015 [Info] [4116650243] proxy/vless/outbound: tunneling request to tcp:ip:443 via [ip]:443
2025/06/07 00:24:40.312028 [Info] [2474672461] transport/internet/splithttp: failed to POST https://domain/path/ > Post "https://domain/deforest-shading-operation-catty-saggy-affidavit-upon-backlogpath/": remote error: tls: handshake failure

[lxhao61]

Based on the information and responses you provided, please proceed as follows:​

1. Check if the Xray client is version v25.5.16. For third-party clients, the core Xray version is required to be v25.5.16.​
2. If there is no issue with the above, it is recommended to compile Nginx v1.28.0 directly using the OpenSSL v3.5.0 source library (--with-openssl=path). It has been tested and works fine (I compiled it this way myself).​

[Amia33]

Based on the information and responses you provided, please proceed as follows:​

1. Check if the Xray client is version v25.5.16. For third-party clients, the core Xray version is required to be v25.5.16.​

2. If there is no issue with the above, it is recommended to compile Nginx v1.28.0 directly using the OpenSSL v3.5.0 source library (--with-openssl=path). It has been tested and works fine (I compiled it this way myself).​

Thanks for your help...
Actually I have a question: Now using my old (first) snippet, xray-core works and I can access my website on chrome, showing it using X25519MLKEM768. I don't know how to check if REALITY is using it...
Maybe this shouldn't be a question in the first place

[Amia33]

@lxhao61 Thanks for your help and I finally found that I was soooooo dumb. By using old working snippet the website can be accessed with X25519MLKEM768, meaning that I've already using latest PQ Key Agreement.

I'm gonna leave this discussion here in case someone wants to know how to build nginx support(