Help for configuring Mercure behind nginx as proxy-server

[Astro-Otter-Space]

Hello,

I'm deploying Mercure hub on my server (VPS hosted by OVH) where an API REST (Symfony+FOSRestBundle) and a front app (VueJS 3, vue-cli) are installed yet, served with Nginx. API and FO have their own domains (https://api.exemple.space/ and https://www.exemple.space/). I've created a domain for Mercure (https://mercure.exemple.com/). Mercure (version 0.15) is installed with binary, not with docker.

I added an nginx host configuration for working as reverse-proxy (https://mercure.rocks/docs/hub/nginx):

# should i keep these lines commented or not ?
#server {
#  listen      80 http2;
#  server_name mercure.exemple.space;
#  return 301 https://mercure.exemple.space;
#}

server {
  listen      443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mercure.exemple.space;

  ssl_certificate /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem; 
  ssl_certificate_key /etc/letsencrypt/live/mercure.exemple.space/privkey.pem;
  
  location / {
    proxy_pass http://127.0.0.1:3000; # <-- is correct ?
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    #proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  # Configuration des logs
  access_log  /var/log/nginx/mercure_access.log;
  error_log   /var/log/nginx/mercure_error.log;
}

i set public and private keys and set them as env variables in /etc/profile.d/mercure.sh

export MERCURE_PUBLISHER_JWT_KEY=$(cat /home/me/mercure/publisher.key.pub)
export MERCURE_PUBLISHER_JWT_ALG=RS256
export MERCURE_SUBSCRIBER_JWT_KEY=$(cat /home/me/mercure/subscriber.key.pub)
export MERCURE_PUBLISHER_JWT_ALG=RS256
export SERVER_NAME=localhost:3000 # <--  mercure.exemple.space or mercure.exemple.space:3000, which one is correct ?

I run with command MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY /usr/bin/mercure run --config /home/stephane/mercure/Caddyfile

My Caddyfile :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3001 # Should i keep this line ?
	https_port 3000 # Should i keep this line ?
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
        # tls line is commented because Mercure cant read keys, change rights/owner ?
	# tls /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem /etc/letsencrypt/live/mercure.exemple.space/privkey.pem

	log {
		format filter {
			wrap console
			fields {
				uri query {
					replace authorization REDACTED
				}
			}
		}
	}

	encode zstd gzip

	mercure {
		# Transport to use (default to Bolt)
		transport_url {$MERCURE_TRANSPORT_URL:bolt://mercure.db}
		# Publisher JWT key
		publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
		# Subscriber JWT key
		subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
		# Extra directives
		# CORS
		cors_origins https://www.exemple.space https://exemple.local:8080 https://localhost:8080 <-- maybe some error here ?
		publish_origins *
		anonymous
		subscriptions
		{$MERCURE_EXTRA_DIRECTIVES}
	}

	{$CADDY_SERVER_EXTRA_DIRECTIVES}

	respond /healthz 200
	respond "Not Found" 404
}

In log i have :

2024/01/03 11:07:15.053	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "mercure.exemple.space", "issuer": "acme.zerossl.com-v2-DV90", "error": "[mercure.astro-otter.space] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: performing request: Post \"https://acme.zerossl.com/v2/DV90/newOrder\": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)"}
2024/01/03 11:07:15.053	DEBUG	events	event	{"name": "cert_failed", "id": "533a237d-c07a-4cd4-a12d-dfe931a9c5aa", "origin": "tls", "data": {"error":{},"identifier":"mercure.exemple.space","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}

2024/01/03 11:07:15.053	ERROR	tls.obtain	will retry	{"error": "[mercure.exemple.space] Obtain: [mercure.exemple.space] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: performing request: Post \"https://acme.zerossl.com/v2/DV90/newOrder\": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 1, "retrying_in": 60, "elapsed": 127.939511407, "max_duration": 2592000}

If i run curl from my local env :

$ curl -X GET https://mercure.exemple.space/.well-known/mercure
Client sent an HTTP request to an HTTPS server.
$ curl -X GET https://mercure.exemple.space:3000/.well-known/mercure
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

In my JS app (from local or prod env), i have CORS error with this code :

  const hubUrl = new URL('https://mercure.exemple.space');
  const domain = 'https://api.exemple.space'
  const topic = 'notifications/all';
  hubUrl.searchParams.append('topic', `${domain}/${topic}`);
  const eventSource = new EventSource(hubUrl.toString(), {withCredentials: true});
  eventSource.onmessage = (event) => {
    console.log(event.data);
  }

I need help for wrtting good configuration for Nginx and Caddyfile. I'll fix CORS errors later.
Thank you for help :)

[dunglas]

Try to disable TLS on Mercure. For instance, set SERVER_NAME to http://localhost (notice the http:// prefix).

[Astro-Otter-Space]

ok I'll do that. I changed SERVER_NAME value in /etc/environment

$ echo $SERVER_NAME 
http://localhost

and changed Caddyfile like this :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3000
	auto_https off
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
	# tls /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem /etc/letsencrypt/live/mercure.exemple.space/privkey.pem

line tls ... commented and disabled like i saw here or here
Is it enough ?

I have no more error in console (a good point ^^).

$ MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY \
MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY \
SERVER_NAME=$SERVER_NAME \
DEBUG=debug \
/usr/bin/mercure run --config Caddyfile
2024/01/04 08:32:28.347	INFO	using provided configuration	{"config_file": "Caddyfile", "config_adapter": ""}
2024/01/04 08:32:28.350	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2024/01/04 08:32:28.351	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/01/04 08:32:28.352	WARN	http.auto_https	automatic HTTPS is completely disabled for server	{"server_name": "srv0"}
2024/01/04 08:32:28.352	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{}]}}, "http": {"http_port":3000,"servers":{"srv0":{"listen":[":3000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Content-Type":["text/html; charset=utf-8"]}}}],"match":[{"path":["/"]}]},{"handle":[{"encodings":{"gzip":{},"zstd":{}},"handler":"encode","prefer":["zstd","gzip"]},{"anonymous":true,"cors_origins":["https://news.exemple.space","https://exemple.local:8080","https://localhost:8080"],"handler":"mercure","publish_origins":["*"],"publisher_jwt":{"alg":"{env.MERCURE_PUBLISHER_JWT_ALG}","key":"{env.MERCURE_PUBLISHER_JWT_KEY}"},"subscriber_jwt":{"alg":"{env.MERCURE_SUBSCRIBER_JWT_ALG}","key":"{env.MERCURE_SUBSCRIBER_JWT_KEY}"},"subscriptions":true,"transport_url":"bolt://mercure.db"}]},{"handle":[{"handler":"static_response","status_code":200}],"match":[{"path":["/healthz"]}]},{"handle":[{"body":"\u003c!DOCTYPE html\u003e\n\t\u003chtml lang=en\u003e\n\t\u003cmeta charset=\"utf-8\"\u003e\n\t\u003cmeta name=\"robots\" content=\"noindex\"\u003e\n\t\u003ctitle\u003eWelcome to Mercure\u003c/title\u003e\n\t\u003ch1\u003eWelcome to Mercure\u003c/h1\u003e\n\t\u003cp\u003eThe URL of your hub is \u003ccode\u003e/.well-known/mercure\u003c/code\u003e.\n\tRead the documentation on \u003ca href=\"https://mercure.rocks\"\u003eMercure.rocks, real-time apps made easy\u003c/a\u003e.","handler":"static_response"}],"match":[{"path":["/"]}]},{"handle":[{"body":"Not Found","handler":"static_response","status_code":404}]}]}],"terminal":true}],"automatic_https":{"disable":true},"logs":{"logger_names":{"localhost":"log0"}}}}}}
2024/01/04 08:32:28.354	DEBUG	http	starting server loop	{"address": "[::]:3000", "tls": false, "http3": false}
2024/01/04 08:32:28.355	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/01/04 08:32:28.355	INFO	autosaved config (load with --resume flag)	{"file": "/home/stephane/.config/caddy/autosave.json"}
2024/01/04 08:32:28.355	INFO	serving initial configuration
2024/01/04 08:32:28.357	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0004c0000"}
2024/01/04 08:32:28.359	WARN	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/home/stephane/.local/share/caddy", "instance": "359fed6e-f64a-4f98-a8cb-e2b7bb24d40c", "try_again": "2024/01/05 08:32:28.359", "try_again_in": 86399.999999583}
2024/01/04 08:32:28.359	INFO	tls	finished cleaning storage units

And i saw my request in log :

2024/01/04 08:37:37.796	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "36484", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1:3000", "uri": "/.well-known/mercure?topic=https%3A%2F%2Fapi.exemple.space%2Fnotifications%2Fdso", "headers": {"Sec-Fetch-Site": ["cross-site"], "Sec-Fetch-Mode": ["cors"], "Referer": ["https://localhost:8080/"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["185.101.209.57"], "Cache-Control": ["no-cache"], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"], "Accept-Language": ["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"], "Sec-Ch-Ua-Platform": ["\"Linux\""], "Sec-Fetch-Dest": ["empty"], "X-Forwarded-Host": ["mercure.exemple.space"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\""], "Accept": ["text/event-stream"], "Origin": ["https://localhost:8080"]}}, "bytes_read": 0, "user_id": "", "duration": 0.000004338, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}

If i curl from local :

curl -k -I -X GET https://mercure.exemple.space/.well-known/mercure
HTTP/2 200 
server: nginx
date: Thu, 04 Jan 2024 08:33:39 GMT
content-length: 0

JS side, i got 200 too but CORS error

Access to resource at 'https://mercure.exemple.space/.well-known/mercure?topic=https%3A%2F%2Fapi.astro-otter.space%2Fnotifications%2Fall' from origin 'https://localhost:8080' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.
mercure:1 
        
        
       GET https://mercure.exemple.space/.well-known/mercure?topic=https%3A%2F%2Fapi.exemple.space%2Fnotifications%2Fall net::ERR_FAILED 200 (OK)
home:1 EventSource's response has a MIME type ("text/plain") that is not "text/event-stream". Aborting the connection.

It seems it's on a good way ^^

[Astro-Otter-Space]

I added in my nginx vhost these lines :

  location / {
    if ($http_origin ~ '^https?://(localhost(:[0-9]+)?|[^/]*\.exemple\.space)') {
        set $cors "true";
    }    

    if ($cors = "true") {
      add_header 'Access-Control-Allow-Origin' '$http_origin' always;    
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
      add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS';
      add_header "Content-Type" "text/event-stream";
    }	    

    proxy_pass http://127.0.0.1:3000;
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

And no more CORS errors.

[dunglas]

This shouldn't be necessary but thanks for the workaround! I'll take a look.

[Astro-Otter-Space]

Without Access-Control-Allow-Origin, Access-Control-Allow-Credentials and Content-Type, i have CORS error JS-side.

With this nginx configuration, what should be values of cors_origins and publish_origins in my Caddyfile ?
You told me to set http://localhost for env variable SERVER_NAME. Should i set proxy_pass http://localhost:3000; instead of proxy_pass http://127.0.0.1:3000; ?

UPDATE 09/01/2024 :
i've changed proxy_pass http://127.0.0.1:3000; into proxy_pass http://localhost:3000; in nginx.
Now when i do curl http://localhost:3000 (on server) and curl https://mercure.astro-otter.space in local, i have the response from Caddy same as defined in Caddyfile,good point (youpi)

If i'm publishing from my Symfony controller or POST curl request:

• i got exception HTTP/1.1 401 Unauthorized returned for "http://localhost:3000/.well-known/mercure"." with symfony
• in log: Topic selectors not matched, not provided or authorization error {"remote_addr": "127.0.0.1:59610", "error": "unable to parse JWT: signature is invalid"}

[Astro-Otter-Space]

Finally i found a way to make it working correctly. I resume :
First, i'm working with binary, not docker image.
Env file /etc/environment :

MERCURE_PUBLISHER_JWT_KEY="mySecretKey"
MERCURE_SUBSCRIBER_JWT_KEY="mySecretKey"
# Mercure URL
SERVER_NAME=http://localhost
MERCURE_PUBLIC_URL=https://mercure.exemple.space/.well-known/mercure

My nginx vhost :

server {
  listen      443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mercure.exemple.space;

  ssl_certificate /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/mercure.exemple.space/privkey.pem; # managed by Certbot
  
  location / {
    if ($http_origin ~ '^https?://(localhost(:[0-9]+)?|127.0.0.1(:[0-9]+)?|[^/]*\.exemple\.space)') {
        set $cors "true";
    }    

    if ($cors = "true") {
      add_header 'Access-Control-Allow-Origin' '$http_origin' always;    
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
      add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS';
      add_header 'Content-Type' 'text/event-stream';
    }	    

    proxy_pass http://localhost:3000;
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    #proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  # Configuration des logs
  access_log  /var/log/nginx/mercure_access.log;
  error_log   /var/log/nginx/mercure_error.log;

I create a systemd file for running mercure as service /etc/systemd/system/mercure.service :

[Unit]
Description=Mercure.Rocks service
After=network.target
StartLimitBurst=5
StartLimitIntervalSec=33

[Service]
Type=simple
WorkingDirectory=/tmp
EnvironmentFile=-/etc/environment
ExecStart=/usr/bin/bash -c "MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY SERVER_NAME=$SERVER_NAME /usr/bin/mercure run --config /var/www/mercure/Caddyfile"
StandardOutput=file:/var/log/nginx/mercure.log
StandardError=file:/var/log/nginx/mercure.log
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

In /var/www/mercure, here's my caddyfile :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3000
	auto_https off
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
	log {
		output file /var/log/caddy/mercure.log {
			roll true
			roll_size_mb 10
		        roll_keep 5
		}
		format filter {
			wrap console
			fields {
				uri query {
					replace authorization REDACTED
				}
			}
		}
		level INFO
	}

	encode zstd gzip

	mercure {
		# Transport to use (default to Bolt)
		transport_url {$MERCURE_TRANSPORT_URL:bolt://mercure.db}
		# Publisher JWT key
		publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
		# Subscriber JWT key
		subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
		# Extra directives
		# CORS
		cors_origins /** add here URL **/
		publish_origins *
		anonymous
		subscriptions
		{$MERCURE_EXTRA_DIRECTIVES}
	}

	{$CADDY_SERVER_EXTRA_DIRECTIVES}

	header / Content-Type "text/html; charset=utf-8"
	respond / `<!DOCTYPE html>
	<html lang=en>
	<meta charset="utf-8">
	<meta name="robots" content="noindex">
	<title>Welcome to Mercure</title>
	<h1>Welcome to Mercure</h1>
	<p>The URL of your hub is <code>/.well-known/mercure</code>.
	Read the documentation on <a href="https://mercure.rocks">Mercure.rocks, real-time apps made easy</a>.`

	respond /healthz 200

Code-side

My backend is a symfony (upgrade to API Platform ne day ^^).
/path/to/backend/c/.env

MERCURE_URL=http://localhost:3000/.well-known/mercure
MERCURE_PUBLIC_URL=https://mercure.exemple.space/.well-known/mercure # use variable from /etc/environment ?
MERCURE_JWT_SECRET="PutYoutJWTHere"

/path/to/backend/config/packages/mercure.yaml

mercure:
  hubs:
    default:
      url: '%env(MERCURE_URL)%'
      public_url: '%env(MERCURE_PUBLIC_URL)%'
      jwt: 
        value: '%env(MERCURE_JWT_SECRET)%'
        publish: '*'

Front-side is a VueJS application, using EventSource, nothing exotic.

I think i need some adjustments but with all these it's working :).

[erseco]

Hi

I encountered a similar issue with Mercure shutting down slowly due to long-running SSE connections. In my case, I was also using an Nginx proxy in front of Mercure, and I managed to fix the problem by adding:

proxy_buffering off;

to the Nginx configuration for the Mercure endpoint.

This ensures that Nginx does not buffer the SSE responses, allowing connections to close properly when Mercure shuts down. After adding this directive, Mercure now shuts down immediately without waiting for the grace period to expire.

Hope this helps!

Best,
Ernesto

[dunglas]

FIxed by #1005.