Using Librenms with nginx reverse proxy

[pr3d1]

Behaviour

I want to use Librenms with nginx reversy proxy, but the redirection does not work properly.

Steps to reproduce this issue

1. Create MySQL server
2. Run the Librenms container with LIBRENMS_BASE_URL env var
3. Run and configure nginx

docker run -d -p 8000:8000 --name librenms --network test --cap-add=NET_ADMIN --cap-add=NET_RAW -e "LIBRENMS_BASE_URL=/librenms/" -e "DB_HOST=mysql" -e "DB_PASSWORD=mysql" -e "DB_USER=root" librenms/librenms:latest
docker run -d --name nginx -p 80:80 --network test nginx

Expected behaviour

Nginx proxies requests for the virtual path

Actual behaviour

The HOST/librenms request forwarded to the librenms container. Then the container redirects it to HOST/login page. This page obviously does not exist, the correct path would be HOST/librenms/login.

Configuration

I tested this behaviour with multiple nginx and docker configurations.

• Docker version (type docker --version) : Docker version 20.10.5, build 55c4c88966
• Platform (Debian 9, Ubuntu 18.04, ...) : Arch Linux, SLES15 SP2
• System info (type uname -a) : 5.11.10-arch1-1 can not connect to librenms gui - 404 page not found #1 SMP PREEMPT Fri, 26 Mar 2021 00:11:29 +0000 x86_64 GNU/Linux
• Include all necessary configuration files : docker-compose.yml, .env, ...

I tried it with many nginx configurations, i think this one should work (/etc/nginx/conf.d/default.conf contents):

server {
  set $forward_scheme http;
  set $server         "localhost";
  set $port           80;
}
listen 80;
listen [::]:80;

listen 443 ssl http2;
listen [::]:443;


  location /librenms {
    proxy_pass       http://librenms:8000;
    rewrite  ^/librenms/(.*)  /$1 break;

  }

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 4
  Running: 3
  Paused: 0
  Stopped: 1
 Images: 11
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.11.10-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.42GiB
 Name: dfaltum-npsh
 ID: R6HH:BXJS:T7JN:C4FL:RF4D:C2NF:IB6C:CODF:MUJV:XXXY:TPDN:5PB6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Logs

curl -v -LI --insecure  http://localhost/librenms

*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> HEAD /librenms HTTP/1.1
> Host: localhost
> User-Agent: curl/7.75.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Server: nginx/1.19.8
Server: nginx/1.19.8
< Date: Tue, 30 Mar 2021 09:09:50 GMT
Date: Tue, 30 Mar 2021 09:09:50 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Location: http://localhost/login
Location: http://localhost/login
< Connection: keep-alive
Connection: keep-alive
< Cache-Control: private, must-revalidate
Cache-Control: private, must-revalidate
< pragma: no-cache
pragma: no-cache
< expires: -1
expires: -1
< Set-Cookie: XSRF-TOKEN=eyJpdiI6IkRoVHFteFhwQnZ5TWhEUkpHRDh5dXc9PSIsInZhbHVlIjoia292eEMrV0VjbW9ZTHg2SmZnQW9aOG1WcnBuUThRK0pITXVIbTR5N1kzTDhFVHFGeDhENzROQTNwVDczMks4ZDc1cnNrcWpWOURVUHhURm84YWxQeEh2RVVSS1pvRGk1VStUb2FNNHY3UDNwMHRZOG5zTU1WYVJvZTdjMGpmR2siLCJtYWMiOiI5MzdiOWJjZjE2NmE4NDExYTIwN2ZhOGU0MTdkN2RlZWI0MGZhYTY0MTBhNGRlYzA5Mjk5OWU0ZDNjZDA4ZDczIn0%3D; expires=Tue, 30-Mar-2021 11:09:50 GMT; Max-Age=7200; path=/; samesite=lax
Set-Cookie: XSRF-TOKEN=eyJpdiI6IkRoVHFteFhwQnZ5TWhEUkpHRDh5dXc9PSIsInZhbHVlIjoia292eEMrV0VjbW9ZTHg2SmZnQW9aOG1WcnBuUThRK0pITXVIbTR5N1kzTDhFVHFGeDhENzROQTNwVDczMks4ZDc1cnNrcWpWOURVUHhURm84YWxQeEh2RVVSS1pvRGk1VStUb2FNNHY3UDNwMHRZOG5zTU1WYVJvZTdjMGpmR2siLCJtYWMiOiI5MzdiOWJjZjE2NmE4NDExYTIwN2ZhOGU0MTdkN2RlZWI0MGZhYTY0MTBhNGRlYzA5Mjk5OWU0ZDNjZDA4ZDczIn0%3D; expires=Tue, 30-Mar-2021 11:09:50 GMT; Max-Age=7200; path=/; samesite=lax
< Set-Cookie: laravel_session=eyJpdiI6IlFWcUwwQTJGbWpLcWZ5WHk4WWdJT3c9PSIsInZhbHVlIjoibWx1YUk5blA5azFWQnBJbEt2aGp4R2V6bzBKRmdVdEw5eHAyYXFMSktxWEQ3dGZQZEYyU1Q2aUxjb0NFTUxKRHpGWTNQeGxiSmRuMDNpcExyYVpIN2doYWNTZy9VWC9od2c0SFBDT08yTnd3enRucGZXbkkvYlVteUhkdGZyTWUiLCJtYWMiOiI3MzdjZDk4N2JmYWJlNWMwMjg3OGYzNWE4MjFlM2ZhM2JmZmNlODk0MzI5YjdmMDI2MmMwNmViODgzM2ZiOWJkIn0%3D; expires=Tue, 30-Mar-2021 11:09:50 GMT; Max-Age=7200; path=/; httponly; samesite=lax
Set-Cookie: laravel_session=eyJpdiI6IlFWcUwwQTJGbWpLcWZ5WHk4WWdJT3c9PSIsInZhbHVlIjoibWx1YUk5blA5azFWQnBJbEt2aGp4R2V6bzBKRmdVdEw5eHAyYXFMSktxWEQ3dGZQZEYyU1Q2aUxjb0NFTUxKRHpGWTNQeGxiSmRuMDNpcExyYVpIN2doYWNTZy9VWC9od2c0SFBDT08yTnd3enRucGZXbkkvYlVteUhkdGZyTWUiLCJtYWMiOiI3MzdjZDk4N2JmYWJlNWMwMjg3OGYzNWE4MjFlM2ZhM2JmZmNlODk0MzI5YjdmMDI2MmMwNmViODgzM2ZiOWJkIn0%3D; expires=Tue, 30-Mar-2021 11:09:50 GMT; Max-Age=7200; path=/; httponly; samesite=lax

< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost/login'
* Found bundle for host localhost: 0x5631ed404970 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 80 (#0)
> HEAD /login HTTP/1.1
> Host: localhost
> User-Agent: curl/7.75.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: nginx/1.19.8
Server: nginx/1.19.8
< Date: Tue, 30 Mar 2021 09:09:50 GMT
Date: Tue, 30 Mar 2021 09:09:50 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 153
Content-Length: 153
< Connection: keep-alive
Connection: keep-alive

< 
* Connection #0 to host localhost left intact

Talking directly with the container:
* Connected to localhost (127.0.0.1) port 8000 (#0)
> HEAD /librenms HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.75.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Server: nginx
Server: nginx
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
Connection: keep-alive
< Keep-Alive: timeout=10
Keep-Alive: timeout=10
< Cache-Control: no-cache, private
Cache-Control: no-cache, private
< Date: Tue, 30 Mar 2021 10:09:23 GMT
Date: Tue, 30 Mar 2021 10:09:23 GMT
< Location: http://localhost:8000/login
Location: http://localhost:8000/login
< Set-Cookie: XSRF-TOKEN=eyJpdiI6Ii81RU9PcDdzMHlLSHhSMjBINkNuU1E9PSIsInZhbHVlIjoiTGhKN0hndEk1ZVRMOURBdzZjaUFqNFhUb3hPTlBrUTNQRmtWd3hnTFVYOURkTlpWMFNCZWZBUmZOQU1zRU13QXJSVDJaS25TNTdUMzlYa3ROMVRWQVh3NmpYamZKTEFQRFRpd3RvRWc0V2lWOUM2cU5DOGsrRVY2ZTNlR3VNZmciLCJtYWMiOiIzYzIyMWRmNTEyNTc5ODYxNjFkMGZmMzRlYmNiYTFlNzU3YjM4ZjhjMjMxZWU3Zjc0MTAwMDkxYzZjNTE3NDgyIn0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; samesite=lax
Set-Cookie: XSRF-TOKEN=eyJpdiI6Ii81RU9PcDdzMHlLSHhSMjBINkNuU1E9PSIsInZhbHVlIjoiTGhKN0hndEk1ZVRMOURBdzZjaUFqNFhUb3hPTlBrUTNQRmtWd3hnTFVYOURkTlpWMFNCZWZBUmZOQU1zRU13QXJSVDJaS25TNTdUMzlYa3ROMVRWQVh3NmpYamZKTEFQRFRpd3RvRWc0V2lWOUM2cU5DOGsrRVY2ZTNlR3VNZmciLCJtYWMiOiIzYzIyMWRmNTEyNTc5ODYxNjFkMGZmMzRlYmNiYTFlNzU3YjM4ZjhjMjMxZWU3Zjc0MTAwMDkxYzZjNTE3NDgyIn0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; samesite=lax
< Set-Cookie: laravel_session=eyJpdiI6IlpsNFEwQU02V2FuOVZ4TXNwdld6OGc9PSIsInZhbHVlIjoicVo5UlFSN04yUnpLckZKR0FWSUxnaEpvRlF0SldvQXlTNXczS2llY2lucTR3SVRtYVgyaGdMVHB4YTBnUjZYQ1BtZWZobnJMOER6MmZxd2w5K2NVdWVIMnlwNjRDYW5tdnhYWndTK1RIN1FDYjlmK3JKV013N251SFMwZEEzLzIiLCJtYWMiOiJkZDhkMzg5MTE4MDI5N2JmNzQ1NzY4NDM1ODQxMjE4NWM2ODk5ZmQ2NzMyZjc1NDY2MGE0YmM1N2MyOWUxMmY2In0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; httponly; samesite=lax
Set-Cookie: laravel_session=eyJpdiI6IlpsNFEwQU02V2FuOVZ4TXNwdld6OGc9PSIsInZhbHVlIjoicVo5UlFSN04yUnpLckZKR0FWSUxnaEpvRlF0SldvQXlTNXczS2llY2lucTR3SVRtYVgyaGdMVHB4YTBnUjZYQ1BtZWZobnJMOER6MmZxd2w5K2NVdWVIMnlwNjRDYW5tdnhYWndTK1RIN1FDYjlmK3JKV013N251SFMwZEEzLzIiLCJtYWMiOiJkZDhkMzg5MTE4MDI5N2JmNzQ1NzY4NDM1ODQxMjE4NWM2ODk5ZmQ2NzMyZjc1NDY2MGE0YmM1N2MyOWUxMmY2In0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; httponly; samesite=lax

curl -v -LI --insecure  http://localhost:8000/librenms
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8000/login'
* Found bundle for host localhost: 0x55c4e1f6a970 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8000 (#0)
> HEAD /login HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.75.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
< Server: nginx
Server: nginx
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
Connection: keep-alive
< Keep-Alive: timeout=10
Keep-Alive: timeout=10
< Cache-Control: no-cache, private
Cache-Control: no-cache, private
< date: Tue, 30 Mar 2021 10:09:23 GMT
date: Tue, 30 Mar 2021 10:09:23 GMT
< Set-Cookie: XSRF-TOKEN=eyJpdiI6IklCTHRYWXBhcS91Ymx4cnVrd0h2NkE9PSIsInZhbHVlIjoiWTEwQ2dxdkpsdjdEckQwVjNyK1pKSDR3bXVwNEwySmlxTUJLQXJaajlzd3diaGdGekFiaW9OQmdudTJRVVdLanNxR05Mczd6OTQweXlJUEtjd2RXVXh1WS9lb2ljeTR6MElkcHFLak5ZNGxWb1hzbVlMSFRTeGxLdjF6UStZU0EiLCJtYWMiOiJmNjg1ZWI4MjlhMjRhYjk4Zjg3ZjgyYmNhNTVjMmU3Mjc5YWQ2OTgzOWE2YmJhN2I3YTA1YzEyNjBlZDc4M2M4In0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; samesite=lax
Set-Cookie: XSRF-TOKEN=eyJpdiI6IklCTHRYWXBhcS91Ymx4cnVrd0h2NkE9PSIsInZhbHVlIjoiWTEwQ2dxdkpsdjdEckQwVjNyK1pKSDR3bXVwNEwySmlxTUJLQXJaajlzd3diaGdGekFiaW9OQmdudTJRVVdLanNxR05Mczd6OTQweXlJUEtjd2RXVXh1WS9lb2ljeTR6MElkcHFLak5ZNGxWb1hzbVlMSFRTeGxLdjF6UStZU0EiLCJtYWMiOiJmNjg1ZWI4MjlhMjRhYjk4Zjg3ZjgyYmNhNTVjMmU3Mjc5YWQ2OTgzOWE2YmJhN2I3YTA1YzEyNjBlZDc4M2M4In0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; samesite=lax
< Set-Cookie: laravel_session=eyJpdiI6ImtocDNUMy9aTWJnckQyNjhxKzJMTmc9PSIsInZhbHVlIjoiUVFwR2h6RHcyUDQyNkJPdkR4dHZUUmVWM0R6RU9yV2dQeUw0U0kyK3B1MlFmZHZWOGdrdjVoeHkzQ0JOcWh4MEtRWHdUa0Q5RitIOXZQMHV3UnVuR0xuOTZBT3lmMTgrRUJvSTk1eW54ZHpDYmtaQm4vNVlDR3c3b1doWWNveUEiLCJtYWMiOiIyZWRlZTE4NGNmMGYxNTRjYWIzNzc2OGVjZmIxOWViZGJiNmVkNmY1MTZjZjQ5ODJhZjE5ZGRkY2U1NTA1YTAwIn0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; httponly; samesite=lax
Set-Cookie: laravel_session=eyJpdiI6ImtocDNUMy9aTWJnckQyNjhxKzJMTmc9PSIsInZhbHVlIjoiUVFwR2h6RHcyUDQyNkJPdkR4dHZUUmVWM0R6RU9yV2dQeUw0U0kyK3B1MlFmZHZWOGdrdjVoeHkzQ0JOcWh4MEtRWHdUa0Q5RitIOXZQMHV3UnVuR0xuOTZBT3lmMTgrRUJvSTk1eW54ZHpDYmtaQm4vNVlDR3c3b1doWWNveUEiLCJtYWMiOiIyZWRlZTE4NGNmMGYxNTRjYWIzNzc2OGVjZmIxOWViZGJiNmVkNmY1MTZjZjQ5ODJhZjE5ZGRkY2U1NTA1YTAwIn0%3D; expires=Tue, 30-Mar-2021 12:09:23 GMT; Max-Age=7200; path=/; httponly; samesite=lax

LIBRENMS CONTAINER LOG:
[cont-init.d] 04-svc-main.sh: exited 0.
[cont-init.d] 05-svc-dispatcher.sh: executing... 
[cont-init.d] 05-svc-dispatcher.sh: exited 0.
[cont-init.d] 06-svc-syslogng.sh: executing... 
[cont-init.d] 06-svc-syslogng.sh: exited 0.
[cont-init.d] 07-svc-cron.sh: executing... 
Creating LibreNMS daily.sh cron task with the following period fields: 15 0 * * *
Fixing crontabs permissions...
[cont-init.d] 07-svc-cron.sh: exited 0.
[cont-init.d] ~-socklog: executing... 
[cont-init.d] ~-socklog: exited 0.
[cont-init.d] done.
[services.d] starting services
crond: crond (busybox 1.31.1) started, log level 8
[services.d] done.
2021/03/30 09:08:37 [notice] 862#862: using the "epoll" event method
2021/03/30 09:08:37 [notice] 862#862: nginx/1.18.0
2021/03/30 09:08:37 [notice] 862#862: OS: Linux 5.11.10-arch1-1
2021/03/30 09:08:37 [notice] 862#862: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/03/30 09:08:37 [notice] 862#862: start worker processes
2021/03/30 09:08:37 [notice] 862#862: start worker process 893
2021/03/30 09:08:37 [notice] 862#862: start worker process 894
2021/03/30 09:08:37 [notice] 862#862: start worker process 895
2021/03/30 09:08:37 [notice] 862#862: start worker process 898
2021/03/30 09:08:37 [notice] 862#862: start worker process 900
2021/03/30 09:08:37 [notice] 862#862: start worker process 920
2021/03/30 09:08:37 [notice] 862#862: start worker process 948
2021/03/30 09:08:37 [notice] 862#862: start worker process 973
[30-Mar-2021 09:08:37] NOTICE: fpm is running, pid 861
[30-Mar-2021 09:08:37] NOTICE: ready to handle connections
172.18.0.4 - - [30/Mar/2021:09:09:19 +0000] "GET /librenms HTTP/1.0" 302 350 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.105 Safari/537.36"

NGINX CONTAINER LOG AT REQUEST:
172.18.0.4 - - [30/Mar/2021:09:07:43 +0000] "GET /librenms HTTP/1.0" 302 350 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.105 Safari/537.36"

[duhow]

does this help?
https://serverfault.com/a/870620

[keeperofdakeys]

As far as I can see the base_url handing is totally broken.

• Neither BASE_URL, LIBRENMS_BASE_URL or APP_URL in a podman/docker env file have any effect.
• Attempting to set the $config['base_url'] in the config dir that's mapped into /data in the container appears to have no effect.

As a work around I've found these two NGINX options for proxy_pass will make it work. This passes the host header and correct protocol to librenms in the container, allowing it to successfully build the URLs correctly.

proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto "https";

[cu]

As far as I can see the base_url handing is totally broken.

I had the same experience, first discovered when visiting /validate in a new container. Other configuration settings seem to work just fine but something somewhere deep in the PHP is overriding $config['base_url'] to /. I attempted to trace the problem but I don't have a good method for debugging PHP inside a container right now.

I came up with a couple of ugly workarounds but decided against them because having a base_url of / doesn't seem to cause any issues when LibreNMS is hosted at the root of the host.

[murrant]

LibreNMS in a subdirectory does not work very well because there is a mix of modern and legacy code interacting with URLs. This is an upstream bug, if you have the will to work on it there, I think a lot of people would appreciate it.

I suggest using a subdomain ala librenms.domain.com instead if you can.