diff --git a/src/pages/selfhosted/configuration-files.mdx b/src/pages/selfhosted/configuration-files.mdx index b3e6a551..187709ee 100644 --- a/src/pages/selfhosted/configuration-files.mdx +++ b/src/pages/selfhosted/configuration-files.mdx @@ -143,7 +143,12 @@ netbird-server: labels: - traefik.enable=true # gRPC router (needs h2c backend for HTTP/2 cleartext) - - traefik.http.routers.netbird-grpc.rule=Host(`netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`)) + # The /management.ProxyService/ path is only required if the reverse proxy + # container (netbirdio/reverse-proxy) connects through Traefik — i.e., it + # runs on a separate host or a different Docker network. If the proxy is on + # the same Docker network as netbird-server, it connects directly and this + # path prefix can be omitted. + - traefik.http.routers.netbird-grpc.rule=Host(`netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`)) - traefik.http.routers.netbird-grpc.entrypoints=websecure - traefik.http.routers.netbird-grpc.tls=true - traefik.http.routers.netbird-grpc.tls.certresolver=letsencrypt @@ -192,7 +197,7 @@ When using the built-in Traefik, the `netbird-server` service uses two routers t | Router | Path Prefixes | Backend Service | Purpose | |--------|---------------|-----------------|---------| -| `netbird-grpc` | `/signalexchange.SignalExchange/`, `/management.ManagementService/` | `netbird-server-h2c` (h2c scheme) | gRPC traffic for signal exchange and management API. Uses HTTP/2 cleartext (h2c) backend because gRPC requires HTTP/2. | +| `netbird-grpc` | `/signalexchange.SignalExchange/`, `/management.ManagementService/`, `/management.ProxyService/` | `netbird-server-h2c` (h2c scheme) | gRPC traffic for signal exchange, management API, and the [Reverse Proxy feature](/manage/reverse-proxy). Uses HTTP/2 cleartext (h2c) backend because gRPC requires HTTP/2. The `/management.ProxyService/` path is only needed if the reverse proxy container connects through Traefik (see comment in the snippet above). | | `netbird-backend` | `/relay`, `/ws-proxy/`, `/api`, `/oauth2` | `netbird-server` (http scheme) | HTTP traffic for relay connections, WebSocket proxying, REST API, and OAuth2/OIDC endpoints. | The dashboard router has `priority=1` (lowest), so it acts as a catch-all for requests that don't match the more specific server routes. diff --git a/src/pages/selfhosted/migration/enable-reverse-proxy.mdx b/src/pages/selfhosted/migration/enable-reverse-proxy.mdx index 8162974c..359853fa 100644 --- a/src/pages/selfhosted/migration/enable-reverse-proxy.mdx +++ b/src/pages/selfhosted/migration/enable-reverse-proxy.mdx @@ -29,8 +29,8 @@ If your current deployment uses a reverse proxy other than Traefik, you'll need ### What stays the same -- Your existing `netbird-server`, `dashboard`, `signal`, and `relay` services are unchanged -- Your `management.json` and other configuration files require no modifications — **unless** you use an external identity provider (not the embedded IdP). See [Configure SSO for external identity providers](#configure-sso-for-external-identity-providers) below. +- Your existing NetBird services are unchanged +- Your configuration files (`config.yaml` for combined setup, `management.json` for multi-container setup) require no modifications - **unless** you use an external identity provider (not the embedded IdP). See [Configure SSO for external identity providers](#configure-sso-for-external-identity-providers) below. - Existing peers, networks, and access policies are unaffected ## Prerequisites @@ -38,14 +38,18 @@ If your current deployment uses a reverse proxy other than Traefik, you'll need Before starting, ensure you have: - **Traefik** as your reverse proxy (see [Why Traefik is required](#why-traefik-is-required) above) -- **Latest `netbird-server` image** - pull the latest version to ensure the management CLI supports token creation +- **Latest NetBird images** - pull the latest version to ensure the management server and CLI support the reverse proxy feature and token creation - **A domain for the proxy** - e.g., `proxy.example.com`. Service subdomains will be created under this domain (e.g., `myapp.proxy.example.com`) - **Wildcard DNS capability** - ability to create a `*.proxy.example.com` DNS record pointing to your server +- **Port 443 accessible** - the proxy needs this for ACME TLS-ALPN-01 challenges (certificate provisioning) + + +This guide covers both the **combined container** setup (`netbirdio/netbird-server`, the default for new deployments) and the **multi-container** setup (separate `management`, `signal`, and `relay` images). Where commands or configuration differ between the two setups, both variants are shown. + The proxy domain **must not** be a subdomain of your NetBird management domain. For example, if your management server is at `netbird.example.com`, do not use `proxy.netbird.example.com`. Use a separate subdomain like `proxy.example.com` instead. Using a subdomain of the management domain causes TLS and routing conflicts between the proxy and management services. -- **Port 443 accessible** - the proxy needs this for ACME TLS-ALPN-01 challenges (certificate provisioning) ## Migration steps @@ -58,16 +62,26 @@ cd netbird-backup-$(date +%Y%m%d) # Backup configuration files cp ../docker-compose.yml . -cp ../management.json . 2>/dev/null +cp ../config.yaml . 2>/dev/null # Combined container setup +cp ../management.json . 2>/dev/null # Multi-container setup cp ../*.env . 2>/dev/null || echo "No .env files found" ``` ### Step 2: Generate a proxy access token -The proxy authenticates with the management server using an access token. Generate one using the management CLI: +The proxy authenticates with the management server using an access token. Generate one using the server CLI. + +**Combined container** (`netbirdio/netbird-server`): + +```bash +docker exec -it netbird-server /go/bin/netbird-server token create \ + --name "my-proxy" --config /config.yaml +``` + +**Multi-container** (separate `netbirdio/management` image): ```bash -docker exec -it netbird-server netbird-mgmt token create --name "my-proxy" +docker exec -it netbird-management /go/bin/netbird-mgmt token create --name "my-proxy" ``` This outputs a token in the format `nbx_...` (40 characters). **Save the token immediately** - it is only displayed once. The management server stores only a SHA-256 hash. @@ -75,27 +89,33 @@ This outputs a token in the format `nbx_...` (40 characters). **Save the token i You can manage tokens later with: ```bash -# List all tokens -docker exec -it netbird-server netbird-mgmt token list +# List all tokens (combined container) +docker exec -it netbird-server /go/bin/netbird-server token list \ + --config /config.yaml + +# List all tokens (multi-container) +docker exec -it netbird-management /go/bin/netbird-mgmt token list + +# Revoke a token by ID (combined container) +docker exec -it netbird-server /go/bin/netbird-server token revoke \ + --config /config.yaml -# Revoke a token by ID -docker exec -it netbird-server netbird-mgmt token revoke +# Revoke a token by ID (multi-container) +docker exec -it netbird-management /go/bin/netbird-mgmt token revoke ``` ### Step 3: Add the proxy service to docker-compose.yml -Add the following service to your `docker-compose.yml`. Replace the placeholder values with your actual domains: +Add the following service to your `docker-compose.yml`. Adjust the `depends_on` value to match your management service name: ```yaml proxy: image: netbirdio/reverse-proxy:latest container_name: netbird-proxy - extra_hosts: - - "netbird.example.com:172.30.0.10" restart: unless-stopped networks: [netbird] depends_on: - - netbird-server + - netbird-server # Use "management" for multi-container setup env_file: - ./proxy.env volumes: @@ -123,21 +143,41 @@ volumes: netbird_proxy_certs: ``` -Replace `netbird.example.com` in the `extra_hosts` entry with your actual NetBird management domain. This hairpin NAT fix ensures the proxy can reach Traefik's static IP within the Docker network. +Then create a `proxy.env` file with the proxy configuration. -Then create a `proxy.env` file with the proxy configuration: +**Combined container** (`netbirdio/netbird-server`): ```bash NB_PROXY_DOMAIN=proxy.example.com NB_PROXY_TOKEN=nbx_your_token_here -NB_PROXY_MANAGEMENT_ADDRESS=https://netbird.example.com:443 +NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80 +NB_PROXY_ALLOW_INSECURE=true +NB_PROXY_ADDRESS=:8443 +NB_PROXY_ACME_CERTIFICATES=true +NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 +NB_PROXY_CERTIFICATE_DIRECTORY=/certs +``` + +**Multi-container** (separate `netbirdio/management` image): + +```bash +NB_PROXY_DOMAIN=proxy.example.com +NB_PROXY_TOKEN=nbx_your_token_here +NB_PROXY_MANAGEMENT_ADDRESS=http://management:33073 +NB_PROXY_ALLOW_INSECURE=true NB_PROXY_ADDRESS=:8443 NB_PROXY_ACME_CERTIFICATES=true NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 NB_PROXY_CERTIFICATE_DIRECTORY=/certs ``` -Replace `proxy.example.com` with your proxy domain and `netbird.example.com` with your management domain. +Replace `proxy.example.com` with your proxy domain. The `NB_PROXY_MANAGEMENT_ADDRESS` must match the Docker Compose service name and port of your management server - `netbird-server:80` for the combined container, or `management:33073` for the multi-container Traefik setup. + + +The proxy connects to the management server directly over the Docker network rather than through Traefik. This avoids the need to route the `/management.ProxyService/` gRPC service through your reverse proxy and eliminates hairpin NAT issues. The `NB_PROXY_ALLOW_INSECURE=true` setting is safe here because the traffic never leaves the Docker network. + +If your proxy and management server run on **separate hosts** and cannot communicate over a shared Docker network, see [Connecting through Traefik](#connecting-through-traefik-instead-of-docker-network) below. + The Traefik labels configure a **TCP router** that: - Catches any request not matched by higher-priority HTTP routers via `HostSNI(*)` (wildcard) @@ -152,7 +192,7 @@ The `HostSNI(*)` rule acts as a catch-all for any domain not matched by the exis ### Step 4: Set up DNS records -Create two DNS records pointing to the server running your NetBird stack — one for the base proxy domain and one wildcard for service subdomains: +Create two DNS records pointing to the server running your NetBird stack - one for the base proxy domain and one wildcard for service subdomains: | Type | Name | Content | |------|------|---------| @@ -194,7 +234,7 @@ If the domain appears, the proxy is connected and ready. You can now [create you ### Who this applies to -This section applies to deployments using a **standalone external identity provider** (Auth0, Okta, Keycloak, Zitadel, etc.) instead of the built-in embedded IdP (Dex). If you deployed using the quickstart script with default settings, you are using the embedded IdP and can skip this section. +This section applies to **multi-container** deployments using a **standalone external identity provider** (Auth0, Okta, Keycloak, Zitadel, etc.) instead of the built-in embedded IdP (Dex). If you are running the combined container or deployed using the quickstart script with default settings, you are using the embedded IdP and can skip this section - the callback is registered automatically. ### Why this is needed @@ -206,16 +246,17 @@ If you want to keep using your current external identity provider, follow these #### Step 1: Add callback URL to management.json -Add the `AuthCallbackURL` field to the `HttpConfig` section of your `management.json`: +Add the `AuthCallbackURL` and `AuthClientID` fields to the `HttpConfig` section of your `management.json`: ```json "HttpConfig": { ...existing fields..., + "AuthClientID": "", "AuthCallbackURL": "https:///api/reverse-proxy/callback" } ``` -Replace `` with your NetBird management server domain (the same domain used for the dashboard). +Replace `` with your NetBird management server domain (the same domain used for the dashboard). Replace `` with the OAuth2 client ID from your identity provider (the same client ID used for the dashboard application). #### Step 2: Register callback in your IdP @@ -242,12 +283,12 @@ Where to find this setting in common providers: Restart the management service to pick up the configuration change: ```bash -docker compose restart netbird-management +docker compose restart management ``` ### Option B: Migrate to the embedded IdP (recommended) -The embedded IdP (Dex) handles the reverse proxy callback registration automatically — no manual configuration needed. If you want a simpler setup, consider migrating to the embedded IdP. +The embedded IdP (Dex) handles the reverse proxy callback registration automatically - no manual configuration needed. If you want a simpler setup, consider migrating to the embedded IdP. With the embedded IdP, external identity providers can still be used as **connectors** alongside local authentication. This means your users can continue to sign in with their existing accounts (Google, Okta, Keycloak, etc.) while the embedded IdP manages the OIDC layer. @@ -268,6 +309,94 @@ After configuring SSO for your external identity provider, verify it works: If the redirect fails or you see an error from your IdP, double-check that the callback URL is correctly registered in both `management.json` and your identity provider's settings. +## Connecting through Traefik instead of Docker network + +If your proxy container cannot reach the management container directly - for example, if they run on **separate hosts** - you can route the proxy's management connection through Traefik instead. This requires three additional configuration steps. + +### 1. Add the ProxyService gRPC route + +The proxy communicates with the management server over two gRPC services: `ManagementService` and `ProxyService`. Both paths must be routed through Traefik. Find the existing gRPC router label in your `docker-compose.yml` - in a standard deployment this is `traefik.http.routers.netbird-grpc` - and add the `ProxyService` path prefix: + +``` +traefik.http.routers.netbird-grpc.rule=Host(`netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`)) +``` + +Without the `/management.ProxyService/` route, the proxy will fail to register with the management server. + +### 2. Fix the Traefik container IP for hairpin NAT + +When the proxy connects to your management domain through Traefik, the DNS resolution inside the Docker network must point to the Traefik container. This is done via an `extra_hosts` entry in the proxy service, but the IP address must match Traefik's actual IP. + +To ensure a stable IP, assign a static IP to the Traefik container within your Docker network: + +```yaml +# In your docker-compose.yml + +networks: + netbird: + driver: bridge + ipam: + config: + - subnet: 172.30.0.0/24 + gateway: 172.30.0.1 + +services: + traefik: + # ...existing traefik config... + networks: + netbird: + ipv4_address: 172.30.0.10 +``` + +Then add the `extra_hosts` entry to the proxy service referencing that IP: + +```yaml + proxy: + # ...existing proxy config... + extra_hosts: + - "netbird.example.com:172.30.0.10" +``` + +Replace `netbird.example.com` with your actual management domain. + +### 3. Increase Traefik's idle timeout for gRPC + +Traefik's default idle timeout (180 seconds) is too short for the long-lived gRPC streams used between the proxy and management server. Without increasing it, the proxy will report connection timeout errors and the dashboard may show the proxy agent as offline. + +Add the following to your Traefik static configuration: + +```yaml +# In traefik.yml +entryPoints: + websecure: + address: ":443" + transport: + respondingTimeouts: + idleTimeout: "0" +``` + +Or as a command-line argument: + +```yaml +# In docker-compose.yml +services: + traefik: + command: + # ...existing args... + - "--entrypoints.websecure.transport.respondingTimeouts.idleTimeout=0" +``` + +Finally, update `proxy.env` to connect through Traefik and remove `NB_PROXY_ALLOW_INSECURE`: + +```bash +NB_PROXY_MANAGEMENT_ADDRESS=https://netbird.example.com:443 +# Do NOT set NB_PROXY_ALLOW_INSECURE when connecting over TLS through Traefik +``` + + +If you do not assign a static IP to Traefik, Docker may assign a different IP on container restart, and the `extra_hosts` entry will silently point to the wrong address. Always configure a fixed subnet and static IP when using this approach. + + ## For users not on Traefik If your self-hosted deployment currently uses Nginx, Caddy, or another reverse proxy, you'll need to switch to Traefik before enabling the Reverse Proxy feature. See the [Traefik setup instructions](/selfhosted/reverse-proxy#traefik) for a step-by-step guide on configuring Traefik for your NetBird deployment. @@ -276,7 +405,7 @@ If your self-hosted deployment currently uses Nginx, Caddy, or another reverse p | Variable | Required | Description | Default | |----------|----------|-------------|---------| -| `NB_PROXY_TOKEN` | Yes | Access token generated via `netbird-mgmt token create`. The proxy refuses to start without it. | - | +| `NB_PROXY_TOKEN` | Yes | Access token generated via `netbird-server token create` (combined) or `netbird-mgmt token create` (multi-container). The proxy refuses to start without it. | - | | `NB_PROXY_DOMAIN` | Yes | Base domain for this proxy instance (e.g., `proxy.example.com`). Determines the domain available for services. | - | | `NB_PROXY_MANAGEMENT_ADDRESS` | No | URL of your NetBird management server. The proxy connects via gRPC to register itself. | `https://api.netbird.io:443` | | `NB_PROXY_ADDRESS` | No | Address the proxy listens on. | `:8443` (Docker), `:443` (binary) | @@ -342,8 +471,15 @@ docker compose up -d You can also revoke the proxy token to prevent the proxy from reconnecting: ```bash -docker exec -it netbird-server netbird-mgmt token list -docker exec -it netbird-server netbird-mgmt token revoke +# Combined container +docker exec -it netbird-server /go/bin/netbird-server token list \ + --config /config.yaml +docker exec -it netbird-server /go/bin/netbird-server token revoke \ + --config /config.yaml + +# Multi-container +docker exec -it netbird-management /go/bin/netbird-mgmt token list +docker exec -it netbird-management /go/bin/netbird-mgmt token revoke ``` ## Additional resources diff --git a/src/pages/selfhosted/reverse-proxy.mdx b/src/pages/selfhosted/reverse-proxy.mdx index 6b918856..a7f5799f 100644 --- a/src/pages/selfhosted/reverse-proxy.mdx +++ b/src/pages/selfhosted/reverse-proxy.mdx @@ -46,6 +46,7 @@ All reverse proxy configurations must route the following endpoints: | `/api/*` | HTTP | management:80 | REST API | | `/ws-proxy/management*` | WebSocket | management:80 | WebSocket upgrade required | | `/management.ManagementService/*` | gRPC | management:80 | HTTP/2 (h2c) required | +| `/management.ProxyService/*` | gRPC | management:80 | HTTP/2 (h2c) required. Only needed if using the [Reverse Proxy feature](/manage/reverse-proxy). | | `/oauth2/*` | HTTP | management:80 | Embedded IdP | | `/*` | HTTP | dashboard:80 | Catch-all for dashboard | @@ -184,7 +185,7 @@ services: - traefik.http.routers.netbird-mgmt-ws.service=netbird-mgmt-ws - traefik.http.services.netbird-mgmt-ws.loadbalancer.server.port=80 # Management gRPC router - - traefik.http.routers.netbird-mgmt-grpc.rule=Host(`netbird.example.com`) && PathPrefix(`/management.ManagementService/`) + - traefik.http.routers.netbird-mgmt-grpc.rule=Host(`netbird.example.com`) && (PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`)) - traefik.http.routers.netbird-mgmt-grpc.entrypoints=websecure - traefik.http.routers.netbird-mgmt-grpc.tls=true - traefik.http.routers.netbird-mgmt-grpc.tls.certresolver=letsencrypt @@ -817,6 +818,7 @@ location /management.ManagementService/ { grpc_send_timeout 1d; grpc_socket_keepalive on; } + ``` #### NPM running on host (not in Docker) @@ -912,6 +914,7 @@ location /management.ManagementService/ { grpc_send_timeout 1d; grpc_socket_keepalive on; } + ``` --- diff --git a/src/pages/selfhosted/sqlite-store.mdx b/src/pages/selfhosted/sqlite-store.mdx index 8f74555f..44c44239 100644 --- a/src/pages/selfhosted/sqlite-store.mdx +++ b/src/pages/selfhosted/sqlite-store.mdx @@ -56,7 +56,7 @@ Starting from version 0.24.0, NetBird Management provides a subcommand to facili The following commands assume you use the latest docker version with the compose plugin. If you have docker-compose installed as a standalone, please use docker-compose as a command. ```bash -$ docker compose exec management /go/bin/netbird-mgmt sqlite-migration +$ docker exec -it netbird-management /go/bin/netbird-mgmt sqlite-migration ``` This will produce an output similar to: ```bash @@ -100,7 +100,7 @@ To migrate from JSON file store to SQLite, follow these steps: ``` 4. Run the migration to SQLite subcommand: ```bash - docker compose exec management /go/bin/netbird-mgmt sqlite-migration upgrade --log-file console + docker exec -it netbird-management /go/bin/netbird-mgmt sqlite-migration upgrade --log-file console ``` 5. Enable SQLite by updating the management.json file and setting the `Engine` field to `sqlite` as the following example: ```json @@ -145,7 +145,7 @@ To rollback to the JSON file store, follow these steps: ``` 4. Run the migration to SQLite subcommand: ```bash - docker compose exec management /go/bin/netbird-mgmt sqlite-migration downgrade --log-file console + docker exec -it netbird-management /go/bin/netbird-mgmt sqlite-migration downgrade --log-file console ``` 5. Enable JSON file by updating the management.json file and setting the `Engine` field to `jsonfile` as the following example: ```json diff --git a/src/pages/selfhosted/troubleshooting.mdx b/src/pages/selfhosted/troubleshooting.mdx index 26463ab0..e1864c6c 100644 --- a/src/pages/selfhosted/troubleshooting.mdx +++ b/src/pages/selfhosted/troubleshooting.mdx @@ -119,7 +119,7 @@ Where you have the following types: `host` (local address), `srflx` (STUN reflex 1. Ensure port 80 is accessible for ACME challenge 2. Check Caddy logs: `docker compose logs caddy` 3. Verify the domain points to the correct IP -4. Manually trigger renewal: `docker compose exec caddy caddy reload` +4. Manually trigger renewal: `docker exec -it netbird-caddy caddy reload` ### Certificate errors with custom reverse proxy