500 Internal Server Error and 403 Forbidden

[SomnusVeritas]

What happened?

Hi! First of all, thank you for all the work! I'd love to see this software succeed.
Sadly, I have a lot of issues right after a fresh setup.
I'm hosting Wanderer on my Truenas Scale Server via Docker behind a Traefik reverse proxy.
Right after setup and initial start, everything seems to work fine. Every tab, the map and I can even create an account and upload a trail.
After a few clicks, I'm getting a 500 Internal Server Error seemingly at random and in some Tabs, that vary.
If I click around a bit more I'm getting a 403 Forbidden in every page. The Error stays until I restart the stack.
My reverse Proxy has no Middlewares set up.
I don't have these issues with Immich, SFTPgo, Nextcloud, copyparty, jellyfin, mastodon, tuwunel, navidrome, owncast and tinyauth, so I don't think it's a wrong traefik configuration.

docker-compose

`MEILI_MASTER_KEY`, `POCKET_KEY` and `UPLOAD_PASS` are all 128 Bit RSA Hexadecimal keys

services:
  search:
    container_name: wanderer-search
    image: getmeili/meilisearch:v1.11.3
    environment:
      MEILI_URL: http://search:7700
      MEILI_MASTER_KEY: $MEILI_MASTER_KEY
      MEILI_NO_ANALYTICS: "true"
      TZ: Europe/Berlin
    networks:
      - wanderer
    volumes:
      - ./meili/data/:/meili_data/data.ms
    restart: unless-stopped
    healthcheck:
      test: curl --fail http://localhost:7700/health || exit 1
      interval: 15s
      retries: 10
      start_period: 20s
      timeout: 10s
  db:
    container_name: wanderer-db
    image: flomp/wanderer-db
    depends_on:
      search:
        condition: service_healthy
    environment:
      MEILI_URL: http://search:7700
      MEILI_MASTER_KEY: $MEILI_MASTER_KEY
      POCKETBASE_ENCRYPTION_KEY: $POCKET_KEY
      ORIGIN: http://localhost:3000
      TZ: Europe/Berlin
    networks:
      - wanderer
    restart: unless-stopped
    volumes:
      - ./pocketbase/data:/pb_data
    healthcheck:
      test: wget --spider -q http://localhost:8090/health || exit 1
      interval: 15s
      retries: 10
      start_period: 20s
      timeout: 10s
  web:
    container_name: wanderer-web
    image: flomp/wanderer-web
    depends_on:
      search:
        condition: service_healthy
      db:
        condition: service_healthy
    environment:
      MEILI_URL: http://search:7700
      MEILI_MASTER_KEY: $MEILI_MASTER_KEY
      ORIGIN: https://wanderer.mydomain.tld
      BODY_SIZE_LIMIT: Infinity
      PUBLIC_POCKETBASE_URL: http://db:8090
      PUBLIC_DISABLE_SIGNUP: "false"
      UPLOAD_FOLDER: /app/uploads
      UPLOAD_USER: wanderer
      UPLOAD_PASSWORD: $UPLOAD_PASS
      PUBLIC_OVERPASS_API_URL: https://overpass-api.de
      #PUBLIC_VALHALLA_URL: https://valhalla1.openstreetmap.de
      PUBLIC_VALHALLA_URL: http://valhalla:8002
      PUBLIC_NOMINATIM_URL: https://nominatim.openstreetmap.org
      TZ: Europe/Berlin
    volumes:
      - ./wanderer/data/uploads:/app/uploads
    networks:
      - wanderer
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.wanderer.rule=Host(`wanderer.mydomain.tld`)
      - traefik.http.services.wanderer-service.loadbalancer.server.url=http://web:3000
    restart: unless-stopped
    healthcheck:
      test: curl --fail http://localhost:3000/ || exit 1
      interval: 15s
      retries: 10
      start_period: 20s
      timeout: 10s
  valhalla:
    container_name: valhalla
    image: ghcr.io/valhalla/valhalla-scripted:latest
    volumes:
      - ./valhalla/data/:/custom_files
    environment:
      - tile_urls=https://download.geofabrik.de/europe-latest.osm.pbf
      - use_tiles_ignore_pbf=True
      - force_rebuild=False
      - force_rebuild_elevation=False
      - build_elevation=True
      - build_admins=True
      - build_time_zones=True
networks:
  wanderer:
    driver: bridge
    external: false
  proxy:
    external: true

Version

v0.18.3

[slothful-vassal]

Hi,
Not sure if it is the reason of each issue, but origin in db container is wrong - should be the same as in web container.
And what do you mean with "128 Bit RSA Hexadecimal keys"? Please Check the requirements of the keys:
https://meilisearch.com/docs/learn/security/basic_security
https://pocketbase.io/docs/going-to-production/?utm_source=chatgpt.com#enable-settings-encryption
So, 128bit for pocketbasekey may be too short.

[SomnusVeritas]

Thanks for your quick answer! Changing the origin does not change the behavior, I tried that.
Meilie documentation requires a key that is "16 bytes long", which is 128 bits (32 characters).

The docker logs aren't helping, the only message that keeps repeating is "Message 'trail'/'list' has syntax error" but I saw in another issue that this is not relevant

How can I debug this issue?

[slothful-vassal]

Thanks for double-checking the secret keys.

Which origin are you using — the internal container address or the external URL? The origin should be set to the external URL.

For debugging purposes, please check the logs of all containers (Meili, DB, Web) as well as the browser’s developer tools.

[SomnusVeritas]

Origin is the external url and I can reach it normally. Since I didn't expose any port server-internally and only forwarded port 3000 with traefik, all my testing was done via my external IP.

All the logs looked fine, but I can check them again.

I'll check the browsers developer tools again next time I can (probably monday).

Just a suggestion, I could set up a fresh instance and let you click through everything

[slothful-vassal]

Yes, feel free to contact me on Discord if I need any private information to access your test instance.

[slothful-vassal]

You could also check the reverse proxy logs (Traefik).

[disgustipated]

if youre using CSP headers in your proxy these are the ones i found that needed set. this is my config for nginx. cloudflareinsights is only needed if youre routing through cloudflare. quite a few needed for the maps display

# CSP Header Vars
set $DEFAULT "default-src 'self' data:";
set $FRAME "frame-src 'self'";
set $FRAME "${FRAME} https://<<your domain>>/";
set $FRAMEANC "frame-ancestors 'self'";
set $SCRIPT "script-src 'self' 'unsafe-inline' 'unsafe-eval'";
set $SCRIPT "${SCRIPT} https://<<your domain>>/";
set $SCRIPT "${SCRIPT} https://static.cloudflareinsights.com/";
set $STYLE "style-src 'self' 'unsafe-inline' 'unsafe-eval' 'unsafe-hashes'";
set $IMG "img-src 'self' data:";
set $IMG "${IMG} https://api.dicebear.com/";
set $IMG "${IMG} https://upload.wikimedia.org/";
set $FONT "font-src 'self' data:";
set $CONNECT "connect-src 'self' data: blob:";
set $CONNECT "${CONNECT} https://<<your domain>>/";
set $CONNECT "${CONNECT} https://www.gstatic.com/draco/versioned/decoders/1.4.3/draco_wasm_wrapper.js";
set $CONNECT "${CONNECT} https://www.gstatic.com/draco/versioned/decoders/1.4.3/draco_decoder.wasm";
set $CONNECT "${CONNECT} https://api.maptiler.com/";
set $CONNECT "${CONNECT} https://tiles.openfreemap.org/";
set $CONNECT "${CONNECT} https://nominatim.openstreetmap.org/";
set $CONNECT "${CONNECT} https://maps.refuges.info/";
set $CONNECT "${CONNECT} https://*.openstreetmap.fr/";
set $CONNECT "${CONNECT} https://*.cartocdn.com/";
set $CONNECT "${CONNECT} https://*.opentopomap.org/";
set $CONNECT "${CONNECT} https://*.top-o-map.de/";
set $WORKERSRC "worker-src 'self' blob:";

add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
add_header Content-Security-Policy "${DEFAULT}; ${FRAME}; ${FRAMEANC}; ${SCRIPT}; ${STYLE}; ${IMG}; ${FONT}; ${CONNECT}; ${WORKERSRC};" always;

[SomnusVeritas]

Hi! So, I tested it via local access and everything seems to work fine! It probably is the combination of wanderer and traefik that causes these issues. I tested it by using all Headers that @disgustipated suggested but it did not work.
The Traefik logs are empty, sadly.
I checked the firefox errors and there were many (pasted below). I'm no webdev, so I have no clue if they are relevant.
I also tested it via Chromium and Firefox android and I'm getting the same errors.
Currently I'm running a few security headers, but the behavior was not different when running no middlewares.

Firefox errors

XML Parsing Error: syntax error Location: https://wanderer.mydomain.tld/profile/@marco Line Number 1, Column 1: @marco:1:1 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 Empty string passed to getElementById(). fields.js:503:35 GET https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js NS_ERROR_CORRUPTED_CONTENT

GET
https://wanderer.mydomain.tld/_app/immutable/chunks/Din1b9t9.js
NS_ERROR_CORRUPTED_CONTENT

GET
https://wanderer.mydomain.tld/_app/immutable/chunks/BgKWRFdS.js
NS_ERROR_CORRUPTED_CONTENT

Loading module from “https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js” was blocked because of a disallowed MIME type (“text/html”). trails
Loading module from “https://wanderer.mydomain.tld/_app/immutable/chunks/Din1b9t9.js” was blocked because of a disallowed MIME type (“text/html”). trails
Loading module from “https://wanderer.mydomain.tld/_app/immutable/chunks/BgKWRFdS.js” was blocked because of a disallowed MIME type (“text/html”). trails
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 13
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 11
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/bounding-box
[HTTP/2 403 6ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/bounding-box
[HTTP/2 403 3ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/filter
[HTTP/2 403 3ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/filter
[HTTP/2 403 5ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/filter
[HTTP/2 403 3ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/filter
[HTTP/2 403 5ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/api/v1/trail/filter
[HTTP/2 403 3ms]

XHRGET
https://wanderer.mydomain.tld/_app/version.json
[HTTP/2 403 3ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/_app/version.json
[HTTP/2 403 6ms]

XHRGET
https://wanderer.mydomain.tld/api/v1/category
[HTTP/2 403 4ms]

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data 5.BrmLeOzp.js:1:3480
XHRGET
https://wanderer.mydomain.tld/md/about.md
[HTTP/2 403 4ms]

GET
https://wanderer.mydomain.tld/_app/immutable/assets/empty_state_feed_dark.DWvezNLB.svg
[HTTP/2 403 4ms]

XHRGET
https://wanderer.mydomain.tld/api/v1/feed?page=2&perPage=10&sort=-created
[HTTP/2 403 6ms]

Uncaught (in promise) SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 13
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/_app/version.json
[HTTP/2 403 4ms]

TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 11
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487
XHRGET
https://wanderer.mydomain.tld/_app/version.json
[HTTP/2 403 3ms]

TypeError: error loading dynamically imported module: https://wanderer.mydomain.tld/_app/immutable/chunks/qNft9Grh.js
Immutable 14
async* https://wanderer.mydomain.tld/profile/@marco:38
promise callback* https://wanderer.mydomain.tld/profile/@marco:37
app.BxBlMyKo.js:2:9487

Security headers:

http:
  middlewares:
    security-headers:
      headers:
        customResponseHeaders:
          X-Content-Type-Options: "nosniff"
          X-Robots-Tag: "noindex, nofollow, nosnippet, noarchive"
          Referrer-Policy: "strict-origin-when-cross-origin"
          X-Frame-Options: "SAMEORIGIN"
          X-XSS-Protection: "1; mode=block"
          Content-Security-Policy: "frame-ancestors 'self'"
        sslRedirect: true
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        frameDeny: true
        contentTypeNosniff: true

[slothful-vassal]

Can you please check the traefik access logs (needs to be activated first) (https://doc.traefik.io/traefik/v2.0/observability/access-logs/)?

[SomnusVeritas]

My bad, here is the access log:

access.log
Client address and hostname redacted

[slothful-vassal]

1. Can you please send a response body of an 403 request?
2. Can you restart the container and send access logs of a state where the service is running fine?

[slothful-vassal]

3. sorry: pocketbase admin ui has further logs. can you please check?