Nextcloud intermittent loss of connection to database "Failed to connect to the database" with 502

[Extarys]

I tried setting Nextcloud with an external Nginx server and had a ton of 502 errors, thinking NPM and Nginx had trouble connecting to each other, and the logs weren't either present or useful.

One reddit user suggest I try the linuxserver docker image for nextcloud, which ships with nginx already preconfigured inside the Docekrfile.

When I set Nextcloud and enter the database address in the installer, it failed with Internal Server Error:

$ tail nextcloud/appdata/log/nginx/error.log 
#6 /config/www/nextcloud/lib/private/AppConfig" while reading response header from upstream, client: 10.89.1.2, server: _, request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "cloud.zzzzz.com"
2022/09/10 15:09:17 [error] 187#187: *152 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] could not translate host name "postgres" to address: Name does not resolve in /config/www/nextcloud/lib/private/DB/Connection.php:139
Stack trace:
#0 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1519): OC\DB\Connection->connect()
#1 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1041): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /config/www/nextcloud/lib/private/DB/Connection.php(261): Doctrine\DBAL\Connection->executeQuery()
#3 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\DB\Connection->executeQuery()
#4 /config/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(281): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /config/www/nextcloud/lib/private/AppConfig.php(411): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /config/www/nextcloud/lib/private/AppConfig" while reading response header from upstream, client: 10.89.1.2, server: _, request: "GET /ocs/v2.php/search/providers?from=%2Fapps%2Fdashboard%2F HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "cloud.zzzzz.com"

After a couple of refresh, it worked, but every 10 seconds I would get antoher 502 and when it works, some css files couldn't be loaded with the same error as above. Disabling cache makes the issue worse and most of the time the main page stops loading.

frontend/compose.yaml:

version: "3.7"
services:
  npm:
    container_name: npm
    image: 'docker.io/jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - target: 443
        published: 443 # Outside port
        mode: host
        protocol: tcp
      - target: 80
        published: 80 # Outside port
        mode: host
        protocol: tcp
      - '81:81'
    volumes:
      - ./npm/data:/data:Z
      - /opt/secrets/letsencrypt:/etc/letsencrypt:Z
    #network_mode: host
    networks:
      - pod-local
      - pod-public
  nextcloud:
    image: lscr.io/linuxserver/nextcloud:latest
    container_name: nextcloud
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Toronto
    volumes:
      - /opt/pods/frontend/nextcloud/appdata:/config:z
      - /opt/data/nextcloud/user_data:/data:z
    #ports:
    #  - 443:443
    restart: unless-stopped
    networks:
      - pod-local
      - pod-public
networks:
  pod-local:
    external: true
  pod-public:
    external: true

backend/compose.yaml:

version: "3"
services:
  postgres:
    container_name: postgres
    image: postgres:14
    restart: always
    security_opt:
      label=disable
    environment:
      - PODMAN_USERNS=keep-id
      - POSTGRES_USER=Elixir0702
      - POSTGRES_PASSWORD_FILE=/run/secrets/pgsql_root_pswd
    #ports:
    #  - "5432:5432"
    networks:
      - pod-local
    volumes:
      - /opt/pods/postgres/data:/var/lib/postgresql/data:Z
      #- /opt/pods/postgres/password.txt:/run/secrets/postgres-passwd:Z
    restart: unless-stopped
    secrets:
      - pgsql_root_pswd
networks:
  pod-local:
    external: true
secrets:
   pgsql_root_pswd:
     file: /secrets/pgsql_root_pswd

I had a similar issue when using NPM, Nginx and the Nextcloud:fpm image, but I would receive the generic 502 error from NPM directly so I thought it was a NPM <> Nginx issue, and no logs were present.

If I podman exec -it nextcloud bash and do pings to postgres, I had 50 pings with 0% packet loss, but that make sense as the IP doesn't need to be resolved at every ping.

I'm suspecting sometimes when the container asks podman for the IP of postgres, it doesn't reply in time, or somehow throttle the number of requests, as some DNS servers sometime do... but I'm not familiar enough with how podman works under the hood so this is a wild guess.

OS info:

$ cat /etc/redhat-release
Rocky Linux release 8.6 (Green Obsidian)

$ uname -a
Linux nabisipi 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Tue Aug 2 16:19:42 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ podman -v
podman version 4.1.1

[vrothberg]

Thanks for reaching out, @Extarys!

Can you share the output of podman info? @Luap99 @flouthoc PTAL

[Extarys]

I'll try to reproduce on my local machine to provide relevant logs today or tomorrow...

Plz don't be mad 😕 I spent 4 hours per day for 5 days trying to make podman works with nextcloud and nginx and just gave up - I started to feel like I couldn't accomplish anything 😂 . Switched to docker, and it's up and running without any issue. I'll of course try to redeploy with podman eventually.