Netbox 504 Error.

[cryansharp]

Does Netbox have any IP whitelisting in place by any chance? We recently switched to a new VPN solution and I can access our AWS Netbox instance via SSH with no problems, but continuously get a 504 error when trying to access the web interface. There is a nginx front end, but I'm not seeing any whitelisting or timeout issues there either. All routing and security groups appear to be fine on the AWS side. I'm wondering if nginx is receiving the request, but not getting a response from the Netbox backend.

[candlerb]

Does Netbox have any IP whitelisting in place by any chance?

No.

continuously get a 504 error when trying to access the web interface

What does the body of the 504 error say?

I'm wondering if nginx is receiving the request, but not getting a response from the Netbox backend.

tcpdump is your friend there. On the Netbox server, you can look for requests being sent from Nginx to Netbox like this:

tcpdump -i lo -nn -s0 -A tcp port 8001

(assuming that this is a standard install where nginx and netbox/gunicorn are running on the same VM)

Error 504 is gateway timeout. I wonder if your new "VPN solution" actually has some proxy of its own? Does Nginx show any logs for connections being received from the client which is on VPN?

The only other thing I can think of is that ALLOWED_HOSTS in configuration.py must include the hostname that the user is connecting as (e.g. if they enter "https://netbox.example.net" in their browser then "netbox.example.net" must be listed). But I don't think that would give a 504 error.

[cryansharp]

Thank you for the detailed information. The nginx error is:

2023/11/28 16:37:39 [error] 1096#1096: *24 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ::ffff:44.196.176.50, server: netbox.ironnet.io, request: "GET /login/?next=/favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8001/login/?next=/favicon.ico", host: "netbox.ironnet.io", referrer: "https://netbox.ironnet.io/login/"

There isn't much in the body of the message in the browser:

I confirmed that the ALLOWED_HOSTS has the correct hostname and should be fine. I also checked all of the other settings in configuration.py. I'll run a tcdump to see if I can find additional errors. Thanks.

[candlerb]

A few others:

• Check journalctl -eu netbox just in case gunicorn is logging anything
• Look at ss -nt | grep SYN to see if there are any sockets in SYN_SENT state
• Check Redis and Postgres are up, and reachable from the Netbox server (if they are on remote hosts, or are AWS managed services)

[cryansharp]

I believe that the journalctl command led me to the issue.

Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: File "/opt/netbox/venv/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: self.connection = self.get_new_connection(conn_params)
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: File "/opt/netbox/venv/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: return func(*args, **kwargs)
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: File "/opt/netbox/venv/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: connection = self.Database.connect(**conn_params)
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: File "/opt/netbox/venv/lib64/python3.9/site-packages/psycopg/connection.py", line 729, in connect
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: raise ex.with_traceback(None)
Nov 28 21:16:35 aws-vm-netbox-01.aws.ironnet.io gunicorn[160762]: django.db.utils.OperationalError: connection is bad: Name or service not known
Nov 28 21:33:25 aws-vm-netbox-01.aws.ironnet.io gunicorn[160758]: Skipping config initialization (database unavailable)

I think that it's a FQDN and naming issue since we had a CNAME in our previous environment that redirected the URL request to the FQDN. I'll dig into that more and see if I can correct the issue at that level. I had a suspicion that was it, but good to be able to confirm. Thank you.