systemctl restart netbox netbox-rq not working

[vongchan]

NetBox version

v3.1.1

Python version

3.9

Steps to Reproduce

I got an issue with the netbox-rq. Our netbox GUI is working fine. but when i tried to restart or check the status of netbox (i.g: systemctl status netbox netbox-rq), it show an error.
Could you please assist me with this issue?

Expected Behavior

no error and should be able to restart the server such as systemctl restart netbox netbox-rq

Observed Behavior

root@local-01:/home/netbox# systemctl status netbox netbox-rq
● netbox.service - NetBox WSGI Service
Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2022-05-10 13:24:20 PDT; 29s ago
Docs: https://docs.netbox.dev/
Process: 363981 ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi (code=exited, status=1/FAILURE)
Main PID: 363981 (code=exited, status=1/FAILURE)

May 10 13:24:37 local-01 systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
May 10 13:24:37 local-01 systemd[1]: netbox-rq.service: Failed with result 'exit-code'.

[candlerb]

What does journalctl -eu netbox-rq show?

[vongchan]

Sorry for my late response. The issue has been fixed. I just moved my netbox repo files to opt directory and start it from there. now it works at my end. Thanks!

[alexzerabr]

@vongchan I have the same problem that you reported, could you show me what you did to solve it?

[vongchan]

@Alexzera-Dev I believe it was the permission issue. After I moved my nebox repository file location to the folder "opt" in the directory, it started and worked fine for me.

[bitcollector1]

almost always permission issues from past experience, good luck!

[Yarli]

I found the same issue, this is my exact script I used to upgrade my netbox and it fixed my issues.

wget https://github.com/netbox-community/netbox/archive/vX.Y.Z.tar.gz
sudo tar -xzf vX.Y.Z.tar.gz -C /opt
sudo ln -sfn /opt/netbox-X.Y.Z/ /opt/netbox

Run these commands but replace netbox-X-Y-Z with your old (existing) install

sudo cp /opt/netbox-X.Y.Z/local_requirements.txt /opt/netbox/
sudo cp /opt/netbox-X.Y.Z/netbox/netbox/configuration.py /opt/netbox/netbox/netbox/
sudo cp /opt/netbox-X.Y.Z/netbox/netbox/ldap_config.py /opt/netbox/netbox/netbox/
sudo cp -r /opt/netbox-X.Y.Z/netbox/local /opt/netbox
sudo cp -pr /opt/netbox-X.Y.Z/netbox/media/ /opt/netbox/netbox/
sudo cp -r /opt/netbox-X.Y.Z/netbox/scripts /opt/netbox/netbox/
sudo cp -r /opt/netbox-X.Y.Z/netbox/reports /opt/netbox/netbox/
sudo cp /opt/netbox-X.Y.Z/gunicorn.py /opt/netbox/

I suspect this is what will fix your issue

sudo chown netbox:netbox /opt/netbox/local -R
sudo chown netbox:netbox /opt/netbox/netbox/media/ -R

Then lastly once that's all done, run the upgrade from within the newly linked netbox folder

cd /opt/netbox
./upgrade.sh

[henry4711lp]

Having the same issue.
When using sudo /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
manually, it works again

[spada05]

Sounds like a permissions problem. If you do systemctl start netbox then journalctl -eu netbox, what do you see?
@candlerb

I am experiencing the same issue on both services (netbox, netbox-rq). Here's the journalctl output, confirming a permissions issue:

May 08 15:11:43 netops1 systemd[1]: netbox-rq.service: Main process exited, code=exited, status=203/EXEC
May 08 15:11:43 netops1 systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
May 08 15:12:13 netops1 systemd[1]: netbox-rq.service: Service RestartSec=30s expired, scheduling restart.
May 08 15:12:13 netops1 systemd[1]: netbox-rq.service: Scheduled restart job, restart counter is at 327.
May 08 15:12:13 netops1 systemd[1]: Stopped NetBox Request Queue Worker.
May 08 15:12:13 netops1 systemd[1]: Started NetBox Request Queue Worker.
May 08 15:12:13 netops1 systemd[17638]: netbox-rq.service: Failed to execute command: Permission denied
May 08 15:12:13 netops1 systemd[17638]: netbox-rq.service: Failed at step EXEC spawning /opt/netbox/venv/bin/python3: Permission denied

[candlerb]

Somehow, you've got into a state where the file /opt/netbox/venv/bin/python3 (or one of its enclosing directories) is not readable by the user that Netbox is running as.

Check the output of:

ls -l /opt/netbox/venv/bin/python3
ls -ld /opt/netbox/venv/bin /opt/netbox/venv /opt/netbox /opt

And what user are you running it as? Check

cat /etc/systemd/system/netbox.service
cat /etc/systemd/system/netbox-rq.service

[spada05]

Thank you for the response, I think I am getting closer to resolution here. I was unable to upgrade Python to 3.8+ using the package installer and opted to use pyenv to install version 3.10.11.

# pyenv versions
  system
* 3.10.11 (set by /root/.pyenv/version)

I attempted to grant permissions for the "netbox" role to this directory with the following:

sudo chmod -R g+rX ~/.pyenv
sudo chown -R :netbox ~/.pyenv
sudo chmod +x ~/.pyenv/versions/3.10.11/bin/python
sudo chmod +x ~/.pyenv/versions/3.10.11/bin/pip

When testing, I am still seeing permissions denied:

# sudo -u netbox ~/.pyenv/versions/3.10.11/bin/python --version
sudo: unable to execute /root/.pyenv/versions/3.10.11/bin/python: Permission denied

[candlerb]

I really wouldn't suggest messing around with pyenv, as Netbox uses virtualenv as well and I don't know how the two of those interact. Also, there's no need to have any version of Python newer than 3.8: minimum requirements are documented here.

What OS are you basing your install on? If it doesn't even have python 3.8 then it must be ancient. 3.6 went end-of-life over a year ago, and 3.7 is just about to.

As for your permissions problem: on most systems the /root directory is owned by root, and has mode bits set which prevent non-root users from accessing any files within it. These are for good reason, and I wouldn't recommend making it more lax. Therefore, if you really want to mess with pyenv, it will have to be installed under the 'netbox' user, not as 'root'.

[Dan-t3h-man88]

My issue started after I hardened my fresh setup of Netbox on an Ubuntu 22.04 LTS Pro as part of a requirement from my client. I am using the CIS Benchmark USG audit and when I the the USG fix it broke everything. Basically, it reset perms to root:root in my /opt/netbox directory.

sudo chown -R netbox:netbox /opt/netbox

At least got my service working. Now on to getting Redis working 😭

[berkeleycowboy]

I'm experiencing this issue also:

My development environment consists of the following:
RHEL 8.8
Python 3.9
PostgresSQL 13
Redis 5.03
NGINX 1.22

I've been performing the installation as a local admin user. Some of the commands required root access so "sudo su" was performed.

Receiving the following
● netbox.service - NetBox WSGI Service
Loaded: loaded (/etc/systemd/system/netbox.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2023-08-21 10:42:48 PDT; 6s ago
Docs: https://docs.netbox.dev/
Process: 1770 ExecStart=/opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/ne>
Main PID: 1770 (code=exited, status=203/EXEC)

When I execute:
[dennis@scdl-ifipam-001 netbox]$ sudo journalctl -eu netbox
[sudo] password for dennis:
Aug 21 11:04:59 scdl-ifipam-001 systemd[1]: Started NetBox WSGI Service.
Aug 21 11:04:59 scdl-ifipam-001 systemd[1]: netbox.service: Main process exited, code=exited, status=203/EXEC
Aug 21 11:04:59 scdl-ifipam-001 systemd[1]: netbox.service: Failed with result 'exit-code'.
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: netbox.service: Service RestartSec=30s expired, scheduling restar>
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: netbox.service: Scheduled restart job, restart counter is at 71.
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: Stopped NetBox WSGI Service.
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: Started NetBox WSGI Service.
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: netbox.service: Main process exited, code=exited, status=203/EXEC
Aug 21 11:05:30 scdl-ifipam-001 systemd[1]: netbox.service: Failed with result 'exit-code'.
Aug 21 11:06:00 scdl-ifipam-001 systemd[1]: netbox.service: Service RestartSec=30s expired, scheduling restar>
Aug 21 11:06:00 scdl-ifipam-001 systemd[1]: netbox.service: Scheduled restart job, restart counter is at 72.
Aug 21 11:06:00 scdl-ifipam-001 systemd[1]: Stopped NetBox WSGI Service.
Aug 21 11:06:00 scdl-ifipam-001 systemd[1]: Started NetBox WSGI Service.

I'm thinking the issue is a rights issue.

[candlerb]

You may get a better error message if you try starting it by hand:

cd /opt/netbox
sudo -u netbox /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi

[berkeleycowboy]

Receive the following error when executing the command provided:

[root@scdl-ifipam-001 netbox]# sudo -u netbox /opt/netbox/venv/bin/gunicorn --pid /var/tmp/netbox.pid --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi
sudo: unable to execute /opt/netbox/venv/bin/gunicorn: Permission denied

[root@scdl-ifipam-001 bin]# ls -l gunicorn
-rwxr-xr-x. 1 root root 235 Aug 21 21:05 gunicorn

Thanks in advance for any direction you can provide.

[candlerb]

You have a permissions problem.

ls -l /opt/netbox/venv/bin/gunicorn
ls -ld /opt /opt/netbox /opt/netbox/venv /opt/netbox/venv/bin

[berkeleycowboy]

I believe I've come to the root cause of my installation.

netbox installation was performed as root so the netbox service started as root instead of netbox
Permission to /opt/netbox should be assit to netbox, not root.

Is there a step that I have missed or is this something precellular with RHEL 8

[markkuleinio]

AFAIK RHEL has a different default umask value, which leads to situation where root-installed things may not be readable for other users. (But I'm not a RHEL expert)

The preferred way to install apps like NetBox in my opinion is to own the files by root so that the running user (netbox) cannot change them. The exceptions are reconfigured in the docs: https://docs.netbox.dev/en/stable/installation/3-netbox/#create-the-netbox-system-user

[berkeleycowboy]

The following are the rights associated with the media, reports and scripts folders:

[dennis@scdl-ifipam-001 netbox]$ ls -l
total 64
drwxrwxr-x. 10 root root 4096 Aug 21 20:53 circuits
drwxrwxr-x. 11 root root 4096 Aug 21 20:53 core
drwxrwxr-x. 12 root root 4096 Aug 21 20:53 dcim
drwxrwxr-x. 14 root root 4096 Aug 21 20:53 extras
-rwxrwxr-x. 1 root root 278 Aug 15 06:56 generate_secret_key.py
drwxrwxr-x. 11 root root 4096 Aug 21 20:53 ipam
-rwxrwxr-x. 1 root root 249 Aug 15 06:56 manage.py
drwxrwxr-x. 4 netbox root 56 Aug 15 06:56 media
drwxrwxr-x. 13 root root 4096 Aug 21 20:53 netbox
drwxrwxr-x. 9 root root 4096 Aug 15 06:56 project-static
drwxrwxr-x. 2 netbox root 25 Aug 15 06:56 reports
drwxrwxr-x. 2 netbox root 25 Aug 15 06:56 scripts
drwxr-x---. 11 root root 4096 Aug 21 20:55 static
drwxrwxr-x. 18 root root 4096 Aug 15 06:56 templates
drwxrwxr-x. 10 root root 4096 Aug 21 20:53 tenancy
drwxrwxr-x. 8 root root 4096 Aug 21 20:53 users
drwxrwxr-x. 9 root root 4096 Aug 21 20:53 utilities
drwxrwxr-x. 10 root root 4096 Aug 21 20:53 virtualization
drwxrwxr-x. 9 root root 4096 Aug 21 20:53 wireless
[dennis@scdl-ifipam-001 netbox]$

Permissions are consistent with the Netbox configuration.

"Top -u netbox" indicates nothing is running as netbox.

Unless I'm missing something, the following applications should be run as netbox: gunicorn, python3

Does anybody know how I can change who starts the applications ?

[markkuleinio]

https://docs.netbox.dev/en/stable/installation/4-gunicorn/#systemd-setup