Error after upgrading to 3.2+ "ModuleNotFoundError: No module named 'cacheops' "

[sstone173]

Hi, I'm currently running 3.1.11 on Ubuntu 20.04 and would now like to upgrade to the latest 3.2 version. The upgrade process seems successful and after a reboot I get the nginx "502 Bad Gateway" page. When I run "journalctl -u netbox" and sort through the logs, it seems like the issue is related to a missing Django module "cacheops". This instance was initially upgraded from 2.6->2.9>2.11>3.0>3.1.11, so I'm thinking it could be related to some older configuration cruft. I've tried upgrading Django and Django-cachops to the latest, did not work. Also tried updating Ubuntu from 20 to 22, but that created a whole different mess and did not work.

Upgrade:

admin@netbox:/opt/netbox# sudo NETBOX_DELETE_LEGACY_DATA=1 ./upgrade.sh
Removing old Python packages (pip3 uninstall -r old_requirements.txt -y)...
WARNING: Skipping django-rest-swagger as it is not installed.
WARNING: Skipping psycopg2 as it is not installed.
WARNING: Skipping pycrypto as it is not installed.
Updating required Python packages (pip3 install -r requirements.txt --upgrade)...
Requirement already up-to-date: Django<2.3,>=2.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 1)) (2.2.28)
Requirement already up-to-date: django-cacheops==4.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 2)) (4.1)
Requirement already up-to-date: django-cors-headers==3.0.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (3.0.2)
Requirement already up-to-date: django-debug-toolbar==2.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 4)) (2.0)
Requirement already up-to-date: django-filter==2.1.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 5)) (2.1.0)
Requirement already up-to-date: django-mptt==0.9.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 6)) (0.9.1)
Requirement already up-to-date: django-prometheus==1.0.15 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 7)) (1.0.15)
Requirement already up-to-date: django-rq==2.1.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 8)) (2.1.0)
Requirement already up-to-date: django-tables2==2.0.6 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 9)) (2.0.6)
Requirement already up-to-date: django-taggit==1.1.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 10)) (1.1.0)
Requirement already up-to-date: django-taggit-serializer==0.1.7 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 11)) (0.1.7)
Requirement already up-to-date: django-timezone-field==3.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 12)) (3.0)
Requirement already up-to-date: djangorestframework==3.9.4 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 13)) (3.9.4)
Requirement already up-to-date: drf-yasg[validation]==1.16.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 14)) (1.16.0)
Requirement already up-to-date: graphviz==0.10.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 15)) (0.10.1)
Requirement already up-to-date: Jinja2==2.10.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 16)) (2.10.1)
Requirement already up-to-date: Markdown==2.6.11 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 17)) (2.6.11)
Requirement already up-to-date: netaddr==0.7.19 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 18)) (0.7.19)
Requirement already up-to-date: Pillow==6.0.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 19)) (6.0.0)
Requirement already up-to-date: psycopg2-binary==2.8.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 20)) (2.8.3)
Requirement already up-to-date: py-gfm==0.1.4 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 21)) (0.1.4)
Requirement already up-to-date: pycryptodome==3.8.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 22)) (3.8.2)
Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.8/dist-packages (from Django<2.3,>=2.2->-r requirements.txt (line 1)) (2022.1)
Requirement already satisfied, skipping upgrade: sqlparse>=0.2.2 in /usr/local/lib/python3.8/dist-packages (from Django<2.3,>=2.2->-r requirements.txt (line 1)) (0.4.2)
Requirement already satisfied, skipping upgrade: funcy<2.0,>=1.8 in /usr/local/lib/python3.8/dist-packages (from django-cacheops==4.1->-r requirements.txt (line 2)) (1.17)
Requirement already satisfied, skipping upgrade: redis>=2.9.1 in /usr/local/lib/python3.8/dist-packages (from django-cacheops==4.1->-r requirements.txt (line 2)) (4.2.2)
Requirement already satisfied, skipping upgrade: six>=1.4.0 in /usr/lib/python3/dist-packages (from django-cacheops==4.1->-r requirements.txt (line 2)) (1.14.0)
Requirement already satisfied, skipping upgrade: django-js-asset in /usr/local/lib/python3.8/dist-packages (from django-mptt==0.9.1->-r requirements.txt (line 6)) (2.0.0)
Requirement already satisfied, skipping upgrade: prometheus-client>=0.0.21 in /usr/local/lib/python3.8/dist-packages (from django-prometheus==1.0.15->-r requirements.txt (line 7)) (0.14.1)
Requirement already satisfied, skipping upgrade: rq>=1.0 in /usr/local/lib/python3.8/dist-packages (from django-rq==2.1.0->-r requirements.txt (line 8)) (1.10.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.34 in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (0.17.21)
Requirement already satisfied, skipping upgrade: coreschema>=0.0.4 in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (0.0.4)
Requirement already satisfied, skipping upgrade: inflection>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (0.5.1)
Requirement already satisfied, skipping upgrade: coreapi>=2.3.3 in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (2.3.3)
Requirement already satisfied, skipping upgrade: uritemplate>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (4.1.1)
Requirement already satisfied, skipping upgrade: swagger-spec-validator>=2.1.0; extra == "validation" in /usr/local/lib/python3.8/dist-packages (from drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (2.7.4)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from Jinja2==2.10.1->-r requirements.txt (line 16)) (2.0.1)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from py-gfm==0.1.4->-r requirements.txt (line 21)) (45.2.0)
Requirement already satisfied, skipping upgrade: deprecated>=1.2.3 in /usr/local/lib/python3.8/dist-packages (from redis>=2.9.1->django-cacheops==4.1->-r requirements.txt (line 2)) (1.2.13)
Requirement already satisfied, skipping upgrade: packaging>=20.4 in /usr/local/lib/python3.8/dist-packages (from redis>=2.9.1->django-cacheops==4.1->-r requirements.txt (line 2)) (21.3)
Requirement already satisfied, skipping upgrade: async-timeout>=4.0.2 in /usr/local/lib/python3.8/dist-packages (from redis>=2.9.1->django-cacheops==4.1->-r requirements.txt (line 2)) (4.0.2)
Requirement already satisfied, skipping upgrade: click>=5.0.0 in /usr/lib/python3/dist-packages (from rq>=1.0->django-rq==2.1.0->-r requirements.txt (line 8)) (7.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.2.6; platform_python_implementation == "CPython" and python_version < "3.11" in /usr/local/lib/python3.8/dist-packages (from ruamel.yaml>=0.15.34->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (0.2.6)
Requirement already satisfied, skipping upgrade: requests in /usr/lib/python3/dist-packages (from coreapi>=2.3.3->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (2.22.0)
Requirement already satisfied, skipping upgrade: itypes in /usr/local/lib/python3.8/dist-packages (from coreapi>=2.3.3->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (1.2.0)
Requirement already satisfied, skipping upgrade: pyyaml in /usr/local/lib/python3.8/dist-packages (from swagger-spec-validator>=2.1.0; extra == "validation"->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (6.0)
Requirement already satisfied, skipping upgrade: jsonschema in /usr/local/lib/python3.8/dist-packages (from swagger-spec-validator>=2.1.0; extra == "validation"->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (3.2.0)
Requirement already satisfied, skipping upgrade: wrapt<2,>=1.10 in /usr/local/lib/python3.8/dist-packages (from deprecated>=1.2.3->redis>=2.9.1->django-cacheops==4.1->-r requirements.txt (line 2)) (1.14.0)
Requirement already satisfied, skipping upgrade: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.4->redis>=2.9.1->django-cacheops==4.1->-r requirements.txt (line 2)) (3.0.8)
Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /usr/lib/python3/dist-packages (from jsonschema->swagger-spec-validator>=2.1.0; extra == "validation"->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (19.3.0)
Requirement already satisfied, skipping upgrade: pyrsistent>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->swagger-spec-validator>=2.1.0; extra == "validation"->drf-yasg[validation]==1.16.0->-r requirements.txt (line 14)) (0.18.1)
Applying database migrations (python3 netbox/manage.py migrate)...
Operations to perform:
Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization
Running migrations:
No migrations to apply.
Collecting static files (python3 netbox/manage.py collectstatic --no-input)...

0 static files copied to '/opt/netbox/netbox/static', 548 unmodified.

sudo journalctl -u netbox output:

Jul 25 15:10:59 INF-IPAM001 systemd[1]: Started NetBox WSGI Service.
Jul 25 15:10:59 INF-IPAM001 gunicorn[12831]: [2022-07-25 15:10:59 -0400] [12831] [INFO] Starting gunicorn 20.1.0
Jul 25 15:10:59 INF-IPAM001 gunicorn[12831]: [2022-07-25 15:10:59 -0400] [12831] [INFO] Listening at: http://127.0.0.1:8001 (12831)
Jul 25 15:10:59 INF-IPAM001 gunicorn[12831]: [2022-07-25 15:10:59 -0400] [12831] [INFO] Using worker: sync
Jul 25 15:10:59 INF-IPAM001 gunicorn[12865]: [2022-07-25 15:10:59 -0400] [12865] [INFO] Booting worker with pid: 12865
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: [2022-07-25 15:11:00 -0400] [12879] [INFO] Booting worker with pid: 12879
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: [2022-07-25 15:11:00 -0400] [12883] [INFO] Booting worker with pid: 12883
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: [2022-07-25 14:11:00 -0500] [12865] [ERROR] Exception in worker process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: Traceback (most recent call last):
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: worker.init_process()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: self.load_wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: self.wsgi = self.app.wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: self.callable = self.load()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: return self.load_wsgiapp()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: return util.import_app(self.app_uri)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: mod = importlib.import_module(module)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 975, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 671, in _load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 848, in exec_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 219, in _call_with_frames_removed
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: application = get_wsgi_application()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: django.setup(set_prefix=False)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/init.py", line 24, in setup
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: apps.populate(settings.INSTALLED_APPS)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/registry.py", line 91, in populate
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: app_config = AppConfig.create(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/config.py", line 224, in create
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: import_module(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: File "", line 973, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: ModuleNotFoundError: No module named 'cacheops'
Jul 25 15:11:00 INF-IPAM001 gunicorn[12865]: [2022-07-25 14:11:00 -0500] [12865] [INFO] Worker exiting (pid: 12865)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: [2022-07-25 14:11:00 -0500] [12879] [ERROR] Exception in worker process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: Traceback (most recent call last):
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: worker.init_process()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: self.load_wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: self.wsgi = self.app.wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: self.callable = self.load()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: return self.load_wsgiapp()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: return util.import_app(self.app_uri)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: mod = importlib.import_module(module)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 975, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 671, in _load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 848, in exec_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 219, in _call_with_frames_removed
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: application = get_wsgi_application()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: django.setup(set_prefix=False)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/init.py", line 24, in setup
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: apps.populate(settings.INSTALLED_APPS)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/registry.py", line 91, in populate
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: app_config = AppConfig.create(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/config.py", line 224, in create
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: import_module(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: File "", line 973, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: ModuleNotFoundError: No module named 'cacheops'
Jul 25 15:11:00 INF-IPAM001 gunicorn[12879]: [2022-07-25 14:11:00 -0500] [12879] [INFO] Worker exiting (pid: 12879)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: [2022-07-25 15:11:00 -0400] [12831] [WARNING] Worker with pid 12879 was terminated due to signal 15
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: [2022-07-25 14:11:00 -0500] [12883] [ERROR] Exception in worker process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: Traceback (most recent call last):
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: worker.init_process()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: self.load_wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: self.wsgi = self.app.wsgi()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: self.callable = self.load()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: return self.load_wsgiapp()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: return util.import_app(self.app_uri)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: mod = importlib.import_module(module)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 975, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 671, in _load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 848, in exec_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 219, in _call_with_frames_removed
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: application = get_wsgi_application()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: django.setup(set_prefix=False)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/init.py", line 24, in setup
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: apps.populate(settings.INSTALLED_APPS)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/registry.py", line 91, in populate
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: app_config = AppConfig.create(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/config.py", line 224, in create
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: import_module(entry)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: return _bootstrap._gcd_import(name[level:], package, level)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 1014, in _gcd_import
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 991, in _find_and_load
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: File "", line 973, in _find_and_load_unlocked
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: ModuleNotFoundError: No module named 'cacheops'
Jul 25 15:11:00 INF-IPAM001 gunicorn[12883]: [2022-07-25 14:11:00 -0500] [12883] [INFO] Worker exiting (pid: 12883)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: Traceback (most recent call last):
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in run
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: self.sleep()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in sleep
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: ready = select.select([self.PIPE[0]], [], [], 1.0)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: self.reap_workers()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: During handling of the above exception, another exception occurred:
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: Traceback (most recent call last):
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/bin/gunicorn", line 8, in
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: sys.exit(run())
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: super().run()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: Arbiter(self).run()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in run
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: self.halt(reason=inst.reason, exit_status=inst.exit_status)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in halt
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: self.stop()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in stop
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: time.sleep(0.1)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: self.reap_workers()
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: File "/opt/netbox/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: raise HaltServer(reason, self.WORKER_BOOT_ERROR)
Jul 25 15:11:00 INF-IPAM001 gunicorn[12831]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

PIP Freeze Django Output:

admin@netbox:~$ pip freeze
Django==3.2.12
django-auth-ldap==4.0.0
django-cacheops==4.1
django-cors-headers==3.11.0
django-debug-toolbar==3.2.4
django-filter==21.1
django-graphiql-debug-toolbar==0.2.0
django-js-asset==2.0.0
django-mptt==0.13.4
django-pglocks==1.0.4
django-prometheus==2.2.0
django-redis==5.2.0
django-rq==2.5.1
django-tables2==2.4.1
django-taggit==2.1.0
django-taggit-serializer==0.1.7
django-timezone-field==5.0
djangorestframework==3.12.4

[markkuleinio]

The output of running upgrade.sh you showed us does not look like NetBox 3.x upgrade.sh output at all, so your /opt/netbox must be at some older version.

[sstone173]

Ugh, you're right, I never executed the pull request from the master, so who knows what I was breaking. Thanks for pointing that out, you definitely saved me more wasted time!

[candlerb]

It's also worth checking your configuration.py to see whether it has been updated to match the new version.

diff -u configuration-example.py configuration.py may highlight some issues. Or you can move your configuration.py out of the way, copy the example one, and then apply your own changes back into this one.