Netbox upgrade from v4.2.4 to v4.5.2 fails on database migration

[osuewad]

NetBox Edition

NetBox Community

NetBox Version

v4.2.4 / v4.5.2

Python Version

3.12

Steps to Reproduce

Upgrade Ubuntu 22.04 to 24.04
Upgrade python 3.10 to 3.12

Other dependencies are met as per documentation

Stop netbox netbox-rq services
Run .\upgrade.sh

Extra info:
within /opt/netbox/netbox/netbox/configuration.py -> changed old database to databaseS format and also added API_TOKEN_PEPPERS:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'netbox',               # Database name
        'USER': 'netbox',               # PostgreSQL username
        'PASSWORD': 'redacted', # PostgreSQL password
        'HOST': 'redacted', # Database server
        'PORT': '5432',                     # Database port (leave blank for default)
        'CONN_MAX_AGE': 300,            # Max database connection age
    }

}

API_TOKEN_PEPPERS = {
    1: 'redacted',
}

Expected Behavior

Upgrade migration works without issue and upgrade is finished to v4.5.2

Observed Behavior

Applying database migrations (python3 netbox/manage.py migrate)...
Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 420, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 464, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 111, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 114, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/recorder.py", line 89, in applied_migrations
    if self.has_table():
       ^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/recorder.py", line 63, in has_table
    with self.connection.cursor() as cursor:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 320, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/dummy/base.py", line 20, in complain
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.

[pheus]

Thanks for the report (and for including the full traceback).

Could you help us narrow down whether something external is influencing the settings during the upgrade?

A couple quick questions/tests:

• Do you have any plugins installed? If so, could you temporarily disable them (e.g. clear PLUGINS / remove plugin config) and re-run the upgrade to see if the migration step succeeds?
• Can you try upgrading to v4.5.1 using the same environment (Ubuntu 24.04 / Python 3.12) and confirm whether migrations run successfully there?

If you can share the outcome of those two checks (plus any additional traceback/log output, if it changes), this would be great. Thanks!

[albert-flowerbed]

Can confirm that disabling plugins (PLUGINS = []) is a workaround !
Was using #PLUGINS = ['netbox_interface_synchronization', 'netbox_topology_views', 'netbox_lifecycle']

[osuewad]

@pheus Yes, we do have plugins, but before doing anything I did comment them out and add an empty field:

# Enable installed plugins. Add the name of each plugin to the list.
PLUGINS = [] #empty plugin line for updating
#PLUGINS = ["netbox_data_flows", "netbox_lifecycle", "netbox_topology_views"]

I re-pulled git for v.4.5.1. It seems to fail on the same step, now with a different traceback:

Applying database migrations (python3 netbox/manage.py migrate)...
Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 412, in run_from_argv
    parser = self.create_parser(argv[0], argv[1])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 375, in create_parser
    self.add_arguments(parser)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 45, in add_arguments
    choices=tuple(connections),
            ^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
                ^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 81, in __getattr__
    self._setup(name)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 68, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 166, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/settings.py", line 277, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: STORAGE_BACKEND and STORAGES are both set, remove the deprecated STORAGE_BACKEND setting.

Extra info of all installed versions:

Current Netbox version installed: "4.5.1"
Current Python3 version installed : Python 3.12.3
Redis server version installed: Redis server v=7.0.15
Max available Python3 version in apt: python3.12
Ubuntu version: Ubuntu 24.04.4 LTS
PostgreSQL server version:  14.20 (Ubuntu 14.20-0ubuntu0.22.04.1)

[pheus]

Thanks for testing and reporting back!

This doesn’t look like a DB migration regression in 4.5.2 after all. Your 4.5.1 run is now surfacing the underlying configuration issue:

STORAGE_BACKEND and STORAGES are both set, remove the deprecated STORAGE_BACKEND setting.

So the upgrade is failing because both the legacy STORAGE_BACKEND setting and the newer STORAGES setting are present at the same time. To proceed, you’ll want to remove STORAGE_BACKEND entirely and keep only the STORAGES configuration (per the current config example/docs).

Given you’ve already had to adjust DATABASE → DATABASES and add API_TOKEN_PEPPERS, I’d recommend one of these approaches:

• Best option: Start from the current configuration_example.py shipped with your target NetBox release and re-apply your customizations on top of it.
• Alternatively, carefully port your existing custom config forward, but double-check you’re not carrying forward any deprecated settings.

If you rebuild from the example, please make sure you bring over all required settings (e.g. SECRET_KEY, DATABASES, Redis config, ALLOWED_HOSTS, etc.) and then retry upgrade.sh.

Once STORAGE_BACKEND is removed, if you still hit any errors, feel free to paste the next traceback and we’ll take another look.

[exu-g]

I also ran into this issue, though in my case the API_TOKEN_PEPPER_1 was too short. Like @osuewad using v4.5.1 pointed me in the right direction

[osuewad]

Hello @pheus ,

Thank you for your reply and pointing me in the right direction. To fix I had to remove STORAGE_BACKEND completely, add STORAGES with only default option and make my AWS the options parameters in the configuration.py. After that I upgraded to v4.5.1 and then 4.5.2. I do have an issue with my uploaded images, but not sure if it's related to my STORAGE configuration..

When I click on an image:

There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'ValueError'>

Required parameter name not set

Python version: 3.12.3
NetBox version: 4.5.2
Plugins: 
  netbox_data_flows: 1.5.0
  netbox_lifecycle: 1.1.7
  netbox_topology_views: 4.5.0
The request which yielded the above error is shown below:

GET https://redacted/extras/image-attachments/106/

If further assistance is required, please post to the [NetBox discussion forum](https://github.com/netbox-community/netbox/discussions) on GitHub.

Maybe I'm still missing a parameter for this in the configuration.

[osuewad]

At the end I was unable to fix the S3 storage configuration parameters. I've chosen to comment out the parameters + storages and re-migrate to have it move the images to the local default folder. Now it works ok.