Error when ugrading netbox 3.2.9 to 3.3.1 with existing data from previous version

[zurcoileh]

NetBox version

v3.3.1

Python version

3.8

Steps to Reproduce

1. Netbox upgrade scritp to 3.3.1 from 3.2.9
2. Error on running migrations on cablepaths

Expected Behavior

Netbox is up and no errros on the interface after restore

Observed Behavior

Running migrations:
Applying dcim.0158_populate_cable_terminations...
Updated 8100/8109 cables (99.89%)Traceback (most recent call last):
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "dcim_cable_termination_unique_termination"
DETAIL: Key (termination_type_id, termination_id)=(43, 1875) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "netbox/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/init.py", line 446, in execute_from_command_line
utility.execute()
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/init.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/base.py", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/base.py", line 98, in wrapped
res = handle_func(*args, **kwargs)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 290, in handle
post_migrate_state = executor.migrate(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 131, in migrate
state = self._migrate_all_forwards(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 163, in _migrate_all_forwards
state = self.apply_migration(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 248, in apply_migration
state = migration.apply(state, schema_editor)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 131, in apply
operation.database_forwards(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/opt/netbox-3.3.1/netbox/dcim/migrations/0158_populate_cable_terminations.py", line 73, in populate_cable_terminations
CableTermination.objects.bulk_create(cable_terminations, batch_size=100)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/models/query.py", line 579, in bulk_create
returned_columns = self._batched_insert(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1458, in _batched_insert
self._insert(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1434, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1621, in execute_sql
cursor.execute(sql, params)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/netbox-3.3.1/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "dcim_cable_termination_unique_termination"
DETAIL: Key (termination_type_id, termination_id)=(43, 1875) already exists.

[kkthxbye-code]

So you upgraded to 3.3.1, then restored the old database and then tried to run the new code with the old database? Is this correctly understood?

If so, that is not supported. Downgrade your netbox install as well if you want to downgrade.

[zurcoileh]

I have tryed to proceed upgrading from 3.2.9 to 3.3.1 without doing any restore on db. I have an error on migrations now

[kornix]

You would upgrade 3.2.9 -> 3.3.0 -> 3.3.1 (and -> 3.3.2 obviously). This way worked as expected (at least i had not any issues when upgrading my install).

[candlerb]

That won't make any difference. Going 3.2.9 -> 3.3.2 is fine (indeed, going 2.11.0 to 3.3.2 is fine).

The problem is that the database itself has got into an inconsistent state somehow, and the migration is barfing on this.

I'm afraid there's no shortcut to diagnosing and fixing this problem properly, at the SQL layer. It's told you what to look for: it's to do with cable terminations and it's having a problem with (termination_type_id, termination_id)=(43, 1875).

To understand what the 43 means, you can do a query like this:

netbox=# select * from django_content_type where id=43;
 id | app_label | model
----+-----------+--------
 43 | ipam      | prefix
(1 row)

That's what my system shows; but it's very unlikely that a cable termination will be on an ipam_prefix so I think you'll get a different answer.

Then you need to look through the data from 3.2.9 and find out if the database somehow has two cables connecting to the same endpoint.

select * from dcim_cable where (termination_a_type_id=43 and termination_a_id=1875)
  or (termination_b_type_id=43 and termination_b_id=1875);

If that returns two rows, then you've got your problem. Find those two cables in the GUI, and decide which one to delete. If it doesn't, then you'll have to dig down deeper.

Good luck!

[zurcoileh]

Thanks a lot for clarifing and that is precisely what's happening .Two cables were created with the same terminations but inverted. I wonder how was that possible if the unique index should prevent that from hapenning, right? Anyways, apreciate it.

SQL result:

select id, termination_a_id, termination_b_id from dcim_cable where (termination_a_type_id=43 and termination_a_id=1875) or (termination_b_type_id=43 and termination_b_id=1875);
id | termination_a_id | termination_b_id
------+------------------+------------------
1273 | 1171 | 1875
5087 | 1875 | 1171
(2 rows)

[candlerb]

Glad you found it.

SQL uniqueness constraints can't check that a value exists in one column but not another, so this must have been checked using Django instead - and they don't always work (e.g. in race conditions).

I believe that the new structure in 3.3, which allows multiple terminations against a cable - and therefore they're all in the same column - can enforce this.