Upgrading to 3.3.8 - NETBOX_DELETE_LEGACY_DATA

[F3d0s]

I am working on a project to upgrade our company Netbox from v2.11.11 to the latest version available. Currently, Netbox is running on a VM Ubuntu 18.04.6 LTS, it was installed from git (there are /opt/netbox and /opt/netbox/.git folders) so I'm following Option B: Clone the Git Repository from the official manual. I want to specify that I'm not a Master Jedi in Netbox nor in Linux.

Before running the update on the production server, I made a copy of it and restored on an isolated network, so that I could test the update procedure before applying it in production.

This is what I have done:

• Run apt upgrade
• Upgraded Python3.10, PostgreSQL 11, and Redis 4.0.9
• Made a copy of the netbox installation folder (/opt/netbox)
• Pulled down the most recent iteration of the master branch, as explained on the guide

finally, I ran sudo ./upgrade.sh but I got this error:

django.db.utils.DataError: Unable to proceed with deleting contact fields from Site model: Found 40 sites with legacy contact data. Please ensure all legacy site contact data has been migrated to contact objects before proceeding. Or, set the NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy site contact data.

Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
  Applying dcim.0145_site_remove_deprecated_fields...Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 98, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 290, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 131, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 163, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 248, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 131, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/dcim/migrations/0145_site_remove_deprecated_fields.py", line 24, in check_legacy_data
    raise DataError(
django.db.utils.DataError: Unable to proceed with deleting contact fields from Site model: Found 40 sites with legacy contact data. Please ensure all legacy site contact data has been migrated to contact objects before proceeding. Or, set the NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy site contact data.

I've seen in another discussion that @jeremystretch suggested to run the command $ sudo NETBOX_DELETE_LEGACY_DATA=1 upgrade.sh, which resolved the problem for the other user, but unfortunately in my case it doesn't work, I am not sure how to proceed

$ sudo NETBOX_DELETE_LEGACY_DATA=1 upgrade.sh
sudo: upgrade.sh: command not found

Thank you in advance for helping me out!

[mtinberg]

Assuming you do want to delete the 40 contact fields on Sites, if you've already hand-migrated those to the Contact data model or don't need them, then it might be as simple as giving the path to the upgrade.sh script, eg. sudo ENV_VAR=value ./script.sh as your example didn't include the ./ to specify where to look for the script since it isn't in the $PATH search list. It might be as simple as that — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: F3d0s ***@***.***> Sent: Monday, November 28, 2022 6:23 AM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: [netbox-community/netbox] Upgrading to 3.4.7 - NETBOX_DELETE_LEGACY_DATA (Discussion #11033) I am working on a project to upgrade our company Netbox from v2.11.11 to the latest version available. Currently, Netbox is running on a VM Ubuntu 18.04.6 LTS, it was installed from git (there are /opt/netbox and /opt/netbox/.git folders) so I'm following Option B: Clone the Git Repository<https://docs.netbox.dev/en/stable/installation/upgrading/#:~:text=py%20/opt/netbox/-,Option%20B%3A%20Clone%20the%20Git%20Repository,-This%20guide%20assumes> from the official manual. I want to specify that I'm not a Master Jedi in Netbox nor in Linux. Before running the update on the production server, I made a copy of it and restored on an isolated network, so that I could test the update procedure before applying it in production. This is what I have done: * Run apt upgrade * Upgraded Python3.10, PostgreSQL 11, and Redis 4.0.9 * Made a copy of the netbox installation folder (/opt/netbox) * Pulled down the most recent iteration of the master branch, as explained on the guide finally, I ran sudo ./upgrade.sh but I got this error: django.db.utils.DataError: Unable to proceed with deleting contact fields from Site model: Found 40 sites with legacy contact data. Please ensure all legacy site contact data has been migrated to contact objects before proceeding. Or, set the NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy site contact data. Operations to perform: Apply all migrations: admin, auth, circuits, contenttypes, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless Running migrations: Applying dcim.0145_site_remove_deprecated_fields...Traceback (most recent call last): File "/opt/netbox/netbox/manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 414, in run_from_argv self.execute(*args, **cmd_options) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 460, in execute output = self.handle(*args, **options) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 98, in wrapped res = handle_func(*args, **kwargs) File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 290, in handle post_migrate_state = executor.migrate( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 131, in migrate state = self._migrate_all_forwards( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 163, in _migrate_all_forwards state = self.apply_migration( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 248, in apply_migration state = migration.apply(state, schema_editor) File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 131, in apply operation.database_forwards( File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards self.code(from_state.apps, schema_editor) File "/opt/netbox/netbox/dcim/migrations/0145_site_remove_deprecated_fields.py", line 24, in check_legacy_data raise DataError( django.db.utils.DataError: Unable to proceed with deleting contact fields from Site model: Found 40 sites with legacy contact data. Please ensure all legacy site contact data has been migrated to contact objects before proceeding. Or, set the NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy site contact data. I've seen in another discussion<#9086> that @jeremystretch<https://github.com/jeremystretch> suggested to run the command $ sudo NETBOX_DELETE_LEGACY_DATA=1 upgrade.sh, which resolved the problem for the other user, but unfortunately in my case it doesn't work, I am not sure how to proceed $ sudo NETBOX_DELETE_LEGACY_DATA=1 upgrade.sh sudo: upgrade.sh: command not found Thank you in advance for helping me out! — Reply to this email directly, view it on GitHub<#11033>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UMY4LORTZ2JSRGBR5JTWKSP3NANCNFSM6AAAAAASNIVQIE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[F3d0s]

Apparently those contacts can't go away, we need them. Is there a guide how to hand migrate the contacts from version 2.11.11 to 3.3.x?

I read that they changed the way that contacts are associated in v3.2. Would be better to update to v3.1, and then run the v32_migration script?

On the manual, it say that it's possible to upgrade to an older version but when I try to checkout version 3.1.11 I got this error:

$ sudo git checkout v3.1.11
error: pathspec 'v3.1.11' did not match any file(s) known to git.

Am I doing something wrong or it might be a documentation error?

[markkuleinio]

markku@netbox-lab:/tmp/netbox$ git clone -b master --depth 1 https://github.com/netbox-community/netbox.git .
Cloning into '.'...
remote: Enumerating objects: 1381, done.
remote: Counting objects: 100% (1381/1381), done.
remote: Compressing objects: 100% (1256/1256), done.
remote: Total 1381 (delta 228), reused 771 (delta 95), pack-reused 0
Receiving objects: 100% (1381/1381), 7.09 MiB | 10.85 MiB/s, done.
Resolving deltas: 100% (228/228), done.

markku@netbox-lab:/tmp/netbox$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

markku@netbox-lab:/tmp/netbox$ git checkout v3.1.11
error: pathspec 'v3.1.11' did not match any file(s) known to git

markku@netbox-lab:/tmp/netbox$ git fetch --all --tags
Fetching origin
remote: Enumerating objects: 98877, done.
remote: Counting objects: 100% (98874/98874), done.
remote: Compressing objects: 100% (20188/20188), done.
remote: Total 97922 (delta 78862), reused 96411 (delta 77404), pack-reused 0
Receiving objects: 100% (97922/97922), 57.04 MiB | 19.57 MiB/s, done.
Resolving deltas: 100% (78862/78862), completed with 720 local objects.
From https://github.com/netbox-community/netbox
 * [new tag]             1.0.0             -> 1.0.0
 * [new tag]             1.6.1             -> 1.6.1
 * [new tag]             Solcon-2020-01-06 -> Solcon-2020-01-06
 * [new tag]             v1.0.3            -> v1.0.3
 * [new tag]             v1.0.3-r1         -> v1.0.3-r1
...
 * [new tag]             v3.3.6            -> v3.3.6
 * [new tag]             v3.3.7            -> v3.3.7
 * [new tag]             v3.4-beta1        -> v3.4-beta1

markku@netbox-lab:/tmp/netbox$ git checkout v3.1.11
Note: switching to 'v3.1.11'.
...
HEAD is now at 10f8a9439 Merge pull request #9039 from netbox-community/develop
markku@netbox-lab:/tmp/netbox$

Maybe someone knows a more compact way to fetch just that one tag (and not all), feel free to comment below.

[F3d0s]

This worked, I could finally upgrade the version 3.1 and continue troubleshooting the contacts issue.

[mtinberg]

I don't know if there is a guide, and I don't have an old version to test wth, but you can create a new Contact record with a name/phone/email/address with the same info as the old contact fields on a Site record, then associate that Contact with the Site. If you have multiple sites with the same Contact info then associate that record to all the Sites you need, and if you need to create more Contact records with other name/email/phone/address data to match the other 40 Site records then do so. Then you can pass the environment variable to proceed through the upgrade deleting the old contact columns on the Site data. As far as checking out to the v3.1.11 tag, I didn't have an issue, I did switch to the develop branch and git pull to update my copy, then switched to the v3.1.11 tag without incident $ git status $ sudo git checkout develop $ sudo git pull $ sudo git checkout v3.1.11 $ git status — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: F3d0s ***@***.***> Sent: Tuesday, November 29, 2022 10:42 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] Upgrading to 3.3.8 - NETBOX_DELETE_LEGACY_DATA (Discussion #11033) Apparently those contacts can't go away, we need them. Is there a guide how to hand migrate the contacts from version 2.11.11 to 3.3.x? I read that they changed the way that contacts are associated in v3.2. Would be better to update to v3.1, and then run the v32_migration script<https://github.com/netbox-community/migration-scripts>? On the manual, it say that it's possible to upgrade to an older version<https://docs.netbox.dev/en/stable/installation/upgrading/#:~:text=Checking%20out%20an%20older%20release> but when I try to checkout version 3.1.11 I got this error: $ sudo git checkout v3.1.11 error: pathspec 'v3.1.11' did not match any file(s) known to git. Am I doing something wrong or it might be a documentation error? — Reply to this email directly, view it on GitHub<#11033 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM54EEHYDW5IE7AFCW3WKYXAZANCNFSM6AAAAAASNIVQIE>. You are receiving this because you commented.Message ID: ***@***.***>

[F3d0s]

That worked, thank you!