Migrating PostgresSQL DB with custom fields

[berkeleycowboy]

I'm would like to migrate a PostgresSQL DB with custom fields.

Source Netbox server: v.2.8.8, PostgresSQL 12.5
Target Netbox server v3.5.8, PostgresSQL 15.2

DB would be exported using the following command:
pg_dump --username netbox --password --host localhost netbox > netbox.sql

Then loading the Exported db with the following commands?
psql -c 'drop database netbox'
psql -c 'create database netbox'
psql netbox < netbox.sql

Will the Target Netbox server recognize the custom fields?
Will PostgresSQL 15.2 be able to read the DB exported from 12.5 ?

[jcollie]

Yes, PostgreSQL 12.5 → 15.2 will be fine. You won't want to jump directly from NetBox 2.8 to 3.5 though. You'll certainly break your install if you do. It's gonna take some time, but you should cycle through each .X version from 2.8.0 → 2.9.0 → 2.10.0 ... 3.0 → 3.1 ... etc. Just need to run each version long enough for the migrations to take place and to verify that things are still working. You also need to look at the release notes for each version because there are some required manual steps that need to be taken for some releases.

I'd also recommend using Docker images because older versions of NetBox didn't pin specific Python dependencies and so it can be difficult to get a specific older release installed. The Docker images encapsulated a working set of dependencies so you don't have to worry about that. Using Docker too will keep your virtual environments from getting poisoned with weird dependencies.

[candlerb]

I would say there is no need to do all the intermediate upgrades, but you do need to go via 2.11.x. First upgrade to 2.11.12, then upgrade to 3.6.3, and that should be sufficient. I agree that you do need to read all the release notes for intervening versions 2.9.0, 2.10.0 etc - and if you have any existing REST API integrations they are likely to need updating. Do all this in a test environment first.

I also agree that the problem with dependencies for old versions is real. If you are unable to install 2.11.12 successfully, then use the pinned requirements from here. (That is a good reason for not going via intermediate versions, because we don't have pip freeze output for any earlier than 2.11.12)

Another thing to beware is the python version. I'm not sure whether Netbox 2.11.12 will run with python later than 3.8. At worst, build a VM with 3.8 (e.g. Ubuntu 20.04), install Netbox 2.11.12, import the database and upgrade it, and export it from there. Then use this as your new baseline dump and import it into a new VM with latest Python and Netbox.

The database export/import process is documented here but is missing the step ALTER DATABASE netbox OWNER TO netbox;

[berkeleycowboy]

I've been trying to figure out a method of migrating the production Netbox (Redhat 7.9, PostgresSQL 12.5, Netbox 2.8.8) to an updated REL version Redhat 8 or 9.

My thought was that I could build a new server using Redhat 8 with updated components and port the SQL data. I'll call this the Rebuild-and-port-SQL-data method.

Since the Rebuild-and-port-SQL-data method isn't possible I'll need build the new RH 8 server using the same Netbox version.

Are there other decencies in the application stack that are relevant( Nginx, Gunicorn, Redis ).

Thank in advance for your support.

[mtinberg]

Since the Rebuild-and-port-SQL-data method isn't possible I'll need build the new RH 8 server using the same Netbox version. That's not quite right, you can build and port the SQL, but you have to make a quick stop at 2.11.12 and run upgrade.sh to apply the database migrations from that version, one time, before finishing on 3.x current, you don't need to install 2.8.x on the new server as part of the migration. eg. * dump DB on EL7 server * restore DB on PG15 on EL8/9 server * install Netbox v2.11.12 with pinned python dependencies on EL8/9 * review changelog for each major version (2.9, 2.10, 2.11, 3.x) and note breaking changes or any manual steps * follow install process up to running upgrade.sh * optionally dump DB with v2.11.12 schema on EL8/9 * delete Netbox v2.11.12, install Netbox 3.current on EL8/9 * continue through the setup process including running upgrade.sh which will update the DB schema to current * start services if there is difficulty in installing/upgrading from 2.11 on EL8/9 then a workaround would be to install a temporary VM running EL7 or something contemporary with 2.11.12 installed to load the 2.8.8 db into run upgrade.sh and output a 2.11.12 db, which you can restore into a 3.current host and run upgrade.sh to bring the schema to current. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: Dennis Weikel ***@***.***> Sent: Thursday, October 5, 2023 2:57 PM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [netbox-community/netbox] Migrating PostgresSQL DB with custom fields (Discussion #13977) I've been trying to figure out a method of migrating the production Netbox (Redhat 7.9, PostgresSQL 12.5, Netbox 2.8.8) to an updated REL version Redhat 8 or 9. My thought was that I could build a new server using Redhat 8 with updated components and port the SQL data. I'll call this the Rebuild-and-port-SQL-data method. Since the Rebuild-and-port-SQL-data method isn't possible I'll need build the new RH 8 server using the same Netbox version. Are there other decencies in the application stack that are relevant( Nginx, Gunicorn, Redis ). Thank in advance for your support. — Reply to this email directly, view it on GitHub<#13977 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM75JLHVNOERIX34XDTX54GI7AVCNFSM6AAAAAA5TTUJMWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEMBSGYYTA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mtinberg]

It requires python3 but doesn't list a specific version, I would expect py36-py38 at least should be fine, it probably works with newer versions as well, but that is something that wouldn't take too long to find out, and it only needs to work for running the various Django management commands to get the db updated, whether the rest of the webapp works right isn't necessary just to upgrade past it. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: Dennis Weikel ***@***.***> Sent: Friday, October 6, 2023 9:55 AM To: netbox-community/netbox ***@***.***> Cc: Mark Tinberg ***@***.***>; Comment ***@***.***> Subject: Re: [netbox-community/netbox] Migrating PostgresSQL DB with custom fields (Discussion #13977) How do I determine the Python version supported by Netbox 2.11.12 ? The pinned requirements doesn't display the Python version. — Reply to this email directly, view it on GitHub<#13977 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UMZHLNY3QBJ636CIY6DX6ALXBAVCNFSM6AAAAAA5TTUJMWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEMJQGU4TK>. You are receiving this because you commented.Message ID: ***@***.***>