Replicating Netbox

[aquelf]

Hi all,

I'm about to replicate Netbox from an existing test server to a new Prod server, the test version is (v3.0.1) & the new version is (v3.1.6)
should this be seamless of do I need to do any in-between steps?

Thanks

Barry

[candlerb]

By replicating, do you mean copying across the entire database? So any existing data in the prod server will be thrown away, and replaced entirely by data from the test server?

If so: then after you've replicated the database to prod, you'll need to run ./upgrade.sh on the prod server to apply the migrations to update the database schema.

If it were me, I would prefer upgrading the test server to 3.1.6 first, and then replicating to prod. After all, the whole point of a test server is to check that changes are good before you roll them out to production :-)

If you don't want to throw away the existing data from prod, then you'll need to be much more specific about what sort of "replication" you want to do, as it may be very difficult to achieve.

[aquelf]

Thanks for the reply, yea so the plan would be to eventually get rid of the existing & move forward with the new db in prod. We couldn't just move the server unfortunately.

I probably jumped the gun as I didn't upgrade the test version beforehand, but what I did was took a backup of the old db & moved it over via the WinSCP (below steps)

To extract the existing db to a file:
pg_dump --username netbox --password --host localhost netbox > netbox.sql
Use WinSCP to take the file off the vm & to import it to the new vm in the same location (netbox folder)
To import the db into the new Netbox instance use the following commands
sudo -u postgres psql -c 'drop database netbox'
sudo -u postgres psql -c 'create database netbox'
sudo -u postgres psql netbox < netbox.sql

Everything IPAM related came over correctly however I did observe some errors & anything I had saved in the Inventory needs to be readded is that normal?

[candlerb]

No, it's not normal. What errors did you see? If you didn't run upgrade.sh on the new server after importing the database then certainly there will be (unspecified) problems, because the database schema won't match netbox 3.1.6.

Also, nothing that you had added into the old (test) Netbox should need to be re-added once it arrives at the prod Netbox. But anything that was in the prod Netbox will be lost.

[aquelf]

So the new prod was just a blank install, I'm doing a lift & shift of the db essentially. So yea I didn't run the Netbox script prior to moving the db to the new instance. (I've attached a snippet of the error logs).

I mean from an IPAM perspective all rows copied over perfectly, the Inventory didn't, if I manually add the inventory will this work?

OR

Will running the upgrade script & doing the same steps work?

netbox db errors.txt
netbox db errors.txt

Thanks again.

[candlerb]

Those errors are terrible. Your sql dump has been mangled somehow, and all the duplicate key stuff and "relation already exists" is indication of a major problem. I think you need someone on site who knows postgres to look over the database dump, and the target system, and work out what's wrong. Also, make sure you're using compatible versions of Postgres; Netbox needs at leas Postgres 10.

I don't actually know what you mean by "The Inventory". Do you mean that Devices are missing? (Netbox also has something called "Inventory Items" but they sit inside Devices). Given that heap of errors, I'd be surprised if anything has made it across. That's where you need to focus your efforts.

[aquelf]

That doesn't sound to good. I mean I just dumped the db as is & then tried to update using the provided steps. Getting someone with postgres knowledge could be an issue, is that the only alternative? Running the upgrader script & trying the same steps will not work? Or wiping the server & starting fresh? The postgres verstion is the same for the 2 devices. (12.9 (Ubuntu 12.9-0ubuntu0.20.04.1)) (12.9 (Ubuntu 12.9-0ubuntu0.20.04.1)) Apologies for asking the same questions but not that familiar with this stuff. Also yes, what I mean every row for IPAM came over, Device Manufacturers & Roles did also but not Devices & Device Types. Thanks

…

On Thu, Feb 3, 2022 at 9:51 PM Brian Candler ***@***.***> wrote: Those errors are terrible. Your sql dump has been mangled somehow, and all the duplicate key stuff and "relation already exists" is indication of a major problem. I think you need someone on site who knows postgres to look over the database dump, and the target system, and work out what's wrong. Also, make sure you're using compatible versions of Postgres; Netbox needs at leas Postgres 10. I don't actually know what you mean by "The Inventory". Do you mean that Devices are missing? (Netbox also has something called "Inventory Items" but they sit inside Devices). Given that heap of errors, I'd be surprised if anything has made it across. That's where you need to focus your efforts. — Reply to this email directly, view it on GitHub <#8534 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUHI2LMJ5I3KC5JET26453DUZL2GXANCNFSM5NORKVEA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[candlerb]

Getting someone with postgres knowledge could be an issue, is that the only alternative?

Someone who can access your system directly would be in the best place to analyse and fix whatever the problem is.

I've seen various ways in which databases can be messed up, including all the netbox tables being put in the wrong database (particularly the system "postgres" database), but it's hard to know without seeing the actual data.

[aquelf]

Hi Brian,

Apologies I was offline for a few days, yea I'll see whether I can get some one from the db team to take a look at the data to see whether they can spot anything.
The only thing I can thing of it the following:

• not running the upgrader script prior to the db dump resulting in incompatible data
• using the same sa pwd on the old as the new
• Or, maybe trying to duplicate import the db (unlikely)

[aquelf]

Hi Brian, I tried upgrading the original test netbox application now running 3.1.7 & the new 3.1.6.

The issue I think that I'm observing is the following:
root@ubuntu2004:/home/ubuntu# sudo -u postgres psql -c 'drop database netbox'
ERROR: database "netbox" is being accessed by other users
DETAIL: There are 7 other sessions using the database.

I think this may have been what I was seeing the last time also, as it doesn't delete the current db therefore when I try to create the new one it says it already exists & when I them copy it throws up duplications. Which I guess make some sense, have you seen this before?

[trrunde]

you need to stop netbox first.
sudo systemctl stop netbox netbox-rq

then you can drop the database

[aquelf]

Yea looks a lot better now, managed to get the db moved over.

Thanks again folks, appreciate the help.