Unassigned IP address migration issue

[irozkov]

NetBox version

v3.2.2

Python version

3.8

Steps to Reproduce

Clean install NetBox 3.2.1 or NetBox 3.2.2
Dump NetBox 3.1.1 database
Run upgrade.sh to migrate from NetBox 3.1.11

Expected Behavior

For IPs not assigned to the object API to return / keep "assigned_object_type" null value

Example (before migration / NetBox 3.1.11):

{
"id": 5631,
"url": "http://localhost:8002/api/ipam/ip-addresses/5631/",
"display": "10.200.0.5/32",
"family": {
"value": 4,
"label": "IPv4"
},
"address": "10.200.0.5/32",
"vrf": {
"id": 28,
"url": "http://localhost:8002/api/ipam/vrfs/28/",
"display": "TNX",
"name": "TNX",
"rd": null
},
"tenant": {
"id": 20,
"url": "http://localhost:8002/api/tenancy/tenants/20/",
"display": "TNX",
"name": "TNX",
"slug": "tnx"
},
"status": {
"value": "active",
"label": "Active"
},
"role": null,
"assigned_object_type": null,
"assigned_object_id": null,
"assigned_object": null,
"nat_inside": null,
"nat_outside": null,
"dns_name": "aws-dc1",
"description": "",
"tags": [
{
"id": 5,
"url": "http://localhost:8002/api/extras/tags/5/",
"display": "MSR-AVL-INT",
"name": "MSR-AVL-INT",
"slug": "msr-avl-int",
"color": "ffeb3b"
}
],
"custom_fields": {},
"created": "2020-09-06",
"last_updated": "2020-09-06T18:04:42.114987Z"
}

Observed Behavior

For IPs not assigned to the object receive API response with "assigned_object_type" set to "dcim.interface"

Example (after migration / NetBox 3.2.2):

{
"id": 5631,
"url": "http://localhost:8003/api/ipam/ip-addresses/5631/",
"display": "10.200.0.5/32",
"family": {
"value": 4,
"label": "IPv4"
},
"address": "10.200.0.5/32",
"vrf": {
"id": 28,
"url": "http://localhost:8003/api/ipam/vrfs/28/",
"display": "TNX",
"name": "TNX",
"rd": null
},
"tenant": {
"id": 20,
"url": "http://localhost:8003/api/tenancy/tenants/20/",
"display": "TNX",
"name": "TNX",
"slug": "tnx"
},
"status": {
"value": "active",
"label": "Active"
},
"role": null,
"assigned_object_type": "dcim.interface",
"assigned_object_id": null,
"assigned_object": null,
"nat_inside": null,
"nat_outside": null,
"dns_name": "aws-dc1",
"description": "",
"tags": [
{
"id": 5,
"url": "http://localhost:8003/api/extras/tags/5/",
"display": "MSR-AVL-INT",
"name": "MSR-AVL-INT",
"slug": "msr-avl-int",
"color": "ffeb3b"
}
],
"custom_fields": {},
"created": "2020-09-06T00:00:00Z",
"last_updated": "2020-09-06T18:04:42.114987Z"
}

[irozkov]

Hi Jeremy,

As you moved this issue to discussion, can you confirm this is the intended change, and if so, what is the intention?. This set of values for unassigned IP:
"assigned_object_type": "dcim.interface",
"assigned_object_id": null,
"assigned_object": null,
looks like an indication of an error linking IP to the specific interface, not as IP not assigned to the interface (what it is in reality), so I would appreciate it if you could provide some background or intention of this change in API response.

Igor

[irozkov]

Thanks Jeremy,

the only step we need to reproduce this is to perform migration, in my case 3.1.11 -> 3.2.x
So, Example (before migration / NetBox 3.1.11) is data how it presented for arbitrary not linked IP via API in NetBox 3.1.11, important piece:
"assigned_object_type": null,
"assigned_object_id": null,
"assigned_object": null,

Then I do

1. pg_dump -h db.host -U postgres --exclude-table-data=extras_objectchange -v netbox > netbox.sql
2. dump DB into fresh NetBox 3.2.1 or 3.2.2
3. run upgrade.sh
4. run manage.py createsuperuser

And result is what is presented under Example (after migration / NetBox 3.2.2), important piece for same arbitrary taken not linked IP:
"assigned_object_type": "dcim.interface",
"assigned_object_id": null,
"assigned_object": null,

Whilst I would expect exact same data as was before migration/in NetBox 3.1.11

Never reported pure migration problem sorry if it was set not clear

[Houtek]

Hi all,
we have the same problem. We migrated from NB 3.2.1 to NB 3.2.2 and the issue occured.

[jeremystretch]

the only step we need to reproduce this is to perform migration, in my case 3.1.11 -> 3.2.x

You'll need to either share the database you're using, or provide instructions for someone else to create the conditions under v3.1.11 that effect the reported behavior. Many people have upgraded to v3.2 without issue, so you'll need to determine what's unique about your database that's causing a problem.

[irozkov]

They likely just have not tried to consume this data via API and check this prop :)
I don't believe something special with the data; IPs in question are all different; we have migrated these records (real and complete data as returned by API in examples) multiple times, starting from NetBox 1.x -> 2.x migration. You will not notice something wrong via UI; all looks normal. Even more, if you will edit this record without changing anything just after the migration (go to UI->IP->Edit press Save), API will return the correct data (adding changelog for this operation on just migrated DB):

Pre-Change Data
vrf: 28
role: ""
tags: [
"MSR-AVL-INT"
]
status: "active"
tenant: 20
address: "10.200.0.5/32"
created: "2020-09-06T00:00:00Z"
dns_name: "aws-dc1"
nat_inside: null
description: ""
last_updated: "2020-09-06T18:04:42.114Z"
custom_fields: {}
assigned_object_id: null
assigned_object_type: 5

Post-Change Data
vrf: 28
role: ""
tags: [
"MSR-AVL-INT"
]
status: "active"
tenant: 20
address: "10.200.0.5/32"
created: "2020-09-06T00:00:00Z"
dns_name: "aws-dc1"
nat_inside: null
description: ""
last_updated: "2022-05-04T15:04:14.271Z"
custom_fields: {}
assigned_object_id: null
assigned_object_type: null

Anyway, I will try to investigate more, and thanks for the answers, appreciate it. We can't share DB as it contains customers' data, but we have 628 not assigned IPs in DB as today. All of them return null assigned_object_type in NetBox 3.1.11.
After migration, 299 IPs return assigned_object_type dcim.interface (other keep correct null) Of them all are different, say IPs having dns name or not, having tags or not, of same subnet or same VRF can fall into changed on migration and into not changed/correct...

[markkuleinio]

You'll need to either share the database you're using

@jeremystretch I have discovered the same issue with some IP addresses after upgrading from 3.1.11 to 3.2.2:

  "assigned_object_type": "dcim.interface",
  "assigned_object_id": null,
  "assigned_object": null,
...
  "created": "2020-05-16T00:00:00Z",
  "last_updated": "2020-05-16T15:57:47.792393Z"

I can provide my 3.1.7 database dump to you (the main developers) privately if you are willing to investigate this internally, I just don't want to share the database contents in public. Let me know if this is ok, I can then email the original dump (under 130 kB zipped) to you at your NS1 address (as shown in your git commits).

My process was:

1. Dump the database from 3.1.7 installation (upgraded many times over the time, originally installed as some 2.x version)
2. Install new 3.1.11, import the database dump, run upgrade.sh etc.
3. Verify that an example address has assigned_object_type as null
4. Upgrade to 3.2.2 (run upgrade.sh etc)
5. See that the example address now had assigned_object_type == 'dcim.interface'

The server used for this 3.1.11 and 3.2.2 test is a Debian 11 host, with Python 3.9.2 and Postgresql 13.5-0+deb11u1.

[irozkov]

Sorry for the delay,
investigated DB issue, and it seems the problem is not with 3.2.x migration but with data that got misformed before. 3.1.11 DB itself contains ipam_ipaddress records with assigned object id set to null but type set to 5 (dcim.interface). I suppose this is some artifact from NetBox 2.9, which introduced all this stuff, migration. However, having this data (null/5), REST API returns 'correct' null/null for 3.1.11 but starts to return null/5 for 3.2.x. I'm probably going to reset such records to assigned_object_type_id null in the current 3.1.11 DB; Jeremy, thanks for your time and comments; I appreciate that.