Custom fields with double underscore in name behave differently

[yarnocobussen]

NetBox version
v2.11.3

Python version
Python 3.6.9

I upgraded an instance of Netbox 2.9.1 to 2.11.3 and ran my test scripts. I noticed the following:
Searches including a custom field whos name contains 2 consecutive underscores yield 0 results.
(Even though the data is there and displays fine when not searching for it directly.)

Steps to reproduce:
What I did to make sure I had a clean test:

• Deployed a fresh Ubuntu 18.04.5 LTS VM
• Followed the install guide with a tarball instal and nginx.
• Created custom field, type text, name "test_single", attach to ipam > prefix.
• Created custom field, type text, name "test__double", attach to ipam > prefix.
• Created prefix, filled in both custom fields as "foo"
• Search for all prefixes:

https://10.30.13.12/api/ipam/prefixes/
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"id": 1,
"url": "https://10.30.13.12/api/ipam/prefixes/1/",
"display": "10.0.0.0/24",
"family": {
"value": 4,
"label": "IPv4"
},
"prefix": "10.0.0.0/24",
"site": null,
"vrf": null,
"tenant": null,
"vlan": null,
"status": {
"value": "active",
"label": "Active"
},
"role": null,
"is_pool": false,
"description": "",
"tags": [],
"custom_fields": {
"test_single": "foo",
"test__double": "foo"
},
"created": "2021-05-07",
"last_updated": "2021-05-07T21:59:47.671738Z"
}
]
}

• Search containing a custom field with a single underscore in the name:

https://10.30.13.12/api/ipam/prefixes/?cf_test_single=foo
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"id": 1,
"url": "https://10.30.13.12/api/ipam/prefixes/1/",
"display": "10.0.0.0/24",
"family": {
"value": 4,
"label": "IPv4"
},
"prefix": "10.0.0.0/24",
"site": null,
"vrf": null,
"tenant": null,
"vlan": null,
"status": {
"value": "active",
"label": "Active"
},
"role": null,
"is_pool": false,
"description": "",
"tags": [],
"custom_fields": {
"test_single": "foo",
"test__double": "foo"
},
"created": "2021-05-07",
"last_updated": "2021-05-07T21:59:47.671738Z"
}
]
}

• Search containing a custom field with 2 consecutive underscores in the name:

https://10.30.13.12/api/ipam/prefixes/?cf_test__double=foo
{
"count": 0,
"next": null,
"previous": null,
"results": []
}

I tried to renaming both fields to see what that does. The renamed 'test__double' field's values are now null. Not a working solution:

https://10.30.13.12/api/ipam/prefixes/
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"id": 1,
"url": "https://10.30.13.12/api/ipam/prefixes/1/",
"display": "10.0.0.0/24",
"family": {
"value": 4,
"label": "IPv4"
},
"prefix": "10.0.0.0/24",
"site": null,
"vrf": null,
"tenant": null,
"vlan": null,
"status": {
"value": "active",
"label": "Active"
},
"role": null,
"is_pool": false,
"description": "",
"tags": [],
"custom_fields": {
"test_single_renamed": "foo",
"test_double_renamed": null
},
"created": "2021-05-07",
"last_updated": "2021-05-07T21:59:47.671738Z"
}
]
}

I decided we could migrate data to new fields and delete the old ones.
However, the problematic custom field doesn't seem to delete correctly:

• Deleted the 'test_single' custom field, restarted netbox and netbox-rq.
• Re-created the field with the same name, searched, value on prefix is now null.
• Deleted the 'test__double' custom field, restarted netbox and netbox-rq.
• Re-created the field with the same name, searched, value on prefix is 'foo' again.

I'm unsure how to proceed in a clean manner, other than creating an entirely new instance and migrating all data. Before doing all that, though: Was using these names a mistake on our part that we could have known about, or did we run into a bug / something that changed in later versions?

Not sure if it matters, but I also upgraded a clone of our prod box from 2.9.1 to 2.10.5 to see if it works there. (2.9.2 through 2.10.4 failed with a pip version error, even after manually upgrading it in the venv. The upgrade.sh changes in 2.10.5 fixed that). The field also fails there. That at least narrows down the problematic version, I hope.

[candlerb]

Double-underscores have a special meaning in Django filters: they are used for lookup relationships, e.g. interface__device__name=switch1.

cf_test__double=foo is taking cf_test attribute, following that link to another object, and accessing the double attribute of that object (which is not currently any use, because custom fields today can't reference other objects)

In principle anything is possible with sufficient effort. Maybe Netbox could be changed to allow such custom field names, e.g. by escaping double-underscore to %5F%5F in URLs or something like that. In practice, it is going to be pragmatic not to use double-underscores in custom field names. Personally I think Netbox's validation should be enhanced to prevent custom fields with such unusable names being created.

As for renaming or removing currently broken custom fields: I'd be inclined to do that in SQL. In Netbox v2.10 and later there's a custom_field_data JSON field on every object - you'd need to edit that JSON, as well as updating the extras_customfield table. In older versions there was a global custom_field_values table, linked to custom_field, so just changing the name inside custom_field should be sufficient.

[yarnocobussen]

Thank you for clarifying this. The SQL solution worked as you predicted for my v2.9.1 instance.
I changed the values for name in extras_customfield and everything kept functioning properly.
(The table name was actually called extras_customfield and not custom_field in 2.9.1 as well)
Everything also remained functional after upgrading the instance to v2.11.3.

Should I open an issue regarding the validation?

[candlerb]

Yes I think so, thanks.