Intermittent corruption of imported module in custom validator

[llamafilm]

I have a weird issue where a Netbox custom validator sometimes fails because an imported module has become corrupt. I'm not sure if that's the right word, but the actual symptom is this:

AttributeError: module 'bloxone' has no attribute 'b1ddi'

The imported module is this: https://github.com/ccmarris/python-bloxone. That attribute exists and my custom validator code works fine, so I don't think there's anything wrong with the module. This exception has happened twice in the last year so it's very rare. But once it happens, the custom validator is completely broken, and it throws this exception every time until I run systemctl restart netbox*.

I've reviewed the system metrics over the last few days and I don't see any sign of resource exhaustion.

My custom validators and scripts are stored in a git repo which gets synced automatically after pushing updates. The sync essentially boils down to git pull && manage.py syncdatasource. That works well, without having to restart Netbox. This particular validator hasn't changed in a long time, but maybe it's possible that the git sync overwrote that file anyway.

traceback.txt

[llamafilm]

My custom validators are defined as dotted classes like this:

    CUSTOM_VALIDATORS['ipam.ipaddress'] = (
        'custom_validators.BloxOneIPAddressValidator',
    )

I wonder if it would help to change them to direct class references instead. If I do that, then they cannot be updated dynamically without restarting Netbox. But that's ok, these don't change nearly as often as custom scripts.

    CUSTOM_VALIDATORS['ipam.ipaddress'] = (
        custom_validators.BloxOneIPAddressValidator(),
    )

[rfdrake]

Could you show the code for the custom validator? I suspect bloxone is occasionally not being created so it ends up as NoneType, or some other object type that doesn't have a b1ddi attribute. It's possible this is due to a timeout to a remote host, a DNS error, or something else that might be intermittent.

You could throw some debug statements in the validator, like, if not bloxone: self.fail('something went wrong with bloxone, here is everything I know...') or if you want to fiddle with logging:

import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

class MyValidator(CustomValidator):
  def validate(self, obj, request):
    logger.debug(f"Here is all I know about bloxone {dir(bloxone)}")

This will (probably) show up in your netbox logs, but that depends on how everything on the system is configured. It might end up on stderr somewhere or into a netbox.log file.