NAPALM stopped working; NetBox error display is showing raw HTML #9771

ZPrimed · 2022-07-18T20:12:49Z

ZPrimed
Jul 18, 2022

Hello all,

I configured NAPALM integration per the documentation, and I did have it working correctly at some point (circa v3.0 or maybe 3.1). I am currently on v3.2.6 (which is latest non-beta as I write this).

I just went in to check status on a couple devices and am getting the red dialog that there's an error when trying to use NAPALM to talk to a device.

Problems:

The error dialog is formatted incorrectly, so it's not centered in the screen and fully readable. It is off on the right 15% of my tab, and there's visible HTML in the dialog box (which I suspect is wrong).
It's telling me a module is missing, but I'm pretty sure that module is there... I source'd into the venv, and then ran pip freeze and the module that it complains is missing (netmiko.ssh_exception) should be there, since the entire netmiko module is part of the environment per pip...

---8<---
napalm==3.3.1
napalm-dellos10==1.0.7
napalm-ros==1.0.1
ncclient==0.6.13
netaddr==0.8.0
netmiko==4.1.1
ntc-templates==3.0.0
oauthlib==3.2.0
packaging==21.3
paramiko==2.11.0
---8<---

Here's a slightly redacted screenshot of the GUI...

Full error text below - this is exactly how it appears in the NetBox webUI - the HTML is visible and the error dialog is offset to the side of the page and not fully readable. I was only able to get all of the text by doing a "select-all" then copy / paste elsewhere.

2022-07-18 15:42 EDT [hostname-redacted] (v3.2.6)
Error Fetching Device Status

<!DOCTYPE html>
<html lang="en">

<head>
<title>Server Error</title>
<link rel="stylesheet" href="/static/netbox-light.css" />
<meta charset="UTF-8">
</head>

<body>
<div class="container-fluid">
<div class="row">
<div class="col col-md-6 offset-md-3">
<div class="card border-danger mt-5">
<h5 class="card-header">
<i class="mdi mdi-alert"></i> Server Error
</h5>
<div class="card-body">

<p>
A module import error occurred during this request. Common causes include the following:
</p>
<p>
<i class="mdi mdi-alert"></i> <strong>Missing required packages</strong> - This installation of NetBox might be
missing one or more required Python packages. These packages are listed in <code>requirements.txt</code> and
<code>local_requirements.txt</code>, and are normally installed as part of the installation or upgrade process. To
verify installed packages, run <code>pip freeze</code> from the console and compare the output to the list of
required packages.
</p>
<p>
<i class="mdi mdi-alert"></i> <strong>WSGI service not restarted after upgrade</strong> - If this installation has
recently been upgraded, check that the WSGI service (e.g. gunicorn or uWSGI) has been restarted. This ensures that
the new code is running.
</p>

<hr />
<p>
The complete exception is provided below:
</p>
<pre class="block"><strong>&lt;class &#x27;ModuleNotFoundError&#x27;&gt;</strong><br />
No module named &#x27;netmiko.ssh_exception&#x27;

Python version: 3.8.10
NetBox version: 3.2.6</pre>
<p>
If further assistance is required, please post to the <a href="https://github.com/netbox-community/netbox/discussions">NetBox discussion forum</a> on GitHub.
</p>
<div class="text-end">
<a href="/" class="btn btn-primary">Home Page</a>
</div>
</div>
</div>
</div>
</div>
</div>
</body>

</html>

Answered by candlerb

Jul 19, 2022

the module that it complains is missing (netmiko.ssh_exception) should be there, since the entire netmiko module is part of the environment per pip...

There's a more direct way to test this:

root@netbox3:~# source /opt/netbox/venv/bin/activate
(venv) root@netbox3:~# python
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netmiko.ssh_exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'netmiko.ssh_exception'
>>> import netmiko
>>> netmiko.ssh_exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <modul…

View full answer

candlerb · 2022-07-19T08:47:39Z

candlerb
Jul 19, 2022

the module that it complains is missing (netmiko.ssh_exception) should be there, since the entire netmiko module is part of the environment per pip...

There's a more direct way to test this:

root@netbox3:~# source /opt/netbox/venv/bin/activate
(venv) root@netbox3:~# python
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netmiko.ssh_exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'netmiko.ssh_exception'
>>> import netmiko
>>> netmiko.ssh_exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'netmiko' has no attribute 'ssh_exception'
>>>

To me, it looks like the error message is correct: there is no such thing as netmiko.ssh_exception. Therefore, you have to look at what code is actually trying to use this. However, the string "ssh_exception" does not occur anywhere in the Netbox source code, so it must be somewhere else.

The best way to do that is to get a full exception backtrace. Setting ADMINS and EMAIL in configuration.py will send out an E-mail if an exception occurs during a normal netbox operation - although I don't know if it does during AJAX calls.

Otherwise, there is a brute-force way:

(venv) root@netbox3:~# grep -R netmiko.ssh_exception /opt/netbox/venv
Binary file /opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/__pycache__/iosxr.cpython-38.pyc matches
/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/iosxr.py:from netmiko.ssh_exception import NetMikoTimeoutException
/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/iosxr.py:from netmiko.ssh_exception import NetMikoAuthenticationException
Binary file /opt/netbox/venv/lib64/python3.8/site-packages/napalm/pyIOSXR/__pycache__/iosxr.cpython-38.pyc matches
/opt/netbox/venv/lib64/python3.8/site-packages/napalm/pyIOSXR/iosxr.py:from netmiko.ssh_exception import NetMikoTimeoutException
/opt/netbox/venv/lib64/python3.8/site-packages/napalm/pyIOSXR/iosxr.py:from netmiko.ssh_exception import NetMikoAuthenticationException
(venv) root@netbox3:~#

I've checked the code in those files, and there's no conditional import or catching of ImportError. So it seems Napalm itself is broken, which is easy to test:

(venv) root@netbox3:~# python
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import napalm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/__init__.py", line 11, in <module>
    import napalm.iosxr
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/iosxr/__init__.py", line 22, in <module>
    from napalm.iosxr.iosxr import IOSXRDriver  # noqa
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/iosxr/iosxr.py", line 28, in <module>
    from napalm.pyIOSXR import IOSXR
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/__init__.py", line 26, in <module>
    from napalm.pyIOSXR.iosxr import IOSXR
  File "/opt/netbox/venv/lib/python3.8/site-packages/napalm/pyIOSXR/iosxr.py", line 38, in <module>
    from netmiko.ssh_exception import NetMikoTimeoutException
ModuleNotFoundError: No module named 'netmiko.ssh_exception'
>>>

There you go. Looks to me like a bug you can report upstream, since you can easily reproduce it standalone without Netbox.

FYI, my local_requirements.txt contains:

napalm
napalm-ros

and the versions I have:

(venv) root@netbox3:~# pip freeze | grep napalm
napalm==3.3.1
napalm-ros==1.0.1
(venv) root@netbox3:~# pip freeze | grep miko
netmiko==4.1.1
paramiko==2.11.0

8 replies

ZPrimed Jul 20, 2022
Author

Weird, I swear I thought I had posted another follow-up on this from a work computer, but maybe I forgot to hit send...

napalm-ros is the problem. It is requiring napalm == 3.* in the setup.py "manifest" (not sure if that's the proper term but I'm going with it).

If you drop napalm-ros from local_requirements.txt, NetBox should pull napalm 4.0.0 (and whatever netmiko is acceptable to that) and the errors go away... but you won't be able to use napalm against your Mikrotik stuff. 😛

I filed an issue on the napalm-ros github, not sure if there's some other (easier?) way to hack around its (possibly unnecessary?) requirement of napalm 3.x instead of 4.x though.

candlerb Jul 20, 2022

Nice debugging, thank you!

candlerb Jul 20, 2022

Maybe a workaround would be to pin napalm, napalm-ros and netmiko to older versions?

It now looks like a pip broken dependencies issue. We have correctly picked up napalm==3.3.1 (due to what napalm-ros requires) but not a version of netmiko which works with it; either because napalm 3 didn't declare the correct range of versions it depends on (see that unmerged patch) or because pip isn't calculating transitive dependencies correctly.

Hence nailing everything to a set of good versions which work together may be enough to solve this for now.

candlerb Jul 20, 2022

Looking in git, napalm with tag 3.3.1 has a declared dependency on netmiko>=3.1.0, so I have put the following in Netbox's local_requirements.txt:

napalm
napalm-ros
netmiko>=3.1.0,<4.0.0

And that works for me, yay!

(Aside: napalm 3.3.1 also depends on paramiko>=2.6.0 but I'm assuming that 2.11.0 that it picks up is not too new)

candlerb Nov 11, 2022

Link to the napalm-ros issue: napalm-automation-community/napalm-ros#98

Unfortunately this issue was closed with the comment "Use version pinning.". That is, it remains up to you to work around this problem in your local_requirements.txt.

NAPALM stopped working; NetBox error display is showing raw HTML #9771

Uh oh!

Uh oh!

ZPrimed Jul 18, 2022

Replies: 1 comment · 8 replies

Uh oh!

Uh oh!

candlerb Jul 19, 2022

Uh oh!

ZPrimed Jul 20, 2022 Author

Uh oh!

candlerb Jul 20, 2022

Uh oh!

Uh oh!

candlerb Jul 20, 2022

Uh oh!

candlerb Jul 20, 2022

Uh oh!

candlerb Nov 11, 2022

ZPrimed
Jul 18, 2022

Replies: 1 comment 8 replies

candlerb
Jul 19, 2022

ZPrimed Jul 20, 2022
Author