Devices: Fix how device interacts with sqpoller table

Essentially, we want the device table to show all devices in the
inventory, even those that were never polled due to whatever reason that
connectivity could not be established. To do this, we pull the info
about devices from the poller table. However, we were pulling all
devices from the poller table which meant even those that were
successful and were already in the device table. This meant any filter
the user had applied such as specifying model, vendor, OS, etc. were
lost when we did the merge.

To avoid this issue, but stay true to the reason we were pulling data
from the poller table, we change the poller query to only return info
about devices that have failed. This fix however has the limitation that
it doesn't handle the case where a device has failed presently, but it
once was polled, and a user filter prevents a device from being selected,
by the device table. We'll tackle that case separately.

Signed-off-by: Dinesh Dutt <[email protected]>

Author: ddutt

suzieq/engines/pandas/device.py

@@ -52,12 +52,15 @@ def get(self, **kwargs):
         if view == 'latest' and 'status' in df.columns:
             df['status'] = np.where(df.active, df['status'], 'dead')
 
-        poller_df = pd.DataFrame()
+        poller_df: pd.DataFrame = pd.DataFrame()
+
+        # Retrieve the devices that have possibly not made it into the device
+        # list because they never could connect due to any reason
         if not ignore_neverpoll:
             poller_df = self._get_table_sqobj('sqPoller').get(
                 namespace=kwargs.get('namespace', []),
                 hostname=kwargs.get('hostname', []),
-                service='device',
+                service='device', status='fail',
                 columns='namespace hostname status timestamp'.split())
 
         if not poller_df.empty:
@@ -77,7 +80,7 @@ def get(self, **kwargs):
                 .reset_index(drop=True)
 
             df = df.merge(poller_df, on=['namespace', 'hostname'],
-                          how='outer', suffixes=['', '_y'])  \
+                          how='outer', suffixes=('', '_y')) \
                 .fillna({'bootupTimestamp': 0,
                          'active': True})
 
@@ -121,6 +124,7 @@ def get(self, **kwargs):
         # The poller merge kills the filtering we did earlier, so redo:
         if status:
             df = df.loc[df.status.isin(status)]
+
         if os_version:
             opdict = {'>': operator.gt, '<': operator.lt, '>=': operator.ge,
                       '<=': operator.le, '=': operator.eq, '!': operator.ne}