Merge branch 'develop' for version 0.20.0

Author: ddutt

README.md

@@ -92,7 +92,7 @@ One of Suzieq's powerful capabilities are asserts, which are statements that sho
 
 # Suzieq Data
 
-**Suzieq supports gathering data from Cumulus, EOS, IOS, IOSXE, IOSXR, JunOS(QFX, MX, EX, SRX supported), NXOS and SONIC routers, and Linux servers.** Suzieq gathers:
+**Suzieq supports gathering data from Cumulus, EOS, IOS, IOSXE, IOSXR, JunOS(QFX, EX, MX and SRX platforms and Evolved OS), Palo Alto's Panos (version 8.0 or higher), NXOS and SONIC routers, and Linux servers.** Suzieq gathers:
 
 * Basic device info including serial number, model, version, platform etc.
 * Interfaces

build-docker.sh

@@ -1,6 +1,5 @@
 #!/usr/bin/env bash
 
-cd $PYTHONPATH
 VERSION=`./suzieq/version.py`
 poetry build
 if [ $# -eq 0 ]; then

build/requirements.txt

@@ -1,4 +1,4 @@
-# Suzieq Priorities
+# SuzieQ Priorities
 
 (last updated June 2022)
 
@@ -8,12 +8,12 @@ order of what we want to deliver.  If we are missing something that would make
 it more useful to you, please let us know. The best way is to
 [file an issue](https://github.com/netenglabs/suzieq/issues/new/choose).
 
-We are trying to provide a mix of adding new collection, new analysis, and making Suzieq a better
+We are trying to provide a mix of adding new collection, new analysis, and making SuzieQ a better
 platform to build on.
 
 ## Areas of Development
 
-There are six major areas that Suzieq development can be broken down into.
+There are six major areas that SuzieQ development can be broken down into.
 
 1. Platforms (new NOS)
 1. Features
@@ -72,12 +72,11 @@ Given the categories, here is a rough list of tasks we will be tackling. We welc
 * Understand BGP routing policy and route maps / etc
 * ISIS
 * ~~Supporting 1M+ routes per device~~
-* Integration with other systems as both source of data and as a client.
 
 ## How You Can Help
 
 * By telling us which of these tasks you care to see sooner than later
 * Signing up to help with some of the tasks
-* Suggesting items that you care about but are missing from this list. The best way to do this is to open an issue. Please do add some color about how you intend to use Suzieq with that feature.
+* Suggesting items that you care about but are missing from this list. The best way to do this is to open an issue. Please do add some color about how you intend to use SuzieQ with that feature.
 * Writing some tests and/or documentation
 * Funding us to do the work :)

docs/2020-priority.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-At this point, Suzieq has two main processes: the poller (sq-poller), and the client (suzieq-cli). The poller is long running and writes data to parquet files. The client doesn't communicate with the poller, it just directly reads the data from parquet. 
+At this point, SuzieQ has two main processes: the poller (sq-poller), and the client (suzieq-cli). The poller is long running and writes data to parquet files. The client doesn't communicate with the poller, it just directly reads the data from parquet. 
 
 ## Relationships
 

docs/analyzer.md

@@ -1,8 +1,8 @@
-# Adding a New NOS to Suzieq
+# Adding a New NOS to SuzieQ
 
-Multi-vendor support is integral to how Suzieq is designed. If you're a developer trying to add a new NOS, here are the key steps that you need to accomplish:
+Multi-vendor support is integral to how SuzieQ is designed. If you're a developer trying to add a new NOS, here are the key steps that you need to accomplish:
 
-- Does the device have SSH, and or REST, support. Suzieq only supports these two methods, but relies on SSH access to automatically identify the NOS, unless you can supply the NOS identifier in the inventory output.
+- Does the device have SSH, and or REST, support. SuzieQ only supports these two methods, but relies on SSH access to automatically identify the NOS, unless you can supply the NOS identifier in the inventory output.
 - Decide the name you intend to use for your NOS. Picking a name that fits with what is popular is better (eos rather than arista-eos, for example). If there can be more than one major variation between the command outputs on different platforms (JunOS is a classic example), then you'll need to pick a name that's more precise (junos-qfx and junos-mx, for example).
 - For every file in suzieq/config/*.yml, look up the command that'll provide the equivalent information for that table.
 - Figure out a method that you can identify the NOS with. The most common command is "show version" that works across many NOS.
@@ -12,13 +12,13 @@ Lets dig in to each piece now. We'll start with a different order than the one l
 
 ## Identifying the NOS
 
-Suzieq can identify the NOS automatically if it can access the device via SSH. Every device's REST API differs from every other NOS' REST API, and worse, can change. For this reason, if you wish to use only REST access for a NOS, the inventory file MUST identify the NOS. Arista's EOS follows this model. NXOS, Junos and Cumulus all use SSH to poll the device.
+SuzieQ can identify the NOS automatically if it can access the device via SSH. Every device's REST API differs from every other NOS' REST API, and worse, can change. For this reason, if you wish to use only REST access for a NOS, the inventory file MUST identify the NOS. Arista's EOS follows this model. NXOS, Junos and Cumulus all use SSH to poll the device.
 
 ### Automatic Identification of NOS
 
 This is not a mandatory step, but this method enables easier inventory specification if provided.gi
 
-Find the command, via SSH, that can be used to identify the NOS type. Specifically, Suzieq needs to identify the following pieces of info for a node:
+Find the command, via SSH, that can be used to identify the NOS type. Specifically, SuzieQ needs to identify the following pieces of info for a node:
 
 - The device's unique NOS name
 - The device's model, if the NOS name is not unique across models, such as JunOS or IOS.
@@ -80,33 +80,33 @@ Once the output is extracted, you need to handle the parsing and specification o
 
 Since each NOS has its own command and output for parsing boot up time, retrieving the version of the NOS etc, you'll need to create a new NOS-specific class in the file suzieq/poller/nodes/node.py. Look at either the NxosNode class or JunosNode class to see how to add a new NOS type and what the additional routines to support are. Typically, you'll need to add two routines, `init_boot_time` and `_parse_device_type_hostname` routines to this Node class as the commands and outputs to get this information is specific to each NOS.  
 
-If you're using the REST API to access the information from this NOS, you will need to add another routine, `rest_gather` to this new NOS Node class. Look at EosNode's `rest_gather` method in the file to get the template for creating this class. In general, there's a little more work to do if you're using the REST API. This maybe a limitation of the current implementation in Suzieq rather than a reflection of REST support.
+If you're using the REST API to access the information from this NOS, you will need to add another routine, `rest_gather` to this new NOS Node class. Look at EosNode's `rest_gather` method in the file to get the template for creating this class. In general, there's a little more work to do if you're using the REST API. This maybe a limitation of the current implementation in SuzieQ rather than a reflection of REST support.
 
 ### Authentication Methods Supported
 
-At the time of this writing, Suzieq supports the following authentication methods:
+At the time of this writing, SuzieQ supports the following authentication methods:
 - Username/password
 - Private key files (recommended)
 - Private Key files with passphrase (recommended)
 - Use SSH Config file for more complex cases
 
-Suzieq MUST NEVER access any update command. The only commands it cares about are read commands. For this reason, its critical to assign a username for Suzieq that has only read permission.
+SuzieQ MUST NEVER access any update command. The only commands it cares about are read commands. For this reason, its critical to assign a username for SuzieQ that has only read permission.
 
 ## Adding the Commands to Gather Information
 
 Once you've created this information, you're now ready to gather inputs.
 
-For every file in config/*.yml in the Suzieq tree, determine the command that provides the information relevant to the table for the NOS you're adding support. For example, BGP information is extracted from the file config/bgp.yml. For Cumulus, the command to extract the relevant BGP information is "net show bgp vrf all neighbor json", and is visible under the cumulus NOS key in that file. Some of the tables need more than one command to gather the output at times (see NXOS' bgp command set in the same file).
+For every file in config/*.yml in the SuzieQ tree, determine the command that provides the information relevant to the table for the NOS you're adding support. For example, BGP information is extracted from the file config/bgp.yml. For Cumulus, the command to extract the relevant BGP information is "net show bgp vrf all neighbor json", and is visible under the cumulus NOS key in that file. Some of the tables need more than one command to gather the output at times (see NXOS' bgp command set in the same file).
 
 Pick a command output that is structured, specifically JSON. Unstructured output is acceptable, but you'll need to know how to write textfsm parsers for those commands. Choose a command that provides all the info over one that is structured, but incomplete. If multiple commands are used, its fine to have mix commands that provide structured output with commands that don't. Textfsm parsers MUST be present in the config/textfsm_templates directory.
 
-For each command, you'll need to advice Suzieq as to how to extract the relevant data. If this is too complicated for you to identify, just adding the command name, you can do something to help the Suzieq developers perform this task for you. Just add the command and leave out the normalize or textfsm key specification. Then, run the poller with the options --run-once=gather --output-dir=<outputdir>. This will create a set of ".output" files in the directory specified. The directory is created if not present. However, the outputs are appended to an existing file if present. For example, the lldp output is appended to the lldp.output file if such a file is present. Therefore, if you're debugging the command and the code, its important to remove all .output files from that directory before running the poller after the very first time. However, as we discuss in the subsequent paragraph, this step is not always required.
+For each command, you'll need to advice SuzieQ as to how to extract the relevant data. If this is too complicated for you to identify, just adding the command name, you can do something to help the SuzieQ developers perform this task for you. Just add the command and leave out the normalize or textfsm key specification. Then, run the poller with the options --run-once=gather --output-dir=<outputdir>. This will create a set of ".output" files in the directory specified. The directory is created if not present. However, the outputs are appended to an existing file if present. For example, the lldp output is appended to the lldp.output file if such a file is present. Therefore, if you're debugging the command and the code, its important to remove all .output files from that directory before running the poller after the very first time. However, as we discuss in the subsequent paragraph, this step is not always required.
 
 Tar this directory and supply it to one of the developers, Dinesh or Justin, at this time, and they can now do the remaining work to add support for extracting the data for the NOS you're adding support for. The more varied the output that is gathered, the better will be the support the developers can provide. For example, if you only provide good outputs, the developers may not be able to anticipate what needs to be done when the output is bad. You can gather the output several times by changing the condition on the devices and running the poller with --run-once=gather multiple times. The outputs can all be sent to the same directory or each can be put in a different directory, and all this input can be provided to the developers.
 
 ## Massaging the Output Gathered
 
-All data stored by Suzieq is structured and has a schema. This schema is stored in config/schema. In many cases, it is not possible for the command output to fit into the schema directly. In such cases, its required to massage the output to fit the schema. Each file in config/*.yml has a corresponding python file under suzieq/poller/services/ directory that contains the code to massage the output. For example, suzieq/poller/services/bgp.py contains code that massages the output for the BGP commands from different NOS. Look at an existing file to determine how to add support for your NOS.
+All data stored by SuzieQ is structured and has a schema. This schema is stored in config/schema. In many cases, it is not possible for the command output to fit into the schema directly. In such cases, its required to massage the output to fit the schema. Each file in config/*.yml has a corresponding python file under suzieq/poller/services/ directory that contains the code to massage the output. For example, suzieq/poller/services/bgp.py contains code that massages the output for the BGP commands from different NOS. Look at an existing file to determine how to add support for your NOS.
 
 ## Adding the NOS to the List of Known NOS
 

docs/architecture.md

@@ -1,6 +1,6 @@
 # Adding New Objects Including Command
 
-Suzieq exposes the underlying data as objects with methods. BGP is an example of such an object as are interfaces and MAC addresses. The methods are realized via engines. Engines themselves are a class with specific instantiations such as pandas, spark, modin, dask etc.
+SuzieQ exposes the underlying data as objects with methods. BGP is an example of such an object as are interfaces and MAC addresses. The methods are realized via engines. Engines themselves are a class with specific instantiations such as pandas, spark, modin, dask etc.
 
 So, to create a new object, you must first create an object under suzieq/sqobjects directory. Then create the engine implementations under engines/<engine>. For example, you’d create one for pandas under suzieq/engines/pandas. The engine methods inherit from the base engine method. If nothing more than what the base method provides is required, then the object-specific method can be empty.
 

docs/config_file.md

@@ -1,8 +1,8 @@
-Contributing code to SuzieQ typically requires programming in python (specifically python 3). The recommended best practice for programming in python requires the use of [virtual environments](https://realpython.com/python-virtual-environments-a-primer/). Suzieq relies on a number of other open source projects to do its job. Installing those packages and the right versions is also crucial before starting to develop for Suzieq.
+Contributing code to SuzieQ typically requires programming in python (specifically python 3). The recommended best practice for programming in python requires the use of [virtual environments](https://realpython.com/python-virtual-environments-a-primer/). SuzieQ relies on a number of other open source projects to do its job. Installing those packages and the right versions is also crucial before starting to develop for SuzieQ.
 
-We use [poetry](https://python-poetry.org/) to both setup the working virtual environment for Suzieq and to install the appropriate packages. Therefore, before you can get started in developing for Suzieq, you must get a working development environment. __I have developed only in Linux, but I know of others who've developed on macOS. I cannot vouch for the development on Windows because several packages work only on Linux or macOS__.
+We use [poetry](https://python-poetry.org/) to both setup the working virtual environment for SuzieQ and to install the appropriate packages. Therefore, before you can get started in developing for SuzieQ, you must get a working development environment. __I have developed only in Linux, but I know of others who've developed on macOS. I cannot vouch for the development on Windows because several packages work only on Linux or macOS__.
 
-Setting up the development environment for Suzieq involves the following steps:
+Setting up the development environment for SuzieQ involves the following steps:
 
 * Make sure you have a python3 version that is > 3.7.1 and less than 3.9.0. If you don't have a system provided python version that matches this requirement, you can use [pyenv](https://realpython.com/intro-to-pyenv/) to install one.
 * If you've used pyenv to install a specific python version, ensure you activate it.
@@ -12,4 +12,4 @@ Setting up the development environment for Suzieq involves the following steps:
 * Create the virtual environment and install the appropriate packages by typing: ```poetry install```
 * Activate the virtual environment by typing ```poetry shell```
 
-Now you're ready to get started writing code for Suzieq.
+Now you're ready to get started writing code for SuzieQ.