Type annotations, linting, unit tests (#5)

* Adding consistent type annotations to most places

Fixing linting warnings.
Added observationlist module
Adding unittests test_event_cluster, test_observationlist, test_security_event + test data
Miscellaneous small fixes and improvements.

* Some more linting and MyPy warnings

* Discovered wrong URL in setup.py

* Fixing some pyroma warnings about the packaging.

Author: ianhelle

README.md

@@ -64,6 +64,7 @@ The following types are built-in:
 - Hashes (MD5, SHA1, SHA256)
 - Windows file paths
 - Linux file paths (this is kind of noisy because a legal linux file path can have almost any character)
+
 You can modify or add to the regular expressions used at runtime.
 
 Output is a dictionary of matches (for single string input) or a DataFrame (for dataframe input).
@@ -92,6 +93,7 @@ This module has two classes for different services:
 
 - GeoLiteLookup - Maxmind Geolite (see <https://www.maxmind.com>)
 - IPStackLookup  - IPStack (see <https://ipstack.com>)
+
 Both services offer a free tier for non-commercial use. However,
 a paid tier will normally get you more accuracy, more detail and
 a higher throughput rate. Maxmind geolite uses a downloadable database,
@@ -109,7 +111,10 @@ items.
 The module contains functions to generate clusterable features from
 string data. For example, an administration command that
 does some maintenance on thousands of servers with a commandline such as:<br>
-```install-update -hostname {host.fqdn} -tmp:/tmp/{GUID}/rollback```<br>
+```
+install-update -hostname {host.fqdn} -tmp:/tmp/{GUID}/rollback
+```
+
 can be collapsed into a single cluster pattern by ignoring the character values
 in the string and using delimiters or tokens to group the values.
 
@@ -135,14 +140,9 @@ This is a collection of display and utility modules
 designed to make working with security data in Jupyter notebooks
 quicker and easier.
 
-- nbwidgets - groups common functionality such as list pickers,
-time boundary settings, saving and retrieving
-environment variables into a single line callable command.
-- nbdisplay - functions that implement common display of things like
-alerts, events in a slightly more consumable way than print()
-- entityschema - implements entity classes (e.g. Host, Account, IPAddress)
-used in Log Analytics alerts and in many of these modules.
-Each entity encaspulates one or more properties related to the entity.
+- nbwidgets - groups common functionality such as list pickers, time boundary settings, saving and retrieving environment variables into a single line callable command.
+- nbdisplay - functions that implement common display of things like alerts, events in a slightly more consumable way than print()
+- entityschema - implements entity classes (e.g. Host, Account, IPAddress) used in Log Analytics alerts and in many of these modules. Each entity encaspulates one or more properties related to the entity.
 
 [Notebooks Tools](./doc/NotebookWidgets.ipynb)
 
@@ -151,14 +151,8 @@ Each entity encaspulates one or more properties related to the entity.
 These components are currently still part of the nbtools sub-package but will be
 refactored to separate them into their own package.
 
-- query manager - collection of modules that implement common
-kql/Log Analytics queries using KqlMagic
-- security_alert and security_event - encapsulation classes for alerts
-and events. Each has a standard 'entities' property reflecting the
-entities found in the alert or event. These can also be used as
-meta-parameters for many of the queries. For example the query:
-```qry.list_host_logons(provs==[query_times, alert])``` will extract the
-value for the ```hostname``` query parameter from the alert.
+- query manager - collection of modules that implement common kql/Log Analytics queries using KqlMagic
+- security_alert and security_event - encapsulation classes for alerts and events. Each has a standard 'entities' property reflecting the entities found in the alert or event. These can also be used as meta-parameters for many of the queries. For example the query: `qry.list_host_logons(provs==[query_times, alert])` will extract the value for the `hostname` query parameter from the alert.
 
 ---
 
@@ -171,14 +165,10 @@ Requires sign-in to Azure Notebooks
 
 See the following notebooks for more examples of the use of this package in practice:
 
-- Windows Alert Investigation in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)
- or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)
-- Windows Host Explorer in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)
- or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)
-- Office 365 Exploration in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)
- or [NbViewer](https://nbviewer.jupyter.org/github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)
-- Cross-Network Hunting in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)
- or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)
+- Windows Alert Investigation in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Investigation%20-%20Process-Alerts.ipynb)
+- Windows Host Explorer in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Windows-Host-Explorer.ipynb)
+- Office 365 Exploration in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Office365-Exploring.ipynb)
+- Cross-Network Hunting in [github](https://github.com/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb) or [NbViewer](https://nbviewer.jupyter.org/github/Azure/Azure-Sentinel/blob/master/Notebooks/Sample-Notebooks/Example%20-%20Guided%20Hunting%20-%20Linux-Windows-Office.ipynb)
 
 ## To-Do Items
 
@@ -190,10 +180,8 @@ See the following notebooks for more examples of the use of this package in prac
 
 - msticpy is OS-independent
 - Requires Python 3.6 or later
-- Requires the following python packages: pandas, bokeh, matplotlib, seaborn, setuptools, urllib3,
-ipywidgets, numpy, attrs, requests, networkx, ipython, scikit_learn, typing
-- The following packages are recommended and needed for some specific functionality: Kqlmagic, maxminddb_geolite2,
-folium, dnspython, ipwhois
+- Requires the following python packages: pandas, bokeh, matplotlib, seaborn, setuptools, urllib3, ipywidgets, numpy, attrs, requests, networkx, ipython, scikit_learn, typing
+- The following packages are recommended and needed for some specific functionality: Kqlmagic, maxminddb_geolite2, folium, dnspython, ipwhois
 
 See [requirements.txt](requirements.txt) for more details and version requirements.
 

msticpy/_version.py

@@ -1,2 +1,2 @@
 """Version file."""
-VERSION = "0.1.04"
+VERSION = "0.1.5"