Hotfix - Azure cloud settings fix

Minor fix to cope with incomplete configuration in Azure settings.

1.4.0 Hotfix

Release details

Some of the features scheduled to go into the 1.4.0 release did not get published correctly.
1.4.1 should fix this.
It also includes some updates to fix problems in the Process Tree visualization.

Fixes

b2b7fd8@Bug fixes for process_tree.py and related modules.
Fixed:

• not working with generic data input.
• not displaying command line correctly
• not display time in Hover tool correctly
• not truncating command line for display correctly
• missing params in doc string
• removed dependency on Linux auditd schema for converting numeric fields to strings

Azure cloud, Matrix plot and ProcessTree for MDE

Highlights

We've added comprehensive support for Azure sovereign clouds (China, Germany and US Government).
To set the appropriate cloud open MpConfigEdit and choose the Azure tab.

Read more about this here

There are also additions and improvements to the visualizations.

Matrix Plot is an interaction plot letting you visualize the interactions between two sets of entities.
A typical use would be seeing connections between source and destination IP Addresses.

Read more about this here

Another visualization change is a major overhaul of the ProcessTree visualization. This now accepts generic data
sets (not specifically Azure Sentinel) and has special support for Microsoft Defender for Endpoint logs.
The data format should be auto-detected.

Finally, we've also collected our visualization methods into a single pandas accessor - mp_plot.
Using this accessor you can plot any of the visualizations from a pandas dataframe:

df.mp_plot.timeline()
df.mp_plot.process_tree()

New Features

889112d@Matrix plot visualization (#197)* Matrix plot visualization
Includes

• vis/matrix_plot.py - main plotting code
• MatrixPlot.ipynb - notebook
• visualization/MatrixPlot.rst - documentation
• tests/vis/test_matrix_plot.py - unit test
• mp_pandas_plot - generic pandas extension "mp_plot" that includes all
  visualizations

d425c77@Process tree updates to handle more generic data (#178)*

• Tidying up some of the code in timeline.py
• Moving the pd accessors to timeline_pd_accessor.py
• Updating notebook EventTimeline.ipynb and documentation EventTimeline.rst
• Unit tests for timeline controls in test_timeline.py
• Removed line to slim down columns - need to keep original data columns because they may be
  referenced in source_columns, etc.
• Process tree updates for MDE process data
• Updating docs, adding pid_fmt parameter
• Adding explanation of schema to NB and doc
• Adding updated notebook
• Expanded the schema table to show all types.

a00741d@Adding support for multiple Azure Clouds (#193)
Co-authored-by: Pete Bryan pebryan@microsoft.com

• Adding cloud options to settings, kql_driver and keyvault
• Updated help text and combined CESimpleSettings class.
• Azure auth multicloud support
• Replacing hard-coded URIs with msrestazure enums
• Modularize multicloud support
• Refactoring code to use AzureCloudConfig instead of reading directly from config
  • azure_auth.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init, spelling corrections
  • azure_auth_core.py - create AzureCloudConfig, and _AzCachedConnect classes to read cloud config values and cache tokens
  • cred_wrapper.py - minor formatting
  • keyvault_settings.py - use AzureCloudConfig
  • ce_common.py - uses AzureCloudConfig in get_def_tenant_id function
  • azure_data.py - uses AzureCloudConfig, added credential_scopes option to client class inits, spelling corrections
  • azure_sentinel.py - uses AzureCloudConfig. added functions to set default subscription and workspace, changed _build_paths to method
  • kql_driver.py - uses AzureCloudConfig, added load_mode == Silent, added mp_az_auth to control using Azure auth instead of native Kqlmagic auth,
  • resource_graph_driver.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init
  • test_azure_auth_core - Added test for AzureCloudConfig class
  • test_resource_graph_driver.py, test_azure_data.py, test_azure_sentinel.py - added real credential classes to test
  • msticpyconfig-test.yaml, msticpyconfig.yaml - added Azure sections to test config.
• Update azure-mgmt-subscription and azure_mgmt-resource versions

Fixes

d81801d@Ianhelle/tests and tidy 2021 09 01 (#200)*

• Added separate unit test for nbwidgets to cover cases missed in notebook.
• Removing deprecated code from utility.py
• Removing unneeded function from mordor_driver.py
• Sourcery code refactor in entity.py
• Deprecating create_ip_record function in ip_utils.py
• Split wsconfig tests out of test_pkg_config.py
• Adding test_wsconfig.py. Added new test for couple of utility methods
• Removing redundant import from test_nbwidgets.py
• Moved lots of test files into sub-package folders corresponding to tested modules
• Merged a few tests into single files - test_entity.py and test_timeline.py
• Re-enabling IP_stack/geoip notebook test in test_geoip.py
• Updating use of tqdm to remove deprecated tqdm_notebook in utility.py
• Adding some debugging output to geoip GeoLite class.
• Updating test_geoip.py to use predictable msticpyconfig-test.yaml
• Updating test to restore env var values after test in test_pkg_config.py
• Fixing aliases element in kql query records
• Handle aliases as list or string in help

• Updates from PR review

f58e35c@Changing behavior so that vtlookupv3.py does not throw an exception when an item is not found. (#199)

• This is especially annoying when doing multiple look-ups and one item is not found.
• Minor changes to VTLookupV3.ipynb - to use get_object API from the vtlookupv3 library

d425c77@Process tree updates to handle more generic data (#178)

• Fixing bug in customjs and adding mde test files
• Update NOTICE.txt
• Fixing progress widget to use correct visibility settings.
• Changing user_config to be more defensive against bad settings.
• Error in query_store passing list instead of strings to MsticpyUserConfigError exception

• Adding a bit more text/warning to check the browser.

Co-authored-by: Pete Bryan pebryan@microsoft.com
ff87ebb@cred-scan suppression (#196)

Kql from Python Scripts, Mordor provider fixes

New Features

We've updated the Kql/Azure Sentinel data provider so that it no longer depends on
IPython/Jupyter functionality to work. This prevented using the provider from
Python scripts and applications. The new implementation uses Kqlmagic as
a standard import and no longer requires IPython magics to work.

08c7cf1@Ianhelle/kql as function 2021 08 04 (#190) Kqlmagic operations called as func vs. IPython magics
Now using Kqlmagic.kql() function in kql_driver, allowing it to be used from python code.

Fixes

The main fix here was to adapt to changes in the Mordor attack data repository
Mordor is now known as OTRF Security Datasets
(Although it will continue to be known as Mordor in MSTICPy for the time being).

• 17f9704@Ianhelle/modor url refix 2021 08 04 (#189)
• fabde53@fix img src for broken preview images (#191)
• 17b9227@Pebryan/7 16 21 sent api update (#187)* added incident collection
  • Updated pandas version to pandas>=1.1.5
  • Updated docstrings for Azure and Azure Sentinel APIs

Potentially breaking changes

• MSTICPY now requires pandas 1.1.5 as a minimum

Timeline duration, MDE Query upgrade

New Features

Timeline duration visualization #176

This new variant of the timeline control allows you to group by one or more columns and view timelines as start/end of activity related to those columns. The graphic below shows durations of communication between different source and destination IPs using a given protocol.

Like other timeline controls it's accessible as a pandas extension.

az_net_flows_df.mp_timeline.plot_duration(
    group_by=["SrcIP", "DestIP", "L7Protocol"]
)

You can also import and use the plot function directly:

from msticpy.nbtools.timeline_duration import display_timeline_duration

display_timeline_duration(
    host_logons,
    group_by="Account",
    ref_events=host_logons.sample(3),
    ref_col="TargetUserName",
);

Read about this in our Event Timeline documentation

Fixes

#184 MDEFix Update MDE connectors

The Microsoft Defender for Endpoint provider and queries have had a major update, correcting many schema issues (we were still using old column names in a number of cases

#185 from microsoft/ianhelle/init_notebook_changes-2021-07-14

• Removed nb_check.py update from azure_ml_tools.py
• Updated text about how to fix missing PyGObject
• Fixed MpConfigEdit widget to show checkbox for making backups prior to updating msticpyconfig
• Fix to remove casefold of OptionButtons value

8707868@Fixing Mordor and azure_ml_tools tests

• Fixed Mordor URL for mordor data provider

1ec1a00@bypassing GeoIP notebook test because of IPStack intermittent problems (#181)

Hotfix for notebooks #2 - Pivot exceptions and nbinit

Fixes

788b4ee@

• Added new context manager in MsticpyUserError to stop standard display of msticpy user exceptions.
  Note this doesn't suppress the exception, just stops exception being output to notebook even if exception is caught.
  
  Example:

from msticpy.common.exceptions import MsticpyUserError
with MsticpyUserError.no_display_exceptions():
   # do stuff that may cause exceptions to be display.

• Fixed typo in path for msticpyconfig.yaml was causing searches to always fail.
• Updated test_nbinit.py - fix one test. Using temp_path instead of tmpdir fixture.

Hotfix for Azure Sentinel notebooks

A few hotfixes for bugs

33e9fd4@Fixes to data_providers, data view, pivots and nbinit

• Replace list.pop with list.remove in data_providers.py - caused an error when using provider internal query_time control to set query start and end params.
• Add exception catching around Pivot instantiations in pivot_register_reader.py to prevent errors when loading Pivots with some
  providers missing configuration settings.
• Prevented push_notebook from being called before displaying control - this was causing an error when first displaying the DataViewer control
• Removed pandas option to return html schema in nbinit.py - Azure ML notebook dataviewer throws a react exception when trying to display DataFrame columns with complex types (e.g. dict or list vs. string/int/etc). Turning this off prevents the AML viewer being used by default.

Dataview, Azure Resource Graph and Sumologic driver

Highlights

The highlights of this release (which is really 1.2.0 with some last-minute fixes) are: two new data providers for Azure Resource Graph and Sumologic and a DataViewer control for pandas dataframes.

Azure Resource Graph provider

The Azure Resource graph
provider lets you query Azure resources using KQL queries. This works much like our other
query providers and gives you a lot more flexibility in navigating around Azure resources
than the current AzureData provider.

Explore more in the Resource Graph Driver notebook and the online docs

Many thanks to Ryan Cobb @rcobb-scwx for creating and contributing this.

Sumologic provider

Sumo Logic Inc. , is a cloud-based machine data analytics service
focusing on security, operations and BI use cases. This provider allows you to connect to
and query your data from MSTICPy via their Search API
Explore more in the Sumologic Data Connector notebook

Many thanks to Julien (@juju4) for building and contributing this.

DataViewer

The data viewer uses the [Bokeh DataTable control[(https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html#datatable)
to display and browse through data in a pandas DataFrame. It lets you sort by column, choose which columns to display and filter by multiple columns. It keeps a synchronized copy of the DataFrame with column choice and filtering applied, so that you can always access the data as it appears in the control.

Explore more in the DataViewer notebook
and the Dataview online docs

New Features and Updates

ab6eb73 - (#164) DataViewer control

• DataViewer control - Bokeh based data table control
• Documentation in DataViewer.rst
• Notebook in DataViewer.ipynb

3ab31ad- Azure Resource Graph Data Provider (#167)

• Co-authored-by: Ryan Cobb rcobb@secureworks.com
• Co-authored-by: Pete Bryan peter.bryan@microsoft.com

2a33d61- Sumologic Data Provider (#165)

• Co-authored-by: juju4 juju4@users.noreply.github.com
• Co-authored-by: Ian Hellen ianhelle@microsoft.com

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

• Updates to init_notebook in nb_init.py
  • Moved most of the logic of the Azure Sentinel notebooks nb_check script into msticpy - azure_ml_tools
  • Will now create msticpyconfig.yaml if it doesn't exist
  • Output message from init_notebook when msticpyconfig.yaml is created in nbinit.py

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

• Adding QueryTime instance to Dataproviders in data-data_providers.py. Let's you specify time range (or use default 1 day range) for all queries from provider
• Added Browshot to mpconfig_defaults.yaml so that it can be edited in MP settings editor

ab6eb73 (#164)

• added recent blog articles and video to a renamed resources page for RTD - blob_articles.

Fixes

8214c90 - Minor fixes to VT TI Provider and TIBrowser (#173)

• Updating VT TI Provider for better severity accuracy with additional "detected_" types.

• Changing ti_browser to display "information" results if no "warning" or "high" items in dataset.

• Updated compound_ctrls.py and mp_config_file.py to work when azure-keyvault-secrets not installed

• Update to QueryTime to fix default max time when "day" is used for units.

e6ea232- Pebryan/2021 6 7 grey noise updates (#171)

• Update GreyNoise naming and added custom UA

4c4b8ca- Ianhelle/getting started fixes 2021 06 04 (#170)

• Fixing a query error in kql_sent_azuresentinel.yaml
• Fixing an issue with contain naming when creating pivot functions
• Fixing link in TIProviders.rst
• Fixing mypy error in azure_auth_core
• Adding check for KV settings before trying to initialize secrets client in provider_settings.py
• Adding function to search for file in utility.py
• mp_config_edit.py:
  • Renaming "Save File" button to "Save Settings"
  • Add config_filepath parameter to specify file to save when settings not imported from a file
  • Changing item update button to "Update" rather than Save in comp_edit.py
  • Strip extraneous quotes from string "default" value in mp_config_control.py
  • Corrected open params to "w" rather than "w+" (write append)
• Added "browse" as alias for browse_queries in data_providers.py.
• Fixed misreporting of DataFrame column name in exception when column parameter does not match a column in the input data. in pivot_register.py
• Adding azure_ml_tools.py (from nb_check.py script)
• Added "severities='all'" parameter option. Added friendly warning (rather than exception) when no data is present in the input (either empty DF or nothing above the default severities - in ti_browser.py
• Fixing additional areas breaking in timeline plot module if you don't supply "source_columns" parameter.
• Fixing mypy warning in domain_utils.py
• Fixed allowing geolite to create a database folder even if parent folders don't exist.
• Adding "col" and "column" aliases for lookup_iocs in tilookup.py
• Changed score logic for IBM XForce - prevent misreporting of score==1 as warning.
• Updating version in _version.py
• Added test_azure_ml_tools.py
• unit_test_lib updates:
  • Added change_directory context manager (with lock) to allow individual test cases to change directory
  • Fixed TEST_DATA_PATH to always return absolute path
• Fixing pylint warnings about the lack of use of context managers in mordor_driver.py, base64unpack.py, morph_charts.py
• Pylint warnings suppression about lowercase enums in tilookup.py
• Fixing version number format problem in azure_ml_tools
• Fixing warning that wasn't processing an f-string in tilookup.py

Pivot functions #3 and Greynoise TI Provider

This minor release includes some major updates to the pivot functions.
It also includes the addition of the Greynoise Community API
to our set of Threat Intel providers. Also

Warning - this includes some potentially breaking changes since we have
renamed a number of pivot functions to have shorter, friendlier names.
In particular a number of Azure Sentinel query functions have shortened names.

Updates

Pivot phase 3 with better joins, more pd accessor functions and short/friendly aliases for
pivot functions (especially queries) #140

• Added short names for many queries
• Adding shortcut functions to entities - these allow you to create links from pivot functions in containers
  e.g. IpAddress.whois() -> IpAddress.util.whois()
  The shortcut methods have the advantage of being easier to see and type in. They
  have the further benefit of working as instance methods. So, given an an instance you
  can do this:

  my_ip = IpAddress(Address="157.23.49.1")
  my_ip.whois()
  # previously you'd have to do IpAddress.util.whois(my_ip) - this still works

• Standard pivot functions now also support joins (previously these were implemented only for Data query and TI functions)
• joins now support join_ignore_case parameter to join case-insensitively
• Added PD accessors for:
  • filter_cols - column filtering with patterns or lists of columns
  • filter - string or regex will match against all columns (can also match on partial numbers with the numeric_col=True option)
  • sort - sort on one or more columns or a pattern to match columns
  • list_to_rows - expand embedded lists in one or more cols to rows
  • parse_json - turn JSON strings in column into Python objects
• Updates to Pivot docs and notebooks for new pivot behavior
• Added remove_pivot_funcs method to pivot.pivot.py + unit test in test_pivot

Implemented Greynoise API as TI Provider

• Greynoise Community API
• TI Docs

Entities

• Added additional entities from schema: iot_device, mail_cluster, mail_message, mailbox, submission_mail
• Adding str and repr to Edge in entity_graph.py
• Adding (not yet used) graph_property.py to handle automatic creation of edges

Other updates

• Added tilookup functions to allow enabling/disabling individual providers
• Add better timestamp conversion in vtlookup3.py.
• Added check for attribute name similarity in query_container - if trying to access an attribute that doesn't exist it will return likely matches in the exception
• Extracted dup code from pivot_data_queries to pivot_register.
• Updating blog articles links in docs with recent publications
• Capturing output from load_user_defaults in nbinit.py to display as HTML
• Adding find_anomaly_periods function and shorter alias for timeseries_anomalies_stl in timeseries.py

Fixes

• Fixed a problem with logon failure query parameters.
• Minor corrections to queries in kql_sent_az_network and kql_sent_azure
• Fixing path handling in file.py entity
• Fixing test in test_pivot_pd_accessor for renamed parameter.
• Updating notebooks for some pivot function renaming.
• Fix spelling errors in SettingsEditor.rst
• Remove unnecessary list comprehension in test_security_alert.py
• Add tests for list_to_rows and parse_json in pivot_pd_accessor.py - test_pivot_pd_accessor.py
• Fixed some errors in entities with description_str and identity fields
• Add missing FullName property to Host
• Fixed exception in IpAddress
• Fixing some test errors in test_mordor_driver (due to test file race conditions)
• Removing duplicate mp_config file
• Correcting problem with passing namespace argument to notebooklets in user_config.py
• Fixes to analyze_imports.py and import_analyzer.py that prevented use from commandline and use with other packages.
• Fix to list_to_rows pd accessor in pivot_pd_accessor.
• Changed timeline.py so that it doesn't error when no source_columns are supplied
• Fixing bug in calculating period in timespan.py
• Cosmetic fixes in eventcluster.py
• Fixing an issue in ti_browser if passed an empty dataframe.
• Updating requirements files for updated versions of lxml and aiohttp for security issues.

Bug fix for pip install --no-binary and Mitre downloads

Fixes

• Using pip install --no-binary fails because requirements.txt, requirements-dev.txt and Readme.md are not included in the package.
  These are referenced by setup.py when using the --no-binary option
• Changed download of Mitre techniques and tactices from web scraping to official JSON. Fixes error in Mordor driver