Merge branch 'develop'

# Conflicts:
#	setup.py

Author: Caleb R Little

CHANGELOG.md

@@ -1,3 +1,14 @@
+# Staged in Develop
+## Fixes
+- Updated stix2 and taxii2-client module version requirements to avoid potential bug 
+## Improvements
+- Created Collections module
+- Added method and cli to turn a collection index into a markdown file for human readability
+- Added method and cli to turn a collection into a collection index for summary purposes
+- Added method and cli to turn raw stix data into a collection
+- Added method and cli to allow for bulk layer generation (expands generator module)
+- Added Data Sources and Data Components support to attackToExcel
+
 # v1.3.0 - 8/20/2021
 This release introduces generator functionality to the library, as well as some improvements to excel matrix generation 
 through attackToExcel.

README.md

@@ -3,6 +3,7 @@
 This repository contains a library of Python-based tools and utilities for working with ATT&CK content.
 - the [navlayers](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/navlayers) module contains a collection of utilities for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers.
 - the [attackToExcel](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/attackToExcel) module provides utilities for converting [ATT&CK STIX data](https://github.com/mitre/cti) to Excel spreadsheets. It also provides access to [Pandas](https://pandas.pydata.org/) DataFrames representing the dataset for use in data analysis.
+- the [collections](https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack/collections) module contains a set of utilities for working with [ATT&CK Collections and Collection Indexes](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md).
 
 ## Requirements
 - [python3](https://www.python.org/)
@@ -20,6 +21,7 @@ Some simple examples are provided here to get you started on using this library.
 |:------------|:------------|:--------------|
 | navlayers | Provides a means by which to import, export, and manipulate [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) layers. These layers can be read in from the filesystem or python dictionaries, combined and edited, and then exported to excel or SVG images as users desire. | Further documentation for the navlayers module can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/navlayers/README.md).|
 | attackToExcel | Provides functionalities for exporting the ATT&CK dataset into Excel Spreadsheets. It also provides programmatic access to the dataset as [Pandas](https://pandas.pydata.org/) DataFrames to enable data analysis using that library. | Further documentation for the attackToExcel module can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/attackToExcel/README.md).|
+| collections | Provides functionalities for converting and summarizing data in collections and collection indexes. It also provides a means by which to generate a collection from a raw stix bundle input. | Further documentation for the collections module can be found [here](https://github.com/mitre-attack/mitreattack-python/blob/master/mitreattack/collections/README.md).| 
 ### Usage Examples
 #### navlayers
 ```python
@@ -145,8 +147,9 @@ mapping to all associated groups, software, or mitigations across the techniques
 ```
 C:\Users\attack>layerGenerator_cli -h
 usage: layerGenerator_cli.py [-h]
-                             (--overview-type {group,software,mitigation} | --mapped-to MAPPED_TO)
-                             [-o OUTPUT] [--domain {enterprise,mobile,ics}]
+                             (--overview-type {group,software,mitigation} | --mapped-to MAPPED_TO | --mass-type {group,software,mitigation})
+                             [-p PATH] [-o OUTPUT]
+                             [--domain {enterprise,mobile,ics}]
                              [--source {taxii,local}] [--local LOCAL]
 
 Generate an ATT&CK Navigator layer
@@ -160,17 +163,97 @@ optional arguments:
                         Output techniques mapped to the given group, software,
                         or mitigation. Argument can be name, associated
                         group/software, or ATT&CK ID.
+  --mass-type {group,software,mitigation}
+                        Output a collection of matrices to the specified
+                        folder, each one representing a different instance of
+                        the target type.
+  -p PATH, --path PATH  Path to the output layer directory (mass-type)
   -o OUTPUT, --output OUTPUT
-                        Path to the output layer file
+                        Path to the output layer file (output)
   --domain {enterprise,mobile,ics}
                         Which domain to build off of
   --source {taxii,local}
                         What source to utilize when building the matrix
   --local LOCAL         Path to the local resource if --source=local
   
 C:\Users\attack>layerGenerator_cli --domain enterprise --source taxii --mapped-to S0065 --output generated_layer.json
+C:\Users\attack>layerGenerator_cli --domain mobile --source taxii --overview-type mitigation --output generated_layer2.json
+C:\Users\attack>layerGenerator_cli --domain ics --source taxii --mass-type software
 ```
 
+##### IndexToMarkdown_cli
+This command line tool allows users to transform a
+[ATT&CK collection index file](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) 
+into a [human-readable markdown file](https://github.com/mitre-attack/attack-stix-data/blob/master/index.md) that 
+documents the contents of said collections.
+```commandline
+C:\Users\attack>indexToMarkdown_cli -h
+usage: index_to_markdown.py [-h] [-index INDEX] [-output OUTPUT]
+
+Print a markdown string to std-out representing a collection index
+
+optional arguments:
+  -h, --help      show this help message and exit
+  -i INDEX, --index INDEX    the collection index file to convert to markdown
+  -o output, --output OUTPUT  markdown output file
+C:\Users\attack>indexToMarkdown_cli --index C:\Users\attack\examples\index.json --output example.md
+```
+##### CollectionToIndex_cli
+This command line tool allows users to transform [ATT&CK collections](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) 
+into an [ATT&CK collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collection-indexes) 
+that summarizes the contents of the linked collections.
+```commandline
+C:\Users\attack>collectionToIndex_cli -h
+usage: collection_to_index.py [-h] [-output OUTPUT]
+                              (-files collection1 [collection2 ...] | -folders FOLDERS [FOLDERS ...])
+                              name description root_url
+
+Create a collection index from a set of collections
+
+positional arguments:
+  name                  name of the collection index. If omitted a placeholder
+                        will be used
+  description           description of the collection index. If omitted a
+                        placeholder will be used
+  root_url              the root URL where the collections can be found.
+                        Specified collection paths will be appended to this
+                        for the collection URL
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -output OUTPUT        filename for the output collection index file
+  -files collection1 [collection2 ...]
+                        list of collections to include in the index
+  -folders FOLDERS [FOLDERS ...]
+                        folder of JSON files to treat as collections
+C:\Users\attack>collectionToIndex_cli test_index "a layer created as a demo" www.example.com -files C:\Users\attack\examples\collection.json -output C:\Users\attack\examples\index.json
+```
+##### StixToCollection_cli
+This command line tool allows users to transform raw stix bundle files into versions featuring [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md#collections) objects.
+It is compatible with both STIX 2.0 and STIX 2.1 bundles.
+```commandline
+C:\Users\attack>stixToCollection_cli -h
+usage: stix_to_collection.py [-h] [-input INPUT] [-output OUTPUT]
+                             [-description DESCRIPTION]
+                             name version
+
+Update a STIX 2.0 or 2.1 bundle to include a collection object referencing the
+contents of the bundle.
+
+positional arguments:
+  name                  the name for the generated collection object
+  version               the ATT&CK version for the generated collection object
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -input INPUT          the input bundle file
+  -output OUTPUT        the output bundle file
+  -description DESCRIPTION
+                        description to use for the generated collection
+
+C:\Users\attack>stixToCollection "2.0 demo bundle" 9.1 -input C:\Users\bundles\enterprise-bundle-2_0.json
+C:\Users\attack>stixToCollection "2.1 demo bundle" 9.1 -input C:\Users\bundles\enterprise-bundle-2_1.json
+```
 ## Related MITRE Work
 #### CTI
 [Cyber Threat Intelligence repository](https://github.com/mitre/cti) of the ATT&CK catalog expressed in STIX 2.0 JSON. This repository also contains [our USAGE document](https://github.com/mitre/cti/blob/master/USAGE.md) which includes additional examples of accessing and parsing our dataset in Python.

mitreattack/__init__.py

@@ -1,2 +1,3 @@
 from .attackToExcel import *
 from .navlayers import *
+from .collections import *

mitreattack/attackToExcel/attackToExcel.py

@@ -34,16 +34,19 @@ def build_dataframes(src, domain):
     :param domain: domain of ATT&CK src corresponds to, e.g "enterprise-attack"
     :returns: a dict lookup of each ATT&CK type to dataframes for the given type to be ingested by write_excel
     """
+    df = {
+            "techniques": stixToDf.techniquesToDf(src, domain),
+            "tactics": stixToDf.tacticsToDf(src, domain),
+            "software": stixToDf.softwareToDf(src, domain),
+            "groups": stixToDf.groupsToDf(src, domain),
+            "mitigations": stixToDf.mitigationsToDf(src, domain),
+            "matrices": stixToDf.matricesToDf(src, domain),
+            "relationships": stixToDf.relationshipsToDf(src)
+        }
     # get each ATT&CK type
-    return {
-        "techniques": stixToDf.techniquesToDf(src, domain),
-        "tactics": stixToDf.tacticsToDf(src, domain),
-        "software": stixToDf.softwareToDf(src, domain),
-        "groups": stixToDf.groupsToDf(src, domain),
-        "mitigations": stixToDf.mitigationsToDf(src, domain),
-        "matrices": stixToDf.matricesToDf(src, domain),
-        "relationships": stixToDf.relationshipsToDf(src)
-    }
+    if domain == 'enterprise-attack':
+        df["datasources"] = stixToDf.sourcesToDf(src, domain)
+    return df
 
 
 def write_excel(dataframes, domain, version=None, outputDir="."):

mitreattack/attackToExcel/stixToDf.py

@@ -214,6 +214,62 @@ def tacticsToDf(src, domain):
     return dataframes
 
 
+def sourcesToDf(src, domain):
+    """
+    Parse STIX Data Sources and their Data components from the given data and return corresponding pandas dataframes.
+    :param src: MemoryStore or other stix2 DataSource object holding the domain data
+    :param domain: domain of ATT&CK src corresponds to, e.g "enterprise-attack"
+    :returns: a lookup of labels (descriptors/names) to dataframes
+    """
+    data = list(chain.from_iterable(  # software are the union of the tool and malware types
+        src.query(f) for f in [Filter("type", "=", "x-mitre-data-component"),
+                               Filter("type", "=", "x-mitre-data-source")]
+    ))
+    refined = remove_revoked_deprecated(data)
+    data_object_rows = []
+    source_lookup = dict()
+    for x in refined:
+        if x['type'] == 'x-mitre-data-source':
+            source_lookup[x['id']] = x['name']
+    for data_object in tqdm(refined, desc="parsing data sources"):
+        # add common STIx fields
+        row = parseBaseStix(data_object)
+        # add software-specific fields
+        if "x_mitre_platforms" in data_object:
+            row["platforms"] = ", ".join(sorted(data_object["x_mitre_platforms"]))
+        if "x_mitre_collection_layers" in data_object:
+            row["collection layers"] = ', '.join(sorted(data_object["x_mitre_collection_layers"]))
+        if "x_mitre_aliases" in data_object:
+            row["aliases"] = ", ".join(sorted(data_object["x_mitre_aliases"][1:]))
+        if data_object["type"] == "x-mitre-data-component":
+            row["name"] = f"{data_object['name']}: {source_lookup[data_object['x_mitre_data_source_ref']]}"
+            row["type"] = "datacomponent"
+        else:
+            row["type"] = "datasource"
+        if "description" in data_object:
+            row['description'] = data_object['description']
+        data_object_rows.append(row)
+
+    citations = get_citations(refined)
+    tempa = pd.DataFrame(data_object_rows).sort_values("name")
+    dataframes = {
+        "datasources": tempa.reindex(columns=["name", "ID", "description", "collection layers", "platforms", "created",
+                                               "modified", "type", "version", "url", "contributors"]),
+    }
+    # add relationships
+    dataframes.update(relationshipsToDf(src, relatedType="datasource"))
+    # add/merge citations
+    if not citations.empty:
+        if "citations" in dataframes:  # append to existing citations from references
+            dataframes["citations"] = dataframes["citations"].append(citations)
+        else:  # add citations
+            dataframes["citations"] = citations
+
+        dataframes["citations"].sort_values("reference")
+
+    return dataframes
+
+
 def softwareToDf(src, domain):
     """
     Parse STIX software from the given data and return corresponding pandas dataframes.
@@ -584,6 +640,7 @@ def relationshipsToDf(src, relatedType=None):
         "group": ["intrusion-set"],
         "mitigation": ["course-of-action"],
         "matrix": ["x-mitre-matrix"],
+        "datasource": ["x-mitre-data-component"],
     }
     stixToAttackTerm = {
         "attack-pattern": "technique",
@@ -592,7 +649,9 @@ def relationshipsToDf(src, relatedType=None):
         "malware": "software",
         "intrusion-set": "group",
         "course-of-action": "mitigation",
-        "x-mitre-matrix": "matrix"
+        "x-mitre-matrix": "matrix",
+        "x-mitre-data-component": "datacomponent",
+        "x-mitre-data-source": "datasource"
     }
 
     # get master list of relationships
@@ -625,7 +684,8 @@ def relationshipsToDf(src, relatedType=None):
                         target["type"] == stixTerm):  # if any stix type is part of the relationship
                     related = True
                     break
-            if not related: continue  # skip this relationship if the types don't match
+            if not related:
+                continue  # skip this relationship if the types don't match
 
         # add mapping data
         row = {}