Merge branch 'develop'

Author: isaisabel

CHANGELOG.md

@@ -1,26 +1,36 @@
-# V1.4.1 - 18 May 2020
+# v1.5 - 8 July 2020
+## New Scripts
+Added scripts used to generate the [sample layers in the ATT&CK Navigator repository](https://github.com/mitre-attack/attack-navigator/tree/develop/layers/data/samples). See issue [#21](https://github.com/mitre-attack/attack-scripts/issues/21) and [the sample layer README](scripts/layers/samples/README.md) for more details. The following scripts were added:
+- [heatmap.py](scripts/layers/samples/heatmap.py)
+- [bear_APT.py](scripts/layers/samples/bear_APT.py)
+- [apt3_apt29_software.py](scripts/layers/samples/apt3_apt29_software.py)
+- [software_execution.py](scripts/layers/samples/software_execution.py)
+## Fixes
+- Fixed a bug in diff_stix where sub-techniques had the wrong URL in hyperlinks.
+
+# v1.4.1 - 18 May 2020
 ## New Scripts
 - New script [technique_mappings_to_csv.py](technique_mappings_to_csv.py) added to support mapping Techniques with Mitigations, Groups or Software. The output is a CSV file. Added in PR [#23](https://github.com/mitre-attack/attack-scripts/pull/23)
 ## Improvements
 - Updated [diff_stix.py](scripts/diff_stix.py) with sub-techniques support. See issue [#12](https://github.com/mitre-attack/attack-scripts/issues/12).
 ## Fixes
 - Fixed bug in LayerOps causing issues with cross-tactic techniques, as well as a bug where a score lambda could affect the outcome of other lambdas.
 
-# V1.4 - 5 May 2020
+# v1.4 - 5 May 2020
 ## New Scripts
 - Added Layers folder with utility scripts for working with [ATT&CK Navigator](https://github.com/mitre-attack/attack-navigator) Layers. See the Layers [README](layers/README.md) for more details. See issues [#2](https://github.com/mitre-attack/attack-scripts/issues/2) and [#3](https://github.com/mitre-attack/attack-scripts/issues/3).
 
-# V1.3 - 8 January 2019
+# v1.3 - 8 January 2019
 ## New Scripts
 - Added [diff_stix.py](scripts/diff_stix.py).
 
-# V1.2 - 24 October 2019
+# v1.2 - 24 October 2019
 - Added ATT&CKcon 2.0 Detection Training. See [the readme](/trainings/detection-training/README.md) for details.
 
-# V1.1 - 29 March 2019
+# v1.1 - 29 March 2019
 ## New Scripts
 - Added [techniques_from_data_source.py](scripts/techniques_from_data_source.py).
 
-# V1.0 - 1 March 2019
+# v1.0 - 1 March 2019
 ## New Scripts
 - Added [techniques_data_sources_vis.py](scripts/techniques_data_sources_vis.py).

scripts/diff_stix.py

@@ -139,13 +139,14 @@ def verboseprint(self, *args, **kwargs):
             print(*args, **kwargs)
 
 
-    def getUrlFromStix(self, datum):
+    def getUrlFromStix(self, datum, is_subtechnique=False):
         """
         Parse the website url from a stix object.
         """
         url = datum['external_references'][0]['url']
         split_url = url.split('/')
-        link = '/'.join(split_url[-2:])
+        splitfrom = -3 if is_subtechnique else -2
+        link = '/'.join(split_url[splitfrom:])
         return link
 
 
@@ -400,13 +401,14 @@ def placard(item):
                         # get revoking technique's parent for display
                         parentID = list(filter(lambda rel: rel["source_ref"] == revoker["id"], subtechnique_of_rels))[0]["target_ref"]
                         parentName = id_to_technique[parentID]["name"] if parentID in id_to_technique else "ERROR NO PARENT"
-                        return f"{item['name']} (revoked by { parentName}: [{revoker['name']}]({self.site_prefix}/{self.getUrlFromStix(revoker)}))"
+                        return f"{item['name']} (revoked by { parentName}: [{revoker['name']}]({self.site_prefix}/{self.getUrlFromStix(revoker, True)}))"
                     else:
                         return f"{item['name']} (revoked by [{revoker['name']}]({self.site_prefix}/{self.getUrlFromStix(revoker)}))"
                 if section == "deletions":
                     return f"{item['name']}"
                 else:
-                    return f"[{item['name']}]({self.site_prefix}/{self.getUrlFromStix(item)})"
+                    is_subtechnique = item["type"] == "attack-pattern" and "x_mitre_is_subtechnique" in item and item["x_mitre_is_subtechnique"]
+                    return f"[{item['name']}]({self.site_prefix}/{self.getUrlFromStix(item, is_subtechnique)})"
 
 
             # build sectionList string
@@ -688,4 +690,4 @@ def verboseprint(*args, **kwargs):
             parser.error('-layers requires exactly three files to be specified or none at all')
 
         layers_dict = diffStix.get_layers_dict()
-        layers_dict_to_files(args.layers, layers_dict)
+        layers_dict_to_files(args.layers, layers_dict)

scripts/layers/samples/README.md

@@ -0,0 +1,10 @@
+# Sample Scripts
+
+The scripts in this folder are used to generate the [sample layers in the ATT&CK Navigator repository](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples). Run the scripts with the -h flag for usage instructions.
+
+| script | sample layers | description |
+|:-------|:------------|:--------|
+| [heatmap.py](heatmap.py) | [heatmap_layer.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/heatmap_layer.json) | Generates a layer wherein all techniques have randomized scores from 1-100. |
+| [bear_APT.py](bear_APT.py) | [Bear_APT.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/Bear_APT.json) | Parses STIX data to create a layer showing all techniques used by an APT group with phrase 'bear' in the group aliases. |
+| [apt3_apt29_software.py](apt3_apt29_software.py) | [APT3_+_APT29_with_software.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/APT3_+_APT29_with_software.json), [APT3_+_APT29_with_software_and_notional_no_detection.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/APT3_+_APT29_with_software_and_notional_no_detection.json) | Creates a layer file showing techniques used by APT3 and APT29 as well as software used by those groups, and a second layer showing the same but with the added concept of detectability by a notional organization. |
+| [software_execution.py](software_execution.py) | [software_execution.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/software_execution.json), [software_malware_execution.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/software_malware_execution.json), [software_tool_execution.json](https://github.com/mitre-attack/attack-navigator/tree/master/layers/data/samples/software_tool_execution.json) | Generates layers showing all techniques that can be executed by software. |

scripts/layers/samples/apt3_apt29_software.py

@@ -0,0 +1,151 @@
+import argparse
+import requests
+import json
+import stix2
+
+def generate(show_nodetect=False):
+    """
+        generate and return a layer dict showing techniques used by APT3 and APT29 as well as software used by those groups
+        param show_nodetect, if true, causes techniques that have no data-sources to be highlighted as well
+    """
+    stix = requests.get("https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json").json()
+    ms = stix2.MemoryStore(stix_data=stix["objects"])
+    apt3 = ms.get("intrusion-set--0bbdf25b-30ff-4894-a1cd-49260d0dd2d9")
+    apt29 = ms.get("intrusion-set--899ce53f-13a0-479b-a0e4-67d46e241542")
+    
+    techniques_used = {} # attackID => {apt3: boolean, apt29: boolean, software: Set, detection: boolean}
+
+    for apt in [apt3, apt29]:
+
+        def use_technique(technique, software=None):
+            """helper function to record a technique as used"""
+            techniqueID = technique["external_references"][0]["external_id"]
+            # init struct if the technique has not been seen before
+            if not techniqueID in techniques_used:
+                techniques_used[techniqueID] = {
+                    "APT3": False,
+                    "APT29": False,
+                    "software": set(),
+                    "datasources": []
+                }
+            # record new data
+            techniques_used[techniqueID][apt["name"]] = True
+            if "x_mitre_data_sources" in technique and len(technique["x_mitre_data_sources"]) > 0: 
+                techniques_used[techniqueID]["datasources"] = technique["x_mitre_data_sources"]
+            if software:
+                techniques_used[techniqueID]["software"].add(software["name"])
+        
+        # traverse relationships
+        for relationship in ms.relationships(apt["id"]):
+            target_obj = ms.get(relationship["target_ref"])
+            # skip relationships with deprecated objects
+            if ("x_mitre_deprecated" in target_obj and target_obj["x_mitre_deprecated"]) or ("revoked" in target_obj and target_obj["revoked"]): continue
+            # technique type relationship
+            if target_obj["type"] == "attack-pattern": 
+                # record technique usage
+                use_technique(target_obj)
+            # software type relationship, traverse to find software-used techniques
+            if target_obj["type"] == "malware" or target_obj["type"] == "tool":
+                software = target_obj
+                for software_relationship in ms.relationships(software["id"]):
+                    software_target_obj = ms.get(software_relationship["target_ref"])
+                    # skip relationships with deprecated objects
+                    if ("x_mitre_deprecated" in software_target_obj and software_target_obj["x_mitre_deprecated"]) or ("revoked" in software_target_obj and software_target_obj["revoked"]): continue
+                    if software_target_obj["type"] == "attack-pattern": 
+                        # record technique usage
+                        use_technique(software_target_obj, software)
+
+    # format the techniques for the output layer
+    techniques_list = []
+
+    def color_lookup(usage): 
+        if show_nodetect and not len(usage["datasources"]) > 0:
+            return "#fc3b3b"
+        if usage["APT3"] and usage["APT29"]: 
+            return "#74c476"
+        if usage["APT3"]: return "#6baed6"
+        if usage["APT29"]: return "#fce93b"
+
+    for techniqueID in techniques_used:
+        # determine the number of used techniques for the score
+        comment = ""
+        if show_nodetect:
+            if len(techniques_used[techniqueID]["datasources"]) > 0:
+                comment = f"considered detectable by a notional organization because it has data-sources {', '.join(techniques_used[techniqueID]['datasources'])}"
+            else:
+                comment = "considered undetectable by a notional organization because it has no data-sources"
+        else:
+            used = []
+            if techniques_used[techniqueID]["APT3"]: used.append("APT3") 
+            if techniques_used[techniqueID]["APT29"]: used.append("APT29") 
+            used += list(techniques_used[techniqueID]["software"])
+            comment = f"used by {', '.join(used)}"
+        # append technique struct to list of layer-formatted techniques
+        techniques_list.append({
+            "techniqueID": techniqueID,
+            "color": color_lookup(techniques_used[techniqueID]),
+            "comment": comment,
+        })
+
+    # construct and return the layer as a dict
+    # set up layer information according to show_nodetect
+    name = "APT3 + APT29 with software"
+    description = "This layer shows techniques (including techniques from software used by the groups) used by APT3 only in blue, APT29 only in yellow, and both APT3 and APT29 in green."
+    legend = [
+        {
+            "label": "Used by APT3 or a software APT3 uses",
+            "color": color_lookup({"APT3": True, "APT29": False, "datasources": ["placeholder"]})
+        },
+        {
+            "label": "Used by APT29 or a software APT29 uses",
+            "color": color_lookup({"APT3": False, "APT29": True, "datasources": ["placeholder"]})
+        },
+        {
+            "label": "Used by both APT3 or a softare APT3 uses and APT29 or a software APT29 uses",
+            "color": color_lookup({"APT3": True, "APT29": True, "datasources": ["placeholder"]})
+        }
+    ]
+    # additional formatting when displaying notional detectability
+    if show_nodetect: 
+        name += " and notional no detection"
+        description += " The techniques in red denote techniques considered undetectable by a notional organization because they have no data-sources. Disclaimer: Data-sources in ATT&CK are sources of information that COULD be used to identify adversary actions, however the exactness of that evidence varies greatly. Therefore the presence of a data source for technique should only be considered a potential metric for detectability."
+        legend.append({
+            "label": "Used by either APT3 or APT29 but considered undetectable by a notional organization because it has no data-sources",
+            "color": color_lookup({"APT3": True, "APT29": True, "datasources": []})
+        })
+        
+    # layer struct
+    return {
+        "name": name,
+        "version": "3.0",
+        "description": description,
+        "domain": "mitre-enterprise",
+        "techniques": techniques_list,
+        "legendItems": legend
+    }
+        
+
+if __name__ == '__main__':
+    # download data depending on domain
+    parser = argparse.ArgumentParser(
+        description="Creates a layer file showing techniques used by APT3 and APT29 as well as software used by those groups, and a second layer showing the same but with the added concept of detectability by a notional organization."
+    )
+    parser.add_argument("--output",
+        type=str,
+        default="APT3_+_APT29_with_software.json",
+        help="output filepath for layer showing APT3 and APT29"
+    )
+    parser.add_argument("--output-notional",
+        type=str,
+        dest="notional",
+        default="APT3_+_APT29_with_software_and_notional_no_detection.json",
+        help="output filepath for layer showing APT3 and APT29 with detectability by a notional organization"
+    )
+    args = parser.parse_args()
+    for arg in ["output", "notional"]:
+        # get the layer
+        layer = generate(arg == "notional")
+        # write the layerfile
+        with open(vars(args)[arg], "w") as f:
+            print("writing", vars(args)[arg])
+            f.write(json.dumps(layer, indent=4))

scripts/layers/samples/bear_APT.py

@@ -0,0 +1,91 @@
+import argparse
+import requests
+import json
+import stix2
+import re
+
+def generate():
+    """parse the STIX on MITRE/CTI and return a layer dict showing all techniques used by an APT group with phrase 'bear' in the group aliases."""
+    # import the STIX data from MITRE/CTI
+    stix = requests.get("https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json").json()
+    ms = stix2.MemoryStore(stix_data=stix["objects"])
+
+    groups = ms.query([ stix2.Filter("type", "=", "intrusion-set") ])
+
+    # find bear groups
+    bear_groups = [] #list of groups with bear in name
+    for group in groups:
+        # filter out deprecated and revoked groups
+        if ("x_mitre_deprecated" in group and group["x_mitre_deprecated"]) or ("revoked" in group and group["revoked"]): continue
+        # check all aliases for bear
+        for alias in group["aliases"]:
+            if re.match(".*bear.*", alias, re.IGNORECASE) is not None:
+                bear_groups.append(group)
+                break # don't match the same group multiple times
+
+    # find techniques used by bear groups
+    techniques_used = {} #attackID => using bear groups
+    for bear in bear_groups:
+        # construct the "bear" name for the comment
+        # if bear occurs in multiple aliases, list them all
+        bearnames = []
+        for alias in bear["aliases"]:
+            if re.match(".*bear.*", alias, re.IGNORECASE) is not None:
+                bearnames.append(alias)
+        bearname = bearnames[0]
+        if len(bearnames) > 1: 
+            bearname += " (AKA " + ",".join(bearnames[1:]) + ")"
+
+        # get techniques used by this group
+        relationships = ms.relationships(bear["id"])
+        for relationship in relationships:
+            # skip all non-technique relationships
+            if "attack-pattern" not in relationship["target_ref"]: continue
+            technique = ms.get(relationship["target_ref"])
+            # filter out deprecated and revoked techniques
+            if ("x_mitre_deprecated" in technique and technique["x_mitre_deprecated"]) or ("revoked" in technique and technique["revoked"]): continue
+            techniqueID = technique["external_references"][0]["external_id"]
+            # store usage in techniques_used struct
+            if techniqueID in techniques_used:
+                techniques_used[techniqueID].append(bearname)
+            else:
+                techniques_used[techniqueID] = [bearname]
+
+    # format the techniques for the output layer
+    techniques_list = []
+    for techniqueID in techniques_used:
+        techniques_list.append({
+            "techniqueID": techniqueID,
+            "comment": "used by " + ", ".join(techniques_used[techniqueID]),
+            "color": "#ff6666"
+        })
+    # construct and return the layer as a dict
+    return {
+        "name": "*Bear APTs",
+        "version": "3.0",
+        "description": "All techniques used by an APT group with phrase 'bear' in the group aliases",
+        "domain": "mitre-enterprise",
+        "techniques": techniques_list,
+        "legendItems": [{
+            "label": "Used by a group the phrase 'bear' in the group aliases",
+            "color": "#ff6666"
+        }]
+    }
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(
+        description="Parses STIX data to create a layer showing all techniques used by an APT group with phrase 'bear' in the group aliases."
+    )
+    parser.add_argument("--output",
+        type=str,
+        default="Bear_APT.json",
+        help="output filepath"
+    )
+    args = parser.parse_args()
+    # get the layer
+    layer = generate()
+    # write the layerfile
+    with open(args.output, "w") as f:
+        print("writing", args.output)
+        f.write(json.dumps(layer, indent=4))

scripts/layers/samples/heatmap.py

@@ -0,0 +1,53 @@
+import argparse
+import requests
+import json
+import stix2
+import random
+
+def generate():
+    """parse the STIX on MITRE/CTI and return a layer dict with techniques with randomized scores"""
+    # import the STIX data from MITRE/CTI
+    stix = requests.get("https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json").json()
+    ms = stix2.MemoryStore(stix_data=stix["objects"])
+    # get all techniques in STIX
+    techniques = ms.query([
+        stix2.Filter("type", "=", "attack-pattern")
+    ])
+    # parse techniques into layer format
+    techniques_list = []
+    for technique in techniques:
+        # skip deprecated and revoked
+        if ("x_mitre_deprecated" in technique and technique["x_mitre_deprecated"]) or ("revoked" in technique and technique["revoked"]): continue
+        techniqueID = technique["external_references"][0]["external_id"] # get the attackID
+        techniques_list.append({
+            "techniqueID": techniqueID,
+            "score": random.randint(1,100) # random score
+        })
+    # return the techniques in a layer dict
+    return {
+        "name": "heatmap example",
+        "version": "3.0",
+        "sorting": 3, # descending order of score
+        "description": "An example layer where all techniques have a randomized score",
+        "domain": "mitre-enterprise",
+        "techniques": techniques_list,
+    }
+
+
+if __name__ == '__main__':
+    # download data depending on domain
+    parser = argparse.ArgumentParser(
+        description="Generates a layer wherein all techniques have randomized scores from 1-100."
+    )
+    parser.add_argument("--output",
+        type=str,
+        default="heatmap_layer.json",
+        help="output filepath"
+    )
+    args = parser.parse_args()
+    # get the layer
+    layer = generate()
+    # write the layerfile
+    with open(args.output, "w") as f:
+        print("writing", args.output)
+        f.write(json.dumps(layer, indent=4))