How to interact with the amount of data ?

[morzen]

Hi,

I am a phd student. I read what I could on how to use CVEs. I went through your schema (ngl the structure is rather complex), but the main issue is how to interact with the CVEs data you have here.

What do you recommend for interacting easily and ergonomically and calling/getting/extracting data? most likely in bulk.

At the very least, I should be able to link the CWEs with the CVEs according to what MITRE says.

From there, I should also be able to link CVEs with IOCs and exploits, correct? As I understand it, you do provide this data as well?

In the schema, I did not see any indication of the CVEs being associated with APTs if they were used by them. Is it not something you do?

Kind regards,

[M-nj]

Including a response to CVEProject/cve-schema#395

You may want to check out https://github.com/CVEProject/cve-services. It uses a mongoDB (docker container) under the hood.

The simple but slow option (optimizing for ease of use):

• git clone this cvelistV5 repo
• use custom scripts, or tools like grep search the cve contents for data that you are looking for.
• occasionally git pull for latest data.

Debatably faster version (optimizing for search runtime):

• setup a local instance of a NoSQL database (such as mongoDB)
• git clone this cvelistV5 repo
• create a script to export the CVE data to the database (For example, if you are using a mongoDB you might want something like this: https://www.mongodb.com/resources/languages/json-to-mongodb for some direction)
• occasionally git pull for latest data to then import the latest changes into your database (also see the delta log files for recent changes)

Your dev script might look like:

import pymongo # pip install pymongo
import json
import os
from pymongo import MongoClient, InsertOne

CONNECTION_STRING = "http://localhost:27017" # You will need to update this for your machine
PATH_TO_CVE_LIST_V5_REPO = "./cves" # you will need to update this for your machine

client = pymongo.MongoClient()
# you may need to create the database and collection if you dont have them already
db = client.cve_dev2
collection = db.Cves2
requesting = []
results = []
for root, dirs, files in os.walk(PATH_TO_CVE_LIST_V5_REPO):
    for file in files:
        if file.startswith("CVE-"):
            with open(os.path.join(root, file), "r", encoding="utf-8") as f:
                record = json.load(f)
                requesting.append(InsertOne(record))
            if len(requesting) >= 1000:
                results.append(collection.bulk_write(requesting))
                requesting = []
                break
    break
results.append(collection.bulk_write(requesting))
client.close()
print(results)



###### FOR SEARCHING SOMETHING LIKE CWEs:
CWE_TO_FIND = "CWE-200"
CWE_FILTER = {
    "$or": [
        {
            "containers.adp.problemTypes.descriptions.cweId": {
                "$regex": f".*\\b{CWE_TO_FIND}\\.*"
            }
        }, 
        {
            "containers.cna.problemTypes.descriptions.cweId": {
                "$regex": f".*\\b{CWE_TO_FIND}\\.*"
            }
        },
        {
            "containers.adp.problemTypes.descriptions.description": {
                "$regex": f".*\\b{CWE_TO_FIND}\\.*"
            }
        },
        {
            "containers.cna.problemTypes.descriptions.description": {
                "$regex": f".*\\b{CWE_TO_FIND}\\.*"
            }
        }

    ]
}

search_results = collection.find(CWE_FILTER)
for record in search_results:
    cna_block = [d for c in record['containers']['cna']['problemTypes'] for d in c.get('descriptions', [])]
    adp_blocks = [d for c in record['containers'].get('adp',[]) for pt in c.get('problemTypes', []) for d in pt.get('descriptions', {})]
    results = [(x.get('cweId',''),x.get('description','')) for x in (cna_block + adp_blocks) if x]
    print(record['cveMetadata']['cveId'], results)

[zmanion]

At the very least, I should be able to link the CWEs with the CVEs

Some CVE Records have CWE information and both Vulnrichment and NVD add CWE information.

Fewer CVE Records have CPE, NVD adds CPE.

CAPEC and ATT&CK are rarely used in the CVE ecosystem, although I've heard of efforts to integrate vulnerabilities (CVE) with ATT&CK.