Merge pull request #997 from nexB/nvd-improvements

Improve NVD handling and more

This PR

-    improves how we handle NVD data
-    refactors the purl2cpe script
-    aligns some key internal names with UI and API (affected and fixed)
-    uses querysets as model managers and streamline views

Signed-off-by: Philippe Ombredanne <[email protected]>

Author: pombredanne

CHANGELOG.rst

@@ -3,14 +3,22 @@ Release notes
 
 
 
-Version v30.2.2
+Version v30.3.0
 ----------------
 
 - We enabled API throttling for a basic user and for a staff user 
   they can have unlimited access on API.
 
 - We added throttle rate for each API endpoint and it can be 
-  configured from the settings #991 https://github.com/nexB/vulnerablecode/issues/991.
+  configured from the settings #991 https://github.com/nexB/vulnerablecode/issues/991
+
+- We improved how we import NVD data
+- We refactored and made the purl2cpe script work to dump purl to CPE mappings
+
+Internally:
+
+- We aligned key names internally with the names used in the UI and API (such as affected and fixed)
+- We now use querysets as model managers and have streamlined view code
 
 
 Version v30.2.1

docs/source/command-line-interface.rst

@@ -3,7 +3,7 @@
 Command Line Interface
 ======================
 
-The main entry point is Django's :guilabel:`manage.py` management commands.
+The main entry point is the Django :guilabel:`manage.py` management command script.
 
 ``$ ./manage.py --help``
 ------------------------
@@ -14,9 +14,10 @@ VulnerableCode's own commands are listed under the ``[vulnerabilities]`` section
     $ ./manage.py --help
     ...
     [vulnerabilities]
-        create_cpe_to_purl_map
-        importer
-        improver
+        import
+        improve
+        purl2cpe
+
 
 ``$ ./manage.py <subcommand> --help``
 ---------------------------------------
@@ -58,3 +59,17 @@ Other variations:
 
 * ``--list`` List all available improvers
 * ``--all`` Run all available improvers
+
+
+
+``$ ./manage.py purl2cpe --destination <directory``
+------------------------------------------
+
+Dump a mapping of CPEs to PURLs grouped by vulnerability in the ``destination``
+directory.
+
+
+Other variations:
+
+* ``--limit`` Limit the number of processed vulnerabilities
+

pyproject.toml

@@ -66,7 +66,7 @@ addopts = [
     "--ignore=vulnerabilities/importers/mozilla.py",
     "--ignore=vulnerabilities/importers/mattermost.py",
     "--ignore=vulnerabilities/importers/xen.py",
-    "--ignore=vulnerabilities/management/commands/create_cpe_to_purl_map.py",
+    "--ignore=vulnerabilities/management/commands/purl2cpe.py",
     "--ignore=vulnerabilities/lib_oval.py",
 ]
 

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = vulnerablecode
-version = 30.2.1
+version = 30.3.0
 license = Apache-2.0 AND CC-BY-SA-4.0
 
 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390

vulnerabilities/api.py

@@ -94,7 +94,7 @@ class VulnerabilitySerializer(serializers.HyperlinkedModelSerializer):
     fixed_packages = MinimalPackageSerializer(
         many=True, source="filtered_fixed_packages", read_only=True
     )
-    affected_packages = MinimalPackageSerializer(many=True, source="vulnerable_to", read_only=True)
+    affected_packages = MinimalPackageSerializer(many=True, read_only=True)
 
     references = VulnerabilityReferenceSerializer(many=True, source="vulnerabilityreference_set")
     aliases = AliasSerializer(many=True, source="alias")