Merge pull request #968 from nexB/release-30.2.0-prep

Prepare release 30.2.0

Author: pombredanne

CHANGELOG.rst

@@ -2,6 +2,36 @@ Release notes
 =============
 
 
+
+Version v30.2.0
+----------------
+
+This is a critical bug fix release including features updates.
+
+- We fixed critical performance issues that made the web UI unusable. This include
+  removing some less interesting redundant details displayed in the web UI for
+  vulnerabilities.
+- We made minor documentation updates.
+- We re-enabled support for Arch linux, Debian, and Ubuntu security advisories importers
+- We added a new improver for Oval data sources
+- We improved Alpine linux and Gitlab security advisories importers 
+
+The summary of performance improvements include these fixes:
+
+- Cascade queries from exact to approximate searches to avoid full table scans
+  in all cases. This is a band-aid for now. The proper solution will likely
+  require using full text search instead.
+- Avoid iceberg queries with "prefetch related" to limit the number of queries
+  that are needed in the UI
+- Do not recreate querysets from scratch but instead allow these to be chained
+  for simpler and correct code.
+- Remove extra details from the vulnerability pacge: each package was further
+  listing its related vulnerabilities creating an iceberg query.
+- Enable the django-debug-toolbar with a setting to easily profile queries on demand
+  by setting both VULNERABLECODE_DEBUG and VULNERABLECODE_DEBUG_TOOLBAR enviroment
+  variables.
+
+
 Version v30.1.1
 ----------------
 

docs/source/contributing.rst

@@ -5,7 +5,7 @@ Contributing to VulnerableCode
 
 Thank you so much for being so interested in contributing to VulnerableCode. We
 are always on the lookout for enthusiastic contributors like you who can make
-our project better, and we're willing to lend a helping hand if you have any
+our project better, and we are willing to lend a helping hand if you have any
 questions or need guidance along the way. That being said, here are a few
 resources to help you get started.
 
@@ -33,13 +33,13 @@ join our community. Below are some examples to get involved:
 First Timers
 ^^^^^^^^^^^^
 
-You are here to help, but you're a new contributor! No worries, we always
+You are here to help, but you are a new contributor! No worries, we always
 welcome newcomer contributors. We maintain some
 `good first issues <https://github.com/nexB/vulnerablecode/labels/good%20first%20issue>`_
 and encourage new contributors to work on those issues for a smooth start.
 
 .. tip::
-    If you're an open-source newbie, make sure to check the extra resources at
+    If you are an open-source newbie, make sure to check the extra resources at
     the bottom of this page to get the hang of the contribution process!
 
 Code Contributions
@@ -75,7 +75,7 @@ Other Ways
 ^^^^^^^^^^
 
 You want to contribute to other aspects of the VulnerableCode project, and you
-can't find what you're looking for! You can always discuss new topics, ask
+cannot find what you are looking for! You can always discuss new topics, ask
 questions, and interact with us and other community members on
 `AboutCode Gitter <https://gitter.im/aboutcode-org/discuss>`_ and `VulnerableCode Gitter <https://gitter.im/aboutcode-org/vulnerablecode>`_
 

docs/source/index.rst

@@ -1,9 +1,18 @@
-VulnerableCode documentation
+Welcome to VulnerableCode!
 =============================
 
-Welcome to VulnerableCode! In this documentation you’ll find information on:
+*VulnerableCode* provides an open database of software packages that are affected
+by known security vulnerabilities aka. *"vulnerable packages"*.
 
-- An overview of VulnerableCode
+VulnerableCode is also a free and open source software (FOSS) project that
+provides the tools to build this open database. The tools handle collecting,
+aggregating and correlating these vulnerabilities and relating them to a correct
+package version. Our project also supports a public cloud instance of this
+database - VulnerableCode.io.
+
+In this documentation you will find information on:
+
+- An overview of VulnerableCode and what you can do with it
 - Installation instructions
 - How to make technical contributions to the project and the community
 
@@ -33,6 +42,7 @@ Welcome to VulnerableCode! In this documentation you’ll find information on:
     reference_improver_overview
     reference_framework_overview
     command-line-interface
+    importers_link
 
 .. toctree::
    :maxdepth: 1

docs/source/introduction.rst

@@ -3,18 +3,61 @@
 VulnerableCode Overview
 ========================
 
-VulnerableCode is a FOSS project that provides tools to build a database
-of software vulnerabilities and the packages they impact. The tools
-handle collecting, aggregating and correlating these vulnerabilities.
-Our project also supports a public Cloud instance of this database –
-VulnerableCode.io.
+*VulnerableCode* provides an open database of software packages that are affected
+by known security vulnerabilities aka. *"vulnerable packages"*.
+
+VulnerableCode is also a free and open source software (FOSS) project that
+provides the tools to build this open database. The tools handle collecting,
+aggregating and correlating these vulnerabilities and relating them to a correct
+package version. Our project also supports a public cloud instance of this
+database - VulnerableCode.io.
+
+
+What can I do with VulnerableCode?
+------------------------------------
+
+**For security researchers and software developers, VulnerableCode offers a web
+UI and a JSON API to efficient find if the FOSS packages and dependencies that
+you use may be affected by known vulnerabilities and which version of a package
+you should upgrade to to fix this issue.**
+
+
+- With the web UI, you can search by package using Package URLs or search by
+  vulnerability like by CVE. From there you can navigate to the package
+  vulnerabilities and to the vulnerable packages.
+
+- With the JSON API, you can perform package queries using Package URLs or query
+  by vulnerability id. You can also query by CPEs and vulnerability aliases.
+  The API provides paginated index and detail endpoints and includes indexes
+  of vulnerable CPEs and vulnerable Package URLs (purl).
+
+You can also install VulnerableCode locally or use the provided publicly hosted instance,
+or host your own installation. You can also contact the VulnerableCode authors and team
+for special needs including commercial support.
+
 
 Why VulnerableCode?
 -------------------
 
-Existing vulnerability database solutions are primarily commercial or
-proprietary which does not make sense because the data is about FOSS
-(Free and Open Source Software).
+VulnerableCode provides open correlated data and eventually does provide curated
+data. Our approach is to privilege upstream data sources and to merge multiple
+vulnerability data sources after comparison and correlation. The vulnerability
+data is keyed by Package URL ensuring quick and accurate lookup with minimal
+friction. We further continuously validate and refine the collected data for
+quality, accuracy and consistency using "improver" jobs.
+An example of such improver can validate that a package version reported as
+vulnerable effectively exists (several do not exist); Or an improver can
+re-evaluate a vulnerable version ranges based on the latest releases of a
+package.
+
+The benefits of our approach is that we will eventually provide better, more
+accurate vulnerability data, more efficiently related to actual packages scanned
+or reported in an SBOM. This should contribute to more efficient vulnerability
+management with less noise from false positives.
+
+Furthermore, existing vulnerability database solutions are primarily commercial
+or proprietary which does not make sense because the bulk of the vulnerability
+data is about FOSS.
 
 The National Vulnerability Database, which is a primary centralized data
 source for known vulnerabilities, is not particularly well suited to

docs/source/tutorial_add_new_importer.rst

@@ -15,7 +15,8 @@ TL;DR
 #. Create a new importer subclass inheriting from the ``Importer`` superclass defined in
    ``vulnerabilites.importer``. It is conventional to end an importer name with *Importer*.
 #. Specify the importer license.
-#. Implement the ``advisory_data`` method to process the data source you're writing an importer for.
+#. Implement the ``advisory_data`` method to process the data source you are
+   writing an importer for.
 #. Add the newly created importer to the importers registry at
    ``vulnerabilites/importers/__init__.py``
 
@@ -136,7 +137,9 @@ version management from `univers <https://github.com/nexB/univers>`_.
 
 .. note::
 
-   It is possible that the versioning scheme you are targetting has not yet been implemented in the `univers <https://github.com/nexB/univers>`_ library. If this is the case, you'll need to head over there and implement one.
+   It is possible that the versioning scheme you are targetting has not yet been
+   implemented in the `univers <https://github.com/nexB/univers>`_ library.
+   If this is the case, you will need to head over there and implement one.
 
 .. code-block:: python
 
@@ -235,12 +238,12 @@ Finally, register your importer in the importer registry at
 
     IMPORTERS_REGISTRY = {x.qualified_name: x for x in IMPORTERS_REGISTRY}
 
-Congratulations! You've written your first importer.
+Congratulations! You have written your first importer.
 
 Run Your First Importer
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If everything went well, you'll see your importer in the list of available importers.
+If everything went well, you will see your importer in the list of available importers.
 
 .. code-block:: console
    :emphasize-lines: 5
@@ -284,7 +287,7 @@ For more visibility, turn on debug logs in :file:`vulnerablecode/settings.py`.
         },
     }
 
-Invoke the import command now and you'll see (in a fresh database):
+Invoke the import command now and you will see (in a fresh database):
 
 .. code-block:: console
 

docs/source/tutorial_add_new_improver.rst

@@ -76,7 +76,7 @@ Explore Package Managers (Optional)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If your Improver depends on the discrete versions of a package, the package managers' VersionAPI
-located at :file:`vulnerabilites/package_managers.py` could come in handy.  You'll need to
+located at :file:`vulnerabilites/package_managers.py` could come in handy.  You will need to
 instantiate the relevant ``VersionAPI`` in the improver's constructor and use it later in the
 implemented methods. See an already implemented improver (NginxBasicImprover) for an example usage.
 
@@ -184,12 +184,12 @@ Finally, register your improver in the improver registry at
 
     IMPROVERS_REGISTRY = {x.qualified_name: x for x in IMPROVERS_REGISTRY}
 
-Congratulations! You've written your first improver.
+Congratulations! You have written your first improver.
 
 Run Your First Improver
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If everything went well, you'll see your improver in the list of available improvers.
+If everything went well, you will see your improver in the list of available improvers.
 
 .. code-block:: console
    :emphasize-lines: 6
@@ -244,7 +244,7 @@ For more visibility, turn on debug logs in :file:`vulnerablecode/settings.py`.
         },
     }
 
-Invoke the improve command now and you'll see (in a fresh database, after importing):
+Invoke the improve command now and you will see (in a fresh database, after importing):
 
 .. code-block:: console
 

pyproject.toml

@@ -46,23 +46,16 @@ addopts = [
     "--doctest-modules",
     # Ignore the following doctests until these files are migrated to
     # import-improve structure
-    "--ignore=vulnerabilities/importers/alpine_linux.py",
     "--ignore=vulnerabilities/importers/apache_httpd.py",
     "--ignore=vulnerabilities/importers/apache_kafka.py",
     "--ignore=vulnerabilities/importers/apache_tomcat.py",
-    "--ignore=vulnerabilities/importers/archlinux.py",
-    "--ignore=vulnerabilities/importers/debian.py",
     "--ignore=vulnerabilities/importers/elixir_security.py",
     "--ignore=vulnerabilities/importers/gentoo.py",
-    "--ignore=vulnerabilities/importers/github.py",
     "--ignore=vulnerabilities/importers/istio.py",
     "--ignore=vulnerabilities/importers/kaybee.py",
     "--ignore=vulnerabilities/importers/npm.py",
-    "--ignore=vulnerabilities/importers/nvd.py",
-    "--ignore=vulnerabilities/importers/openssl.py",
     "--ignore=vulnerabilities/importers/postgresql.py",
     "--ignore=vulnerabilities/importers/project_kb_msr2019.py",
-    "--ignore=vulnerabilities/importers/redhat.py",
     "--ignore=vulnerabilities/importers/retiredotnet.py",
     "--ignore=vulnerabilities/importers/ruby.py",
     "--ignore=vulnerabilities/importers/rust.py",

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = vulnerablecode
-version = 30.1.1
+version = 30.2.0
 license = Apache-2.0 AND CC-BY-SA-4.0
 
 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390

vulnerabilities/importers/alpine_linux.py

@@ -15,19 +15,13 @@
 from typing import Mapping
 from urllib.parse import urljoin
 
-import requests
 from bs4 import BeautifulSoup
-from django.db.models.query import QuerySet
 from packageurl import PackageURL
 from univers.versions import AlpineLinuxVersion
 
 from vulnerabilities.importer import AdvisoryData
 from vulnerabilities.importer import AffectedPackage
 from vulnerabilities.importer import Importer
-from vulnerabilities.improver import MAX_CONFIDENCE
-from vulnerabilities.improver import Improver
-from vulnerabilities.improver import Inference
-from vulnerabilities.models import Advisory
 from vulnerabilities.references import WireSharkReference
 from vulnerabilities.references import XsaReference
 from vulnerabilities.references import ZbxReference

vulnerabilities/importers/archlinux.py

@@ -10,7 +10,6 @@
 from typing import Iterable
 from typing import List
 from typing import Mapping
-from urllib.request import urlopen
 
 from packageurl import PackageURL
 from univers.version_range import ArchLinuxVersionRange