Merge branch 'main' into parallel-importer

Author: pombredanne

.gitignore

@@ -102,3 +102,4 @@ Pipfile
 .ve
 *.bak
 /.cache/
+/tmp/

CHANGELOG.rst

@@ -1,55 +1,103 @@
 Release notes
 =============
 
+
+Version v30.1.1
+----------------
+
+- We added a new web UI link to explain how to obtain an API for the publicly
+  hosted VulnerableCode
+
+
+Version v30.1.0
+----------------
+
+- We added a new "/packages/all" API endpoint to get all Package URLs know to be vulnerable.
+
+
 Version v30.0.0
 ----------------
 
+This is a major version that is not backward compatible.
+
 - We refactored the core processing with Importers that import data and Improvers that
   transform imported data and convert that in Vulnerabilities and Packages. Improvers can
   also improve and refine imported and existing data as well as enrich data using external
   data sources. The migration to this new architecture is under way and not all importers
-  are available. You can track the progress in this issue: https://github.com/nexB/vulnerablecode/issues/597 
+  are available.
+
   Because of these extensive changes, it is not possible to migrate existing imported
   data to the new schema. You will need instead to restart imports from an empty database
-  or request access to the new vulnerablecode.io live instance.
+  or access the new public.vulnerablecode.io live instance. We also provide a database dump.
+
+- You can track the progress of this refactoring in this issue:
+  https://github.com/nexB/vulnerablecode/issues/597 
 
 - We added new data sources including PYSEC, GitHub and GitLab.
 
 - We improved the documentation including adding development examples for importers and improvers.
 
-- We removed the ability to edit relationships from the UI. The UI is now read-only
-  and we will need to design a different UI for proper review and curation of vulnerabilities.
+- We removed the ability to edit relationships from the UI. The UI is now read-only.
+
+- We replaced the web UI with a brand new UI based on the same overall look and feel as ScanCode.io.
 
 - We added support for NixOS as a Linux deployment target.
 
 - The aliases of a vulnerabily are reported in the API vulnerabilities/ endpoint
 
-
 - There are breaking Changes at API level with changes in the data structure:
 
   - in the /api/vulnerabilities/ endpoint:
 
     - Rename `resolved_packages` to `fixed_packages` 
     - Rename `unresolved_packages` to `affected_packages`
     - Rename `url` to `reference_url` in the reference list
+    - Add is_vulnerable property in fixed and affected_packages.
 
   - in the /api/packages/ endpoint:
 
     - Rename `unresolved_vulnerabilities` to `affected_by_vulnerabilities`
     - Rename  `resolved_vulnerabilities` to `fixing_vulnerabilities`
     - Rename `url` to `reference_url` in the reference list
+    - Add new attribute `is_resolved`
+    - Add namespace filter
 
-- We have provided backward compatibility for `url` and `unresolved_vulnerabilities` for now
+- We have provided backward compatibility for `url` and `unresolved_vulnerabilities` for now.
+  These will be removed in the next major version and should be considered as deprecated.
 
-- There is a new experimental cpe/ API endpoint to lookup for vulnerabilities by CPE and 
+- There is a new experimental `cpe/` API endpoint to lookup for vulnerabilities by CPE and 
   another aliases/ endpoint to lookup for vulnerabilities by aliases. These two endpoints will be
   replaced by query parameters on the main vulnerabilities/ endpoint when stabilized.
 
+- We added filters for vulnerabilities endpoint to get fixed packages in accordance
+  to the details given in filters: For example, when you call the endpoint this way
+  ``/api/vulnerabilities?type=pypi&namespace=foo&name=bar``, you will receive only
+  fixed versioned purls of the type ``pypi``, namespace ``foo`` and name ``bar``.
+
+- Package endpoint will give fixed packages of only those that
+  matches type, name, namespace, subpath and qualifiers of the package queried.
+
+- Paginated initial listings to display a small number of records 
+  and provided page per size with a maximum limit of 100 records per page.
+
+- Add fixed packages in vulnerabilities details in packages endpoint.
+
+- Add bulk search support for CPEs.
+
+- Add authentication for REST API endpoint.
+  The autentication is disabled by default and can be enabled using the
+  VULNERABLECODEIO_REQUIRE_AUTHENTICATION settings.
+  When enabled, users have to authenticate using 
+  their API Key in the REST API.
+  Users can be created using the Django "createsuperuser" management command.
+
+- The data license is now CC-BY-SA-4.0 as this is the highest common
+  denominator license among all the data sources we collect and aggregate. 
 
 Other:
 
-- we dropped calver to use a plain semver.
-- we adopted vers and the new univers library to handle version ranges.
+- We dropped calver to use a plain semver.
+- We adopted vers and the new univers library to handle version ranges.
 
 
 Version v20.10

Makefile

@@ -109,7 +109,7 @@ sqlite:
 	@$(MAKE) migrate
 
 run:
-	${MANAGE} runserver 8001 --noreload --insecure
+	${MANAGE} runserver 8001 --insecure
 
 test:
 	@echo "-> Run the test suite"

docs/source/command-line-interface.rst

@@ -6,7 +6,7 @@ Command Line Interface
 The main entry point is Django's :guilabel:`manage.py` management commands.
 
 ``$ ./manage.py --help``
------------------------
+------------------------
 
 Lists all sub-commands available, including Django built-in commands.
 VulnerableCode's own commands are listed under the ``[vulnerabilities]`` section::

docs/source/conf.py

@@ -18,8 +18,8 @@
 # -- Project information -----------------------------------------------------
 
 project = "VulnerableCode"
-copyright = "nexb Inc. and others"
-author = "nexb Inc. and others"
+copyright = "nexB Inc. and others"
+author = "nexB Inc. and others"
 
 
 # -- General configuration ---------------------------------------------------

docs/source/importers_link.rst

@@ -0,0 +1,6 @@
+.. _importers_link:
+
+Importers
+=========
+
+.. include:: ../../SOURCES.rst

docs/source/installation.rst

@@ -109,7 +109,7 @@ Local development installation
 Supported Platforms
 ^^^^^^^^^^^^^^^^^^^
 
-**VulnerableCode* has been tested and is supported on the following operating systems:
+**VulnerableCode** has been tested and is supported on the following operating systems:
 
     #. **Debian-based** Linux distributions
     #. **macOS** 12.1 and up
@@ -122,7 +122,7 @@ Pre-installation Checklist
 
 Before you install VulnerableCode, make sure you have the following prerequisites:
 
- * **Python: 3.8+* found at https://www.python.org/downloads/
+ * **Python: 3.8+** found at https://www.python.org/downloads/
  * **Git**: most recent release available at https://git-scm.com/
  * **PostgreSQL**: release 10 or later found at https://www.postgresql.org/ or
    https://postgresapp.com/ on macOS
@@ -212,8 +212,6 @@ application.
     This setup is **not suitable for deployments** and **only supported for local
     development**.
 
-An overview of the web application usage is available at :ref:`user_interface`.
-
 
 Upgrading
 ^^^^^^^^^

docs/source/introduction.rst

@@ -7,8 +7,8 @@ VulnerableCode is a work-in-progress towards a free and open vulnerabilities
 database and the packages they impact and the tools to aggregate and correlate
 these vulnerabilities.
 
-Why VulnerableCode ?
----------------------
+Why VulnerableCode?
+-------------------
 
 The existing solutions are commercial proprietary vulnerability databases, which
 in itself does not make sense because the data is about FOSS (Free and Open
@@ -27,12 +27,12 @@ security issues because:
    fundamental questions "Is package foo vulnerable" and "Is package foo
    vulnerable to vulnerability bar?"
 
-How does it work ?
--------------------
+How does it work?
+-----------------
 
 VulnerableCode independently aggregates many software vulnerability data sources
 and supports data re-creation in a decentralized fashion. These data sources
-(see complete list `here <./SOURCES.rst>`_) include security advisories
+(see complete list :ref:`here <importers_link>`) include security advisories
 published by Linux and BSD distributions, application software package managers
 and package repositories, FOSS projects, GitHub and more. Thanks to this
 approach, the data is focused on specific ecosystems yet aggregated in a single
@@ -59,14 +59,17 @@ exposure due to various reasons like but not limited to the complicated
 procedure to receive CVE ID or not able to classify a bug as a security
 compromise.
 
-Recent presentations:
 
-- `Open Source Summit 2020 <https://github.com/nexB/vulnerablecode/blob/main/docs/Presentations/Why-Is-There-No-Free-Software-Vulnerability-Database-v1.0.pdf>`_
+Is VulnerableCode being actively developed?
+-------------------------------------------
 
-Should I use VulnerableCode ?
--------------------------------
+Yes -- VulnerableCode is a work in progress! Please stay in touch on our `Gitter channel <https://gitter.im/aboutcode-org/vulnerablecode>`_; and if you have any feedback, feel free to `enter an issue in our GitHub repo <https://github.com/nexB/vulnerablecode/issues>`_.
 
-VulnerableCode is a work in progress project and will likely go through major changes. Please stay in touch on our `Gitter channel <https://gitter.im/aboutcode-org/vulnerablecode>`_
+
+Recent presentations
+--------------------
+
+- `Open Source Summit 2020 <https://github.com/nexB/vulnerablecode/blob/main/docs/Presentations/Why-Is-There-No-Free-Software-Vulnerability-Database-v1.0.pdf>`_
 
 .. Some of this documentation is borrowed from the metaflow documentation and is also
    under Apache-2.0

docs/source/misc.rst

@@ -17,7 +17,7 @@ Here is an example::
 
     [Service]
     Type=oneshot
-    ExecStart=/path/to/venv/bin/python /path/to/vulnerablecode/manage.py import --all
+    ExecStart=/path/to/venv/bin/python /path/to/vulnerablecode/manage.py import --all && /path/to/venv/bin/python /path/to/vulnerablecode/manage.py improve --all
 
     $ cat ~/.config/systemd/user/vulnerablecode.timer
 

docs/source/reference_importer_overview.rst

@@ -3,34 +3,33 @@
 Importer Overview
 ==================
 
-Importers are responsible to scrape vulnerability data from various data sources without creating
-a complete relational model between vulnerabilites, their fixes and store them in a structured
-fashion.
+Importers are responsible for scraping vulnerability data such as vulnerabilities and their fixes
+and for storing the scraped information in a structured fashion. The structured data created by the
+importer then provides input to an improver (see :ref:`improver-overview`), which is responsible
+for creating a relational model for vulnerabilities, affected packages and fixed packages.
 
-All importer implementation related code is defined in :file:`vulnerabilites/importer.py`.
+All importer implementation-related code is defined in :file:`vulnerabilites/importer.py`.
 
-Whereas, the framework related code for actually invoking and processing the importers are
-situated in :file:`vulnerabilites/import_runner.py`.
+In addition, the framework-related code for actually invoking and processing the importers is
+located in :file:`vulnerabilites/import_runner.py`.
 
-The importers, after scraping, provide with ``AdvisoryData`` objects. These objects are then
+The importers, after scraping, provide ``AdvisoryData`` objects. These objects are then
 processed and inserted into the ``Advisory`` model.
 
 While implementing an importer, it is important to make sure that the importer does not alter the
-upstream data at all. Its only job is to convert the data from a data source into structured - yet
-non relational - data. The importers must **not** be smart or performing trickeries
-under the hood.
-This ensures that we always have a *true* copy of an advisory without any speculations or
-improvements.
+upstream data at all. Its only job is to convert the data from a data source into structured -- yet
+non-relational -- data.  This ensures that we always have a *true* copy of an advisory without any
+modifications.
 
-As importers do not speculate and given that a lot of advisories publish version ranges of affected
+Given that a lot of advisories publish version ranges of affected
 packages, it is necessary to store those ranges in a structured manner. *Vers* was designed to
 solve this problem. It has been implemented in the `univers <https://github.com/nexB/univers>`_
 library whose development goes hand in hand with VulnerableCode.
 
-The data imported by importers is not useful by itself, it must be processed into a relational
-model. The version ranges are required to be dissolved into concrete ranges. These are achieved by
-``Improvers``. For more, see: :ref:`improver-overview`
+The data imported by importers is not useful by itself: it must be processed into a relational
+model. The version ranges are required to be resolved into concrete ranges. These are achieved by
+``Improvers`` (see :ref:`improver-overview` for details).
 
-As of now, the following importers have been implemented in VulnerableCode
+As of now, the following importers have been implemented in VulnerableCode:
 
 .. include:: ../../SOURCES.rst