Merge remote-tracking branch 'origin/main' into vulntotal

Author: pombredanne

.gitignore

@@ -102,3 +102,4 @@ Pipfile
 .ve
 *.bak
 /.cache/
+/tmp/

CHANGELOG.rst

@@ -1,62 +1,170 @@
 Release notes
 =============
 
+
+
+Version v30.3.1
+----------------
+
+This is a minor bug fix release.
+
+- We enabled proper CSRF configuration for deployments
+
+
+Version v30.3.0
+----------------
+
+This is a feature update release including minor bug fixes and the introduction
+of API keys and API throttling.
+
+- We enabled API throttling for a basic user and for a staff user 
+  they can have unlimited access on API.
+
+- We added throttle rate for each API endpoint and it can be 
+  configured from the settings #991 https://github.com/nexB/vulnerablecode/issues/991
+
+- We improved how we import NVD data
+- We refactored and made the purl2cpe script work to dump purl to CPE mappings
+
+Internally:
+
+- We aligned key names internally with the names used in the UI and API (such as affected and fixed)
+- We now use querysets as model managers and have streamlined view code
+
+
+Version v30.2.1
+----------------
+
+- We refactored and fixed the LaunchPad API code.
+- We now ignore qualifiers and subpath from PURL search lookups.
+- We fixed severity table column spillover.
+
+
+Version v30.2.0
+----------------
+
+This is a critical bug fix release including features updates.
+
+- We fixed critical performance issues that made the web UI unusable. This include
+  removing some less interesting redundant details displayed in the web UI for
+  vulnerabilities.
+- We made minor documentation updates.
+- We re-enabled support for Arch linux, Debian, and Ubuntu security advisories importers
+- We added a new improver for Oval data sources
+- We improved Alpine linux and Gitlab security advisories importers
+
+The summary of performance improvements include these fixes:
+
+- Cascade queries from exact to approximate searches to avoid full table scans
+  in all cases. This is a band-aid for now. The proper solution will likely
+  require using full text search instead.
+- Avoid iceberg queries with "prefetch related" to limit the number of queries
+  that are needed in the UI
+- Do not recreate querysets from scratch but instead allow these to be chained
+  for simpler and correct code.
+- Remove extra details from the vulnerability pacge: each package was further
+  listing its related vulnerabilities creating an iceberg query.
+- Enable the django-debug-toolbar with a setting to easily profile queries on demand
+  by setting both VULNERABLECODE_DEBUG and VULNERABLECODE_DEBUG_TOOLBAR enviroment
+  variables.
+
+
+Version v30.1.1
+----------------
+
+- We added a new web UI link to explain how to obtain an API for the publicly
+  hosted VulnerableCode
+
+
+Version v30.1.0
+----------------
+
+- We added a new "/packages/all" API endpoint to get all Package URLs know to be vulnerable.
+
+
 Version v30.0.0
 ----------------
 
+This is a major version that is not backward compatible.
+
 - We refactored the core processing with Importers that import data and Improvers that
   transform imported data and convert that in Vulnerabilities and Packages. Improvers can
   also improve and refine imported and existing data as well as enrich data using external
   data sources. The migration to this new architecture is under way and not all importers
-  are available. You can track the progress in this issue: https://github.com/nexB/vulnerablecode/issues/597 
+  are available.
+
   Because of these extensive changes, it is not possible to migrate existing imported
   data to the new schema. You will need instead to restart imports from an empty database
-  or request access to the new vulnerablecode.io live instance.
+  or access the new public.vulnerablecode.io live instance. We also provide a database dump.
+
+- You can track the progress of this refactoring in this issue:
+  https://github.com/nexB/vulnerablecode/issues/597
 
 - We added new data sources including PYSEC, GitHub and GitLab.
 
 - We improved the documentation including adding development examples for importers and improvers.
 
-- We removed the ability to edit relationships from the UI. The UI is now read-only
-  and we will need to design a different UI for proper review and curation of vulnerabilities.
+- We removed the ability to edit relationships from the UI. The UI is now read-only.
+
+- We replaced the web UI with a brand new UI based on the same overall look and feel as ScanCode.io.
 
 - We added support for NixOS as a Linux deployment target.
 
 - The aliases of a vulnerabily are reported in the API vulnerabilities/ endpoint
 
-
 - There are breaking Changes at API level with changes in the data structure:
 
   - in the /api/vulnerabilities/ endpoint:
 
-    - Rename `resolved_packages` to `fixed_packages` 
+    - Rename `resolved_packages` to `fixed_packages`
     - Rename `unresolved_packages` to `affected_packages`
     - Rename `url` to `reference_url` in the reference list
+    - Add is_vulnerable property in fixed and affected_packages.
 
   - in the /api/packages/ endpoint:
 
     - Rename `unresolved_vulnerabilities` to `affected_by_vulnerabilities`
     - Rename  `resolved_vulnerabilities` to `fixing_vulnerabilities`
     - Rename `url` to `reference_url` in the reference list
+    - Add new attribute `is_resolved`
+    - Add namespace filter
 
-- We have provided backward compatibility for `url` and `unresolved_vulnerabilities` for now
+- We have provided backward compatibility for `url` and `unresolved_vulnerabilities` for now.
+  These will be removed in the next major version and should be considered as deprecated.
 
-- There is a new experimental cpe/ API endpoint to lookup for vulnerabilities by CPE and 
+- There is a new experimental `cpe/` API endpoint to lookup for vulnerabilities by CPE and
   another aliases/ endpoint to lookup for vulnerabilities by aliases. These two endpoints will be
   replaced by query parameters on the main vulnerabilities/ endpoint when stabilized.
 
-- Added filters for vulnerabilities endpoint to get fixed packages in accordance to the details given in filters:
-  For example:
-  - /api/vulnerabilities?type=pypi&namespace=foo&name=bar 
-    will give only fixed versioned purls of this type `pkg:pypi/foo/bar`
+- We added filters for vulnerabilities endpoint to get fixed packages in accordance
+  to the details given in filters: For example, when you call the endpoint this way
+  ``/api/vulnerabilities?type=pypi&namespace=foo&name=bar``, you will receive only
+  fixed versioned purls of the type ``pypi``, namespace ``foo`` and name ``bar``.
 
 - Package endpoint will give fixed packages of only those that
   matches type, name, namespace, subpath and qualifiers of the package queried.
 
+- Paginated initial listings to display a small number of records
+  and provided page per size with a maximum limit of 100 records per page.
+
+- Add fixed packages in vulnerabilities details in packages endpoint.
+
+- Add bulk search support for CPEs.
+
+- Add authentication for REST API endpoint.
+  The autentication is disabled by default and can be enabled using the
+  VULNERABLECODEIO_REQUIRE_AUTHENTICATION settings.
+  When enabled, users have to authenticate using
+  their API Key in the REST API.
+  Users can be created using the Django "createsuperuser" management command.
+
+- The data license is now CC-BY-SA-4.0 as this is the highest common
+  denominator license among all the data sources we collect and aggregate.
+
 Other:
 
-- we dropped calver to use a plain semver.
-- we adopted vers and the new univers library to handle version ranges.
+- We dropped calver to use a plain semver.
+- We adopted vers and the new univers library to handle version ranges.
 
 
 Version v20.10

Makefile

@@ -109,7 +109,7 @@ sqlite:
 	@$(MAKE) migrate
 
 run:
-	${MANAGE} runserver 8001 --noreload --insecure
+	${MANAGE} runserver 8001 --insecure
 
 test:
 	@echo "-> Run the test suite"

docs/source/api.rst

@@ -0,0 +1,50 @@
+.. _api:
+
+API overview
+========================
+
+
+Browse the Open API documentation
+------------------------------------
+
+- https://public.vulnerablecode.io/api/docs/ for documentation with Swagger
+- https://public.vulnerablecode.io/api/schema/ for the OpenAPI schema
+
+
+Enable the API key authentication
+------------------------------------
+
+There is a setting VULNERABLECODEIO_REQUIRE_AUTHENTICATION for this. Use it this
+way::
+
+    $ VULNERABLECODEIO_REQUIRE_AUTHENTICATION=1 make run
+
+
+Create an API key-only user
+------------------------------------
+
+This can be done in the admin and from the command line::
+
+    $ ./manage.py create_api_user --email "[email protected]" --first-name="Phil" --last-name "Goel"
+    User [email protected] created with API key: ce8616b929d2adsddd6146346c2f26536423423491
+
+
+Access the API using curl
+-----------------------------
+
+    curl -X GET -H 'Authorization: Token <YOUR TOKEN>' https://public.vulnerablecode.io/api/
+
+
+API endpoints
+---------------
+
+
+There are two primary endpoints:
+
+- packages/: this is the main endpoint where you can lookup vulnerabilities by package.
+
+- vulnerabilities/: to lookup by vulnerabilities
+
+And two secondary endpoints, used to query vulnerability aliases (such as CVEs)
+and vulnerability by CPEs: cpes/ and aliases/
+

docs/source/command-line-interface.rst

@@ -3,20 +3,21 @@
 Command Line Interface
 ======================
 
-The main entry point is Django's :guilabel:`manage.py` management commands.
+The main entry point is the Django :guilabel:`manage.py` management command script.
 
 ``$ ./manage.py --help``
------------------------
+------------------------
 
 Lists all sub-commands available, including Django built-in commands.
 VulnerableCode's own commands are listed under the ``[vulnerabilities]`` section::
 
     $ ./manage.py --help
     ...
     [vulnerabilities]
-        create_cpe_to_purl_map
-        importer
-        improver
+        import
+        improve
+        purl2cpe
+
 
 ``$ ./manage.py <subcommand> --help``
 ---------------------------------------
@@ -58,3 +59,17 @@ Other variations:
 
 * ``--list`` List all available improvers
 * ``--all`` Run all available improvers
+
+
+
+``$ ./manage.py purl2cpe --destination <directory``
+------------------------------------------
+
+Dump a mapping of CPEs to PURLs grouped by vulnerability in the ``destination``
+directory.
+
+
+Other variations:
+
+* ``--limit`` Limit the number of processed vulnerabilities
+

docs/source/conf.py

@@ -18,8 +18,8 @@
 # -- Project information -----------------------------------------------------
 
 project = "VulnerableCode"
-copyright = "nexb Inc. and others"
-author = "nexb Inc. and others"
+copyright = "nexB Inc. and others"
+author = "nexB Inc. and others"
 
 
 # -- General configuration ---------------------------------------------------

docs/source/contributing.rst

@@ -5,7 +5,7 @@ Contributing to VulnerableCode
 
 Thank you so much for being so interested in contributing to VulnerableCode. We
 are always on the lookout for enthusiastic contributors like you who can make
-our project better, and we're willing to lend a helping hand if you have any
+our project better, and we are willing to lend a helping hand if you have any
 questions or need guidance along the way. That being said, here are a few
 resources to help you get started.
 
@@ -33,13 +33,13 @@ join our community. Below are some examples to get involved:
 First Timers
 ^^^^^^^^^^^^
 
-You are here to help, but you're a new contributor! No worries, we always
+You are here to help, but you are a new contributor! No worries, we always
 welcome newcomer contributors. We maintain some
 `good first issues <https://github.com/nexB/vulnerablecode/labels/good%20first%20issue>`_
 and encourage new contributors to work on those issues for a smooth start.
 
 .. tip::
-    If you're an open-source newbie, make sure to check the extra resources at
+    If you are an open-source newbie, make sure to check the extra resources at
     the bottom of this page to get the hang of the contribution process!
 
 Code Contributions
@@ -75,7 +75,7 @@ Other Ways
 ^^^^^^^^^^
 
 You want to contribute to other aspects of the VulnerableCode project, and you
-can't find what you're looking for! You can always discuss new topics, ask
+cannot find what you are looking for! You can always discuss new topics, ask
 questions, and interact with us and other community members on
 `AboutCode Gitter <https://gitter.im/aboutcode-org/discuss>`_ and `VulnerableCode Gitter <https://gitter.im/aboutcode-org/vulnerablecode>`_
 

docs/source/importers_link.rst

@@ -0,0 +1,6 @@
+.. _importers_link:
+
+Importers
+=========
+
+.. include:: ../../SOURCES.rst

docs/source/index.rst

@@ -1,9 +1,18 @@
-VulnerableCode documentation
+Welcome to VulnerableCode!
 =============================
 
-Welcome to VulnerableCode! In this documentation you’ll find information on:
+*VulnerableCode* provides an open database of software packages that are affected
+by known security vulnerabilities aka. *"vulnerable packages"*.
 
-- An overview of VulnerableCode
+VulnerableCode is also a free and open source software (FOSS) project that
+provides the tools to build this open database. The tools handle collecting,
+aggregating and correlating these vulnerabilities and relating them to a correct
+package version. Our project also supports a public cloud instance of this
+database - VulnerableCode.io.
+
+In this documentation you will find information on:
+
+- An overview of VulnerableCode and what you can do with it
 - Installation instructions
 - How to make technical contributions to the project and the community
 
@@ -33,6 +42,8 @@ Welcome to VulnerableCode! In this documentation you’ll find information on:
     reference_improver_overview
     reference_framework_overview
     command-line-interface
+    importers_link
+    api
 
 .. toctree::
    :maxdepth: 1