Prioritize hashes and download URL for PurlDB mapping (#430)

* Prioritize hashes and download URL for PurlDB mapping

In order to get an accurate mapping for a package in DejaCode to PurlDB entries the patched query prioritizes the hashes. This is needed in cases where the same PURL (without query parameters) can have multiple different download URLs as is the case with Python packages and various binaries for different hardware architectures or interpreter versions. Additionally, lookups for SHA-256 and MD5 are added as SHA-1 may not be populated under all circumstances. Hashes from SBOM imports, generated by tools such as cdxgen, commonly do not use SHA-1 anymore, since it is a mostly deprecated hashing algorithm due to the risk of hash collisions. SHA-512 could not yet be added as PurlDB does not support a lookup for it. The reason for the order of prioritization is that hashes give the most accurate for the content of the package, download URL at least points to the download location which would still allow to differentiate between the different target architectures, and lastly the PURL itself in case no fully accurate matches could be found otherwise. The results are then filtered by checking that PURLs match. Here a modification is made to also strip the query parameters from the PurlDB PURL as they may also contain them and previously caused matches to not be found. For reference see the following issues: #307 #383

Signed-off-by: Robert Guetzkow <[email protected]>

* Update component_catalog/models.py

Remove code duplication and reduce database queries to a single one

Signed-off-by: tdruez <[email protected]>

* Add details about matching order in docstring

Signed-off-by: tdruez <[email protected]>

---------

Signed-off-by: Robert Guetzkow <[email protected]>
Signed-off-by: tdruez <[email protected]>
Co-authored-by: tdruez <[email protected]>

Author: rogu-beta

component_catalog/models.py

@@ -2506,9 +2506,16 @@ def create_from_url(cls, url, user):
         if download_url and not purldb_data:
             package_data = collect_package_data(download_url)
 
-        if sha1 := package_data.get("sha1"):
-            if sha1_match := scoped_packages_qs.filter(sha1=sha1):
-                package_link = sha1_match[0].get_absolute_link()
+        # Check for existing package by hash fields with a single database query
+        hash_fields = ["sha512", "sha256", "sha1", "md5"]
+        hash_filters = models.Q()
+        for hash_field in hash_fields:
+            if hash_value := package_data.get(hash_field):
+                hash_filters |= models.Q(**{hash_field: hash_value})
+
+        if hash_filters:
+            if package_match := scoped_packages_qs.filter(hash_filters).first():
+                package_link = package_match.get_absolute_link()
                 raise PackageAlreadyExistsWarning(
                     f"{url} already exists in your Dataspace as {package_link}"
                 )
@@ -2527,10 +2534,10 @@ def get_purldb_entries(self, user, max_request_call=0, timeout=10):
         """
         Return the PurlDB entries that correspond to this Package instance.
 
-        Matching on the following fields order:
-        - Package URL
-        - SHA1
-        - Download URL
+        Matching is performed in order of decreasing accuracy:
+        1. Hash - Most accurate, matches exact file content
+        2. Download URL - High accuracy, matches specific package source
+        3. Package URL - Broadest match, may return multiple versions/variants
 
         A `max_request_call` integer can be provided to limit the number of
         HTTP requests made to the PackageURL server.
@@ -2542,12 +2549,16 @@ def get_purldb_entries(self, user, max_request_call=0, timeout=10):
         purldb_entries = []
 
         package_url = self.package_url
-        if package_url:
-            payloads.append({"purl": package_url})
+        if self.sha256:
+            payloads.append({"sha256": self.sha256})
         if self.sha1:
             payloads.append({"sha1": self.sha1})
+        if self.md5:
+            payloads.append({"md5": self.md5})
         if self.download_url:
             payloads.append({"download_url": self.download_url})
+        if package_url:
+            payloads.append({"purl": package_url})
 
         purldb = PurlDB(user.dataspace)
         for index, payload in enumerate(payloads):