Prioritize hashes and download URL for PurlDB mapping

In order to get an accurate mapping for a package in DejaCode to PurlDB entries the patched query prioritizes the hashes. This is needed in cases where the same PURL (without query parameters) can have multiple different download URLs as is the case with Python packages and various binaries for different hardware architectures or interpreter versions. Additionally, lookups for SHA-256 and MD5 are added as SHA-1 may not be populated under all circumstances. Hashes from SBOM imports, generated by tools such as cdxgen, commonly do not use SHA-1 anymore, since it is a mostly deprecated hashing algorithm due to the risk of hash collisions. SHA-512 could not yet be added as PurlDB does not support a lookup for it. The reason for the order of prioritization is that hashes give the most accurate for the content of the package, download URL at least points to the download location which would still allow to differentiate between the different target architectures, and lastly the PURL itself in case no fully accurate matches could be found otherwise. The results are then filtered by checking that PURLs match. Here a modification is made to also strip the query parameters from the PurlDB PURL as they may also contain them and previously caused matches to not be found. For reference see the following issues: #307 #383

Signed-off-by: Robert Guetzkow <[email protected]>

Author: rogu-beta

component_catalog/models.py

@@ -2506,13 +2506,34 @@ def create_from_url(cls, url, user):
         if download_url and not purldb_data:
             package_data = collect_package_data(download_url)
 
+        if sha512 := package_data.get("sha512"):
+            if sha512_match := scoped_packages_qs.filter(sha512=sha512):
+                package_link = sha512_match[0].get_absolute_link()
+                raise PackageAlreadyExistsWarning(
+                    f"{url} already exists in your Dataspace as {package_link}"
+                )
+
+        if sha256 := package_data.get("sha256"):
+            if sha256_match := scoped_packages_qs.filter(sha256=sha256):
+                package_link = sha256_match[0].get_absolute_link()
+                raise PackageAlreadyExistsWarning(
+                    f"{url} already exists in your Dataspace as {package_link}"
+                )
+
         if sha1 := package_data.get("sha1"):
             if sha1_match := scoped_packages_qs.filter(sha1=sha1):
                 package_link = sha1_match[0].get_absolute_link()
                 raise PackageAlreadyExistsWarning(
                     f"{url} already exists in your Dataspace as {package_link}"
                 )
 
+        if md5 := package_data.get("md5"):
+            if md5_match := scoped_packages_qs.filter(md5=md5):
+                package_link = md5_match[0].get_absolute_link()
+                raise PackageAlreadyExistsWarning(
+                    f"{url} already exists in your Dataspace as {package_link}"
+                )
+
         # Duplicate the declared_license_expression into the license_expression field.
         if declared_license_expression := package_data.get("declared_license_expression"):
             package_data["license_expression"] = declared_license_expression
@@ -2528,9 +2549,9 @@ def get_purldb_entries(self, user, max_request_call=0, timeout=10):
         Return the PurlDB entries that correspond to this Package instance.
 
         Matching on the following fields order:
-        - Package URL
-        - SHA1
+        - Hash
         - Download URL
+        - Package URL
 
         A `max_request_call` integer can be provided to limit the number of
         HTTP requests made to the PackageURL server.
@@ -2542,12 +2563,16 @@ def get_purldb_entries(self, user, max_request_call=0, timeout=10):
         purldb_entries = []
 
         package_url = self.package_url
-        if package_url:
-            payloads.append({"purl": package_url})
+        if self.sha256:
+            payloads.append({"sha256": self.sha256})
         if self.sha1:
             payloads.append({"sha1": self.sha1})
+        if self.md5:
+            payloads.append({"md5": self.md5})
         if self.download_url:
             payloads.append({"download_url": self.download_url})
+        if package_url:
+            payloads.append({"purl": package_url})
 
         purldb = PurlDB(user.dataspace)
         for index, payload in enumerate(payloads):