JVM d2d: match dependencies from pom.xml

[chinyeungli]

• Ref: back2source: Run pipeline for Java and native code and Android apk and dex classes #1473

A significant number of non-matching files appear to originate from dependencies included during the build process.
For instance:
poi-tl-v1.12.0
Devel: https://github.com/Sayi/poi-tl/archive/refs/tags/v1.12.0.zip#from
Deploy: https://github.com/Sayi/poi-tl/releases/download/v1.12.0/poi-tl-cli.jar#to
Some of the non-matching files found in the deployment:

to/com/beust/jcommander/validators/NoValidator.class
to/com/beust/jcommander/validators/NoValueValidator.class
to/com/beust/jcommander/validators/PositiveInteger.class
to/com/beust/jcommander/WrappedParameter.class
to/com/codewaves/codehighlight/core/Highlighter$HighlightResult.class
to/com/codewaves/codehighlight/core/Highlighter.class
to/com/codewaves/codehighlight/core/HighlightParser$1.class
to/com/codewaves/codehighlight/core/HighlightParser$ParentWrapper.class
to/com/codewaves/codehighlight/core/HighlightParser.class

However, these "missing" code packages exist in the pom.xml
./poi-tl-cli/pom.xml

		<dependency>
			<groupId>com.beust</groupId>
			<artifactId>jcommander</artifactId>
			<version>1.72</version>
		</dependency>

./poi-tl-plugin-highlight/pom.xml

		<dependency>
			<groupId>com.deepoove</groupId>
			<artifactId>codehighlight</artifactId>
			<version>1.0.3</version>
		</dependency>

It would be great if we could parse/extract/index the dependency packages listed in the pom.xml, and then correlate them with the "missing" source files to improve the D2D process.

[chinyeungli]

Would the correct approach be to parse the package dependencies from the pom.xml, construct a PURL (i.e pkg:maven/com.beust/jcommander@1.72 and pkg:maven/com.deepoove/codehighlight@1.0.3), use that PURL to download the archive/codebase, run an info scan (scan_single_package/scan_codebase) on it, and perform the matching based on the scan results?

[pombredanne]

Would the correct approach be to parse the package dependencies from the pom.xml, construct a PURL (i.e pkg:maven/com.beust/jcommander@1.72 and pkg:maven/com.deepoove/codehighlight@1.0.3), use that PURL to download the archive/codebase, run an info scan (scan_single_package/scan_codebase) on it, and perform the matching based on the scan results?

Actaully, we cannot rely on all the POM dependencies to be effectively deployed and bundled in a JAR, as many of these will be installed separately, hence not in the d2d scope.

Here based on the POM you listed the culprit is:

• https://github.com/Sayi/poi-tl/blob/v1.12.0/poi-tl-cli/pom.xml#L152 using....
• https://maven.apache.org/plugins/maven-assembly-plugin/index.html
• ... which seems to bundle all dependencies and is new to me. This is like Maven shade.

We may be able to parse Maven assembly plugin output in the JARs

We cannot parse Maven assembly plugin instructions, but what we could do as a generally working approach would be to:

• include the binary JARs of all POM dependencies in the FROM side
• d2d the class with missing sources on that

Or do the same with PurlDB.

Then also report these are issues as we can find the PURLs, but surfacing the missing sources and bundling of binaries is what matters.

We should be able to do something

[chinyeungli]

So far, I’ve implemented the following workflow:

• Parse the section of the pom.xml from the development codebase
• Construct Maven Package URLs (purls) for each dependency
• Download and extract the corresponding packages
• Retrieve basic metadata for each resource
• Update the project codebase

I am using the following functions to do the download, extract, scan, and update:

• fetch.fetch_package_url()
• scancode.extract_archives()
• scancode.get_resource_info()
• pipes.update_or_create_resource()

I am using the "codewaves/codehighlight/core" listed above as a sample:
The .class files from the "to" side are now correctly mapped

mapped via SHA-1 matches

What I’d Like to Discuss

I'm currently generating "virtual" resources in the project codebase to represent these dependencies. Specifically, each dependency package extracted from the pom.xml is placed alongside the original pom.xml, with a -dep suffix added to distinguish it

and I've added is_virtual: yes in the "Extra" field.

Note that the tree view will not have these newly added dependency shown:

So, I'd like to get some thoughts on whether this is a "workable" approach?

[chinyeungli]

Re the "virtual" resources, I didn't set to create the resources virtually, but this seems to be how update_or_create_resource() behaves, as opposed to collect_and_create_codebase_resources() that uses model_class.objects.bulk_create(batch, batch_size)