Add support to mine maven Package-URLs #660 #678

JonoYang · 2025-09-01T21:30:26Z

References:

Signed-off-by: Jono Yang <[email protected]>

* Create basic work loop Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

JonoYang · 2025-09-04T02:14:07Z

minecode_pipeline/pipes/__init__.py

        f.write(saneyaml.dump(data))
+
+
+def write_purls_to_repo(repo, package, packages, push_commit=False):


@AyanSinhaMahapatra I've moved your write_packageurls_to_file and write_purls_to_repo functions to this file and I see that we have overlapping functionality here, especially with the code in https://github.com/aboutcode-org/purldb/blob/main/minecode_pipeline/pipes/pypi.py#L135

I think we should use a common function to do these actions

Signed-off-by: Jono Yang <[email protected]>

ziadhany · 2025-09-05T14:43:42Z

minecode_pipeline/pipes/__init__.py


 def write_packageurls_to_file(repo, base_dir, packageurls):
-    purl_file_rel_path = os.path.join(base_dir, PURLS_FILENAME)
+    purl_file_rel_path = os.path.join(base_dir, hashid.PURLS_FILENAME)


@JonoYang @AyanSinhaMahapatra I think we should avoid computing the package URL path and just use this function. ppath = hashid.get_package_purls_yml_file_path(package)

Signed-off-by: Jono Yang <[email protected]>

* leading segment will be the repo itself, not a directory within the repo Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

* Create function to get checkpoint from checked out settings repo Signed-off-by: Jono Yang <[email protected]>

…uctor #660 Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

* This is done in the collect_packages_from_maven step Signed-off-by: Jono Yang <[email protected]>

Signed-off-by: Jono Yang <[email protected]>

AyanSinhaMahapatra

@JonoYang thanks++, LGTM!

A couple of minor comments for your consideration.
I've also enabled trusted publishing, and created the repos, we can start testing this in the CI.

minecode_pipelines/pipelines/mine_maven.py

minecode_pipelines/pipes/maven.py

pyproject-minecode_pipeline.toml

Signed-off-by: Jono Yang <[email protected]>

JonoYang added 6 commits August 25, 2025 16:43

Create minecode_pipeline module and move relevent code over #660

b12150b

Signed-off-by: Jono Yang <[email protected]>

Move maven crawler code into new module #660

e4fdae9

* Create basic work loop Signed-off-by: Jono Yang <[email protected]>

Yield packages from index, collect, and write to file #660

8f1f9ac

Signed-off-by: Jono Yang <[email protected]>

Push commits at regular intervals

837d0d1

Signed-off-by: Jono Yang <[email protected]>

Use new branch of scio #660

b92a8bd

Signed-off-by: Jono Yang <[email protected]>

Create write_purls_to_repo #660

6b3372d

Signed-off-by: Jono Yang <[email protected]>

pombredanne changed the title ~~660 purl next maven~~ Add support to mine maven Package-URLs #660 Sep 2, 2025

pombredanne mentioned this pull request Sep 2, 2025

purl-next: Collect PURLs for Rust ecosystem #666

Closed

JonoYang added 2 commits September 3, 2025 16:56

Merge branch 'main' into 660-purl-next-maven

4ff859b

Merge branch 'main' into 660-purl-next-maven

44f4363

JonoYang commented Sep 4, 2025

View reviewed changes

Clone repo per package #660

b0a3d6e

Signed-off-by: Jono Yang <[email protected]>

ziadhany reviewed Sep 5, 2025

View reviewed changes

JonoYang added 9 commits September 5, 2025 16:37

Add methods to fetch maven index properties and increments #660

d9c8b8d

Signed-off-by: Jono Yang <[email protected]>

Merge branch 'main' into 660-purl-next-maven

ac7f6d8

Move files to minecode_pipelines #660

1bfb2d8

Signed-off-by: Jono Yang <[email protected]>

Move tests over to minecode_pipelines #660

dda2bca

Signed-off-by: Jono Yang <[email protected]>

Create purls.yml path without leading segment #660

74bd21c

* leading segment will be the repo itself, not a directory within the repo Signed-off-by: Jono Yang <[email protected]>

Use last incremental from index properties #660

810956f

Signed-off-by: Jono Yang <[email protected]>

Move code to proper place #660

0d441f9

Signed-off-by: Jono Yang <[email protected]>

Merge branch 'main' into 660-purl-next-maven

e5ed41a

Use new common functions for writing and commiting purls #660

eb4a4fb

* Create function to get checkpoint from checked out settings repo Signed-off-by: Jono Yang <[email protected]>

This was referenced Sep 17, 2025

purl-next: Collect PURLs - 2 - package collector for deployment #657

Closed

purl-next: Collect PURLs - 3 - collect pilot ecosystem #658

Closed

Handle downloading and deleting index archive using constructor/destr…

289c336

…uctor #660 Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the 660-purl-next-maven branch from 1e044b5 to 289c336 Compare September 18, 2025 19:58

JonoYang added 4 commits September 18, 2025 15:15

Split repo deletion into separate pipeline step #660

88956bf

Signed-off-by: Jono Yang <[email protected]>

Fix logic in destructor #660

9449ee2

Signed-off-by: Jono Yang <[email protected]>

Merge branch 'main' into 660-purl-next-maven

a35a351

Declare mine_maven pipeline in pyproject #660

81b1243

Signed-off-by: Jono Yang <[email protected]>

JonoYang added 4 commits September 19, 2025 14:29

Remove repo clone step #660

a225502

* This is done in the collect_packages_from_maven step Signed-off-by: Jono Yang <[email protected]>

Update code style #660

d126ea0

Signed-off-by: Jono Yang <[email protected]>

Revert change to scio dep version #660

113bce5

Signed-off-by: Jono Yang <[email protected]>

Remove unnecessary logger #660

78045ef

Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the 660-purl-next-maven branch from de1526c to 78045ef Compare September 19, 2025 22:02

JonoYang added 2 commits September 19, 2025 17:36

Clean up all downloads in destructor #660

644fbc0

Signed-off-by: Jono Yang <[email protected]>

Update docstrings #660

77295a3

Signed-off-by: Jono Yang <[email protected]>

JonoYang force-pushed the 660-purl-next-maven branch from e12a084 to 77295a3 Compare September 20, 2025 01:00

AyanSinhaMahapatra approved these changes Sep 23, 2025

View reviewed changes

minecode_pipelines/pipelines/mine_maven.py Show resolved Hide resolved

minecode_pipelines/pipes/maven.py Show resolved Hide resolved

pyproject-minecode_pipeline.toml Outdated Show resolved Hide resolved

JonoYang added 5 commits September 23, 2025 13:07

Simplify download tracking #660

bc240f2

Signed-off-by: Jono Yang <[email protected]>

Update MinePypi pipeline name

fabe276

Signed-off-by: Jono Yang <[email protected]>

Update pipeline #660

0b36220

Signed-off-by: Jono Yang <[email protected]>

Update expected test result

16adc5c

Signed-off-by: Jono Yang <[email protected]>

Update package batch size #660

ac3032a

Signed-off-by: Jono Yang <[email protected]>

JonoYang merged commit 3428caf into main Sep 24, 2025
6 checks passed

JonoYang deleted the 660-purl-next-maven branch September 24, 2025 01:03

JonoYang mentioned this pull request Oct 2, 2025

purl-next: Collect PURLs for Maven ecosystem #660

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add support to mine maven Package-URLs #660 #678

Add support to mine maven Package-URLs #660 #678

Uh oh!

JonoYang commented Sep 1, 2025 •

edited by pombredanne

Loading

Uh oh!

JonoYang Sep 4, 2025

Uh oh!

ziadhany Sep 5, 2025

Uh oh!

AyanSinhaMahapatra left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		f.write(saneyaml.dump(data))


		def write_purls_to_repo(repo, package, packages, push_commit=False):

Uh oh!

Add support to mine maven Package-URLs #660 #678

Add support to mine maven Package-URLs #660 #678

Uh oh!

Conversation

JonoYang commented Sep 1, 2025 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonoYang Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ziadhany Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

AyanSinhaMahapatra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JonoYang commented Sep 1, 2025 •

edited by pombredanne

Loading