[MDEPLOY-312] [REGRESSION] deploy no longer updates project model

[sos-jira-migration-bot]

Jared Stehler opened MDEPLOY-312 and commented

Prior to 3.0.0, the maven-deploy-plugin would update artifacts on the Project model:

• https://github.com/apache/maven/blob/master/maven-compat/src/main/java/org/apache/maven/artifact/deployer/DefaultArtifactDeployer.java#L147
• https://github.com/apache/maven-deploy-plugin/blob/maven-deploy-plugin-2.8.2/src/main/java/org/apache/maven/plugin/deploy/DeployFileMojo.java#L276

This is no longer occurring with the migration to maven-resolver, which is breaking our downstream plugins relying on the resolved SNAPSHOT version.

 

---

Affects: 3.0.0

Remote Links:

• GitHub Pull Request #44
  

[sos-jira-migration-bot]

Michael Osipov commented

Tamas Cservenak

[sos-jira-migration-bot]

Tamas Cservenak commented

What are you trying to resolve here? The PR is IMHO wrong, in a sense that it redirects project artifacts to non-existent artifacts (locally).

Again, what is the itch you want to solve here?

[sos-jira-migration-bot]

Jared Stehler commented

I need to know what the resolved snapshot version was for project artifacts which have been deployed to a remote repository during the build. We have a custom artifact resolver which relies on this information being properly published by builds.

The previous version of deploy-plugin did this by working directly with the Artifact instances from the project model, updating them with resolved version. Since moving to the aether deployer, this no longer occurs. I didn't see this change in behavior called out in PR comments, so it seems like an inadvertant change rather than explicit / desired.

Regarding the concern about pointing to non-existant local artifacts: the project model Artifact class has support for this via getBaseVersion().

[sos-jira-migration-bot]

Jared Stehler commented

I'm not necessarily tied to the idea of pushing back the resolved version into the project model; if we could somehow put it into metadata which I could access in a downstream plugin mojo that would work for me as well.

 

[sos-jira-migration-bot]

Tamas Cservenak commented

Those are not "resolved" but "transformed" (and by deploy, on purpose). Also, those (deployed, remote) files has nothing to do with Project ongoing. Moreover, in Maven4 within deployer is where consumer POM transformation happens, and deploy is on purpose encapsulated and "hidden" from everything else, unlike in old Maven2.

To me it looks like your "custom resolver" should rely on remote repository metadata, to properly resolve the freshly deployed artifacts? As to me it seems that currently it does not. Your logic relied on Maven2 behaviour (hence is/was in maven-compat), where this is not the case since Maven3 and introduction of Resolver...

Given this is totally orthogonal (at least to Maven core) and core decidedly has nothing to do with this, I'd rather recommend to listen for org.eclipse.aether.RepositoryEvent.EventType#ARTIFACT_DEPLOYED events, as those will carry the "transformed" (timestamped) version of artifact. Would that work for you?

[sos-jira-migration-bot]

Tamas Cservenak commented

Few pointers:

• Here is a resolver demo of deploying
• uses this RepositoryListener
• and produces console output like this (receives the transformed/timestamped version):

Deployed test:org.apache.maven.aether.examples:jar:0.1-20231013.161347-1 to org.apache.maven.aether.examples (file:/home/cstamas/Worx/apache-maven/maven-resolver/target/dist-repo, default, releases+snapshots)

[sos-jira-migration-bot]

Tamas Cservenak commented

Is by any chance your custom resolver open source, so I can take a peek at it?

[sos-jira-migration-bot]

Jared Stehler commented

Thanks for the pointers! I'd imagine that to use the listener approach, I'd need to call RepositorySystemSession.setRepositoryListener() in a mojo or extension prior to the deploy mojo execution, and that callback would happen within the context of the deploy mojo execution?

Unfortunately our resolver isn't open source, but operates like this:

 
The first part is the API. Every time a snapshot artifact is built, it reports to the API with the groupId, artifactId, base version, and the new resolved snapshot version. We keep track of the latest snapshot version for each group/artifact/version combo in a db table.

The client portion is implemented as a Maven extension (installed by dropping a JAR in $M2_HOME/lib). When you run Maven, this extension gets loaded and hits the API. It asks the API for any new snapshot versions since the last time it ran. For each new snapshot, it writes the new snapshot timestamp to a file inside the .m2 repo.

The extension also implements a custom update policy. Maven asks the update policy whether it needs to hit Nexus to check for a new snapshot version. To answer this question, the extension reads the aforementioned file containing the latest snapshot timestamp. It then loads the current maven-metadata.xml from disk and parses out the snapshot timestamp from there. If the timestamp is newer than the snapshot timestamp from the maven-metadata.xml, then there is a new snapshot version and it needs to hit Nexus to fetch it. Otherwise, we already have the latest snapshot and don't need to fetch anything.

[sos-jira-migration-bot]

Tamas Cservenak commented

Yes, all correct. You need to register your listener somehow, most recommended way is from an extension.

Re extensions: I hope you are aware that enlisting extensions in .mvn/extensions.xml is possible (then Maven resolves and uses enlisted extensions along with dependencies), and is actually much cleaner, than using "custom distributions" (or mutilating existing distributions by dropping stuff into /lib of Maven). Unless, there is some technical obstacle, not allowing you to use former way. See https://maven.apache.org/configure.html#mvn-extensions-xml-file

So, IMHO the "reporter" side could be a plain extension, that does just that: listens to deploy events, and reports those to API (maybe some config to narrow reported groupIds or alike, exclude them?)

This sounds interesting, too sad is not open....

[sos-jira-migration-bot]

Jared Stehler commented

The suggested approach seems like it will work well for us, thanks again!

I will look into utilizing .mvn/extensions more, appreciate that suggestion.

 

[sos-jira-migration-bot]

Jonathan Haber commented

There's a simple version that we open-sourced a while back available here:
https://github.com/HubSpot/maven-snapshot-accelerator

And accompanying blog post with more background:
https://product.hubspot.com/blog/speeding-up-maven-snapshots

Our internal version has picked up a variety of optimizations over time (For example, skipping updates if the commit sha hasn't changed. And also the API now generates and returns maven-metadata files directly, so once the build starts all of the updated Maven snapshot metadata has already been written directly into your local repo. And then the only thing we fetch from the remote repo are the transformed snapshot versions which are immutable artifacts and can be cached forever)

[sos-jira-migration-bot]

Tamas Cservenak commented

THIS is very cool! Thanks! Will dive into that a bit, as I was also tinkering about several things (one is mentioned on some other JIRA, basically stop using "snapshot normalization" [rename downloaded timestamped snapshot files into -SNAPSHOT files, effectively making all local repository content immutable]), and this "warming up" the local repository....

[sos-jira-migration-bot]

Tamas Cservenak commented

And use got relevant information how to do what they want to do.