The short & medium term future of library packaging in the tools.deps ecosystem

[RickMoynihan]

Hi @slipset I started writing a slack message to you, but figured I should probably open a public issue to discuss it instead. This isn't so much a single issue though, it involves how we should organise and arrange a variety of libraries in the tools.deps tools eco system to better support packaging and deploying libraries (and applications).

In the microcosm the problems are motivated by where to put the code in my PR #21. My feeling is that it doesn't currently belong in deps-deploy because the pom should really be created prior to it being packaged in the jar in the first place.

My initial reaction was then to push it into depstar, however after speaking to @seancorfield he thought that would be complecting things, and that he's essentially of the opinion that tools.build will eventually better solve these problems, so he's not willing to invest much time fixing things in this area.

My view here is slightly different to sean's in that depstar is already complected, but along the wrong axis. i.e. it complects uberjarring (an application concern) with library making, and that most of the constraints about how depstar needs to be managed are to ensure no 3rd party deps are included on the classpath when uberjarring.

So I think there are a few ways to try and resolve this:

1. Do the simple decomplected thing, and move my PR for updating the pom into a separate project. Users wishing to deploy libraries then need to complect (tie) the pom generation, library generation and deployment together themselves in their own code.
2. Do the easy complected thing that I suspect most of the community would like, which is to complect library-making concerns together, in a way that is untangled from uberjarring. i.e. create a new "lib-maker" project that can update a pom, generate a library jar containing the appropriate assets, and deploy it to either clojars or a private s3 bucket.

My feeling is that it might be best to persue 1 first and either leave enough signposts for others or our future selves to fill in the gaps and implement 2. Hopefully by then tools.build will be done, but if it's not the community might have a better option than the current state of affairs.

Anyway I thought I'd write this up incase you or others have useful things to say on the matter.

Many thanks.

[seancorfield]

It's a good time to update this issue: I've switched several of my OSS projects over to using tools.build and a build.clj file now -- see depstar, next.jdbc, and HoneySQL. The version is automatically generated from the commits and both <version> and <scm> / <tag> are updated automatically by tools.build as part of the JAR file creation. All three projects also use tools.build for their CI workflows now -- running various tests and then generating a pom.xml and a JAR file if they are successful.

I've enhanced depstar so it has tools.build-compatible, drop-in replacements for the jar and uber tasks. Given the functionality in tools.build for pom.xml and (library) JAR file handling, I'm probably going to deprecate that part of depstar now -- tools.build still has a ways to go before I can use its uber task, however. I'm probably going to update the app and lib templates in deps-new (and maybe clj-new as well) to add build.clj and use just tools.build for library JARs but use both tools.build and depstar for the application uberjar for the time being (I'm waiting a bit for adoption of the 1.10.3.933 CLI to increase). My goal for depstar is to deprecate it completely in favor of tools.build at some point.

Alex has talked about adding Clojars deployment to tools.build at some point. At that point, I'll probably switch to that and stop using deps-deploy -- and I'll update deps-new to do the same for new projects. I don't know how @slipset feels about the future of deps-deploy in a world where tools.build could do the same thing?

[slipset]

If clojars deployment is built into tools.build, I see no reason to continue with deps-deploy. And, just so there is no misunderstanding. I would welcome such a change.

[RickMoynihan]

I guess it may depend on whether or not tools.build supports important features of deps-deploy in particular private-S3-repositories

#17

Alternatively that could be a good candidate for a separate tool.

[seancorfield]

Another update is due at this point. I have archived depstar because tools.build has reached parity with it so I see no point in causing fragmentation by maintaining a project that compete with a core team project. Having switched my active OSS projects from depstar to tools.build, we have now switched our monorepo at work over to tools.build as well.

I've created build-clj as a library that wraps tools.build and codifies and expands on the examples in the official tools.build guide on clojure.org.

From discussions today on Slack, it seems very unlikely that tools.build is going to get deployment functionality (beyond local install to ~/.m2) -- so deps-deploy looks like being the "go to" choice for deployment for deps.edn projects for the foreseeable future.

write-pom in tools.build is a pretty good way to go from a template source pom.xml file to a fully-configured, fully-updated deployment pom.xml for use with deps-deploy and that's the approach I've taken in all my projects, via my build-clj library.

So I think my final position on the original proposal by @RickMoynihan at this point is:

• tools.build solves this problem and does so in a way that is compatible with deps-deploy,
• deps-deploy should not incorporate any functionality that tools.build already has.

[slipset]

Let's track this in discussion

[RickMoynihan]

Discussion moved to here: #44