Add support for Coursier's --extra-jars option

[blmeadows]

Hey @alejandrohdezma, I hope you're well!

I am using scala-steward-action for my organization's private repositories, which use sbt-gcs-plugin for resolving private dependencies stored in Google Cloud. The sbt-gcs-plugin is added in the meta build of most repos, and an internal sbt plugin is added to the build itself. During the action job run, both the sbt-gcs-plugin dependency in the meta build and the internal dependencies in the build seem to be ignored, with no errors or logs. Any public dependencies are updated without issue. I asked if anyone dealt with this in the tooling channel of the Scala Discord and came to the conclusion with @fthomas's help that being able to set the Coursier's extra-jars option may get me on the right track (at least until if/when the other additional protocol handler issues are complete).

[alejandrohdezma]

Hey @blmeadows! Nice to see you!

I created a PR for adding the new option, you should be able to test it with this version:

uses: scala-steward-org/scala-steward-action@snapshots/467

[fthomas]

My thinking here is that the Coursier that is used by Scala Steward needs to be aware of a protocol handler for GCS and one way to make it aware of it is to add a JAR to Scala Steward's classpath that contains this protocol handler (see https://get-coursier.io/docs/extra.html#extra-protocols and scala-steward-org/scala-steward#2628) and support for --extra-jars could help with that.

What is currently not clear to me is how the additional JAR is downloaded to the action's runner. Can users of the action run cs fetch --classpath "io.github.987nabil::gar-coursier:1.1" for example before the action runs and use its output as input for the action's extra-args?

[alejandrohdezma]

Yeah, from what I got from Coursier's internal code you need to provide a local directory or jar file, so yes, I suppose doing cs fetch and pipings its output to extra-jars will do the trick.

[blmeadows]

Hmm, I ran cs fetch --classpath --scala-version 2.12 --sbt-version 1.0 --sbt-plugin org.latestbit:sbt-gcs-plugin:1.6.1 and set the output to extra-jars but don't see any change in behavior (no additional logs or errors and still not picking up those dependencies). (Our dependencies are in Google Cloud Storage instead of Artifact Registry, which is why I can't use gar-coursier.)

- name: Fetching sbt-gcs-plugin classpath
  id: fetch_classpath
  run: echo "classpath=$(cs fetch --classpath --scala-version 2.12 --sbt-version 1.0 --sbt-plugin org.latestbit:sbt-gcs-plugin:1.6.1)" >> $GITHUB_OUTPUT

with extra-jars: ${{ steps.fetch_classpath.outputs.classpath }}

If it's helpful, ${{ steps.fetch_classpath.outputs.classpath }} is:

/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/latestbit/sbt-gcs-plugin_2.12_1.0/1.6.1/sbt-gcs-plugin-1.6.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.12.15/scala-compiler-2.12.15.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.15/scala-library-2.12.15.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12.15/scala-reflect-2.12.15.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/cloud/google-cloud-storage/2.4.1/google-cloud-storage-2.4.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/guava/guava/31.0.1-jre/guava-31.0.1-jre.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/checkerframework/checker-qual/3.21.1/checker-qual-3.21.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/http-client/google-http-client/1.41.2/google-http-client-1.41.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/io/opencensus/opencensus-contrib-http-util/0.31.0/opencensus-contrib-http-util-0.31.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/http-client/google-http-client-jackson2/1.41.2/google-http-client-jackson2-1.41.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api-client/google-api-client/1.33.1/google-api-client-1.33.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/oauth-client/google-oauth-client/1.33.0/google-oauth-client-1.33.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/http-client/google-http-client-gson/1.41.2/google-http-client-gson-1.41.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/http-client/google-http-client-apache-v2/1.41.2/google-http-client-apache-v2-1.41.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/apis/google-api-services-storage/v1-rev20211201-1.32.1/google-api-services-storage-v1-rev20211201-1.32.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/code/gson/gson/2.8.9/gson-2.8.9.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/cloud/google-cloud-core/2.4.0/google-cloud-core-2.4.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/auto/value/auto-value-annotations/1.9/auto-value-annotations-1.9.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api/grpc/proto-google-common-protos/2.7.2/proto-google-common-protos-2.7.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/cloud/google-cloud-core-http/2.4.0/google-cloud-core-http-2.4.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/http-client/google-http-client-appengine/1.41.2/google-http-client-appengine-1.41.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api/gax-httpjson/0.96.0/gax-httpjson-0.96.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api/gax/2.11.0/gax-2.11.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/auth/google-auth-library-credentials/1.4.0/google-auth-library-credentials-1.4.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/auth/google-auth-library-oauth2-http/1.4.0/google-auth-library-oauth2-http-1.4.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api/api-common/2.1.3/api-common-2.1.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/io/opencensus/opencensus-api/0.31.0/opencensus-api-0.31.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/io/grpc/grpc-context/1.44.0/grpc-context-1.44.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/api/grpc/proto-google-iam-v1/1.2.1/proto-google-iam-v1-1.2.1.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.19.3/protobuf-java-3.19.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/protobuf/protobuf-java-util/3.19.3/protobuf-java-util-3.19.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/threeten/threetenbp/1.5.2/threetenbp-1.5.2.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.13.1/jackson-core-2.13.1.jar

[alejandrohdezma]

🤔 I don't think the extra-jars option would work for sbt-gcs-plugin, right @fthomas?

[fthomas]

I don't know.

There are two things I would try:

1. Set the cache-ttl option to 0minute so that Scala Steward will look for new versions.
2. Don't use the --classpath option in the cs fetch invocation. Maybe the --extra-jars option doesn't work with a colon separated list.

[fthomas]

I don't think the extra-jars option would work for sbt-gcs-plugin, right @fthomas?

Yeah, I think this is correct. According to the Coursier docs, there must be a URLStreamHandlerFactory implementation under the coursier.cache.protocol package and I don't see that anywhere in sbt-gcs-plugin.

[blmeadows]

🤔 hmm, this plugin has been in place for our builds for about a year with Coursier enabled and the plugin and resolution of internal dependencies have been working as expected. I tried setting cache-ttl to 0minute with no luck. The --extra-jars option expects List[String] and the action is accepting NonEmptyString, maybe that's an issue?

Otherwise, I'll need to check if it's desired for me to dig into this further (by either researching how this works internally without issue or a possible gar-coursier alternative for cloud storage).

[fthomas]

this plugin has been in place for our builds for about a year with Coursier enabled and the plugin and resolution of internal dependencies have been working as expected.

It seems to me that as sbt plugin it has other options to provide a URLStreamHandlerFactory implementation to Coursier in sbt: https://github.com/abdolence/sbt-gcs-resolver/blob/1ed51561d77101bf8d43f5ac5d42b69034167e66/sbt-gcs-plugin/src/main/scala/org/latestbit/sbt/gcs/GcsUrlHandlerFactory.scala#L36-L72

[alejandrohdezma]

🤔 Would it help if you just try to install the sbt-gcs-plugin in a previous step before running Scala Steward? Maybe I'm not completely understanding the reason why SS is not able to "see" those dependencies

[fthomas]

My understanding why Scala Steward does not update dependencies in GCS is the following:

1. Scala Steward can correctly extract dependencies and resolvers (even the GCS resolvers which I guess are MavenRepositorys) from sbt builds and these infos are stored in $workspace/store/repo_cache/v1/github/$group/$repo/repo_cache.json files.
2. Scala Steward then tries to find new versions of these dependencies using its VersionCache which uses Coursier under the hood. The resolvers that have been extracted in step 1 are used for that.
3. The VersionCache caches its results to files like $workspace/store/versions/v2/https/repo1.maven.org/maven2/org/typelevel/cats-core_2.13/versions.json and there are also version.json files for the dependencies that are only available in the GCS resolver. But these files do not contain the versions but an error message which indicates that Coursier was not able to retrieve versions for this dependency and resolver combination because Coursier in Scala Steward is not aware of a protocol handler for GCS resolvers. It cannot reuse any protocol handler that sbt-gcs-plugin installed in the sbt build before because Scala Steward and the sbt build are completely different processes.
4. Adding additional JARs via --extra-jar would enable Coursier in Scala Steward to also retrieve versions from resolvers which are not supported by vanilla Coursier. I believe for example that io.github.987nabil::gar-coursier:1.1 as additional JAR would enable Scala Steward to also retrieve the versions from artifactregistry:// resolvers.

[blmeadows]

I tried before adding sbt-gcs-plugin as a global plugin as a previous step with no luck. I will try to make a gar-coursier alternative for gs:// resolvers if/when I get internal time to do so and then give that a try. Thank you!

[987Nabil]

Hey! I am the author of gar-coursier and it currently only supports artifactregistry:// not gs://. Tho it should be possible to make it work.
I also know sbt-gcs-plugin quite well and iirc, it does not support coursiers protocol extension. It only works with sbt.