Be aware of Cats-Effect depending on Cats-Core 2.9.0

[danicheg]

Saying this straight off the bat, this is not a Redis4Cats issue! However, IMHO, some awareness (read better visibility) for the issue I encountered should be raised.
TL;DR, if a project lacks the explicit dependency on Cats-Core but has both Cats-Effect (v3.5.7) and Redis4Cats (v1.7.2) — an NSME will be thrown:

[error] java.lang.NoSuchMethodError: 'java.lang.Object cats.syntax.ApplyOps$.$times$greater$extension(java.lang.Object, java.lang.Object, cats.Apply)'
[error] 	at dev.profunktor.redis4cats.connection.RedisClient$.$anonfun$acquireAndRelease$4(RedisClient.scala:45)
[error] 	at cats.effect.kernel.Resource$.$anonfun$make$1(Resource.scala:848)
[error] 	at cats.effect.IOFiber.runLoop(IOFiber.scala:343)
[error] 	at cats.effect.IOFiber.execR(IOFiber.scala:1362)
[error] 	at cats.effect.IOFiber.run(IOFiber.scala:112)
[error] 	at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:634)

That's unpleasant but until Cats-Effect is not on the current version of Cats-Core, the only possible solution is to have Cats-Core defined explicitly among the dependencies in a project. Again, there is nothing to be done on the Redis4Cats side.

See the complete minimal representation here https://github.com/danicheg/redis4cats-nsme-showcase.

Resolution

Ultimately, as @kubukoz explored and confirmed, this was a redis4cats omission. It was addressed in #956.

[danicheg]

Linking the blocker for the update Cats-Core to 2.12.0 on the Cats-Effect side for interested readers. typelevel/cats-effect#4076

UPD. As we discussed in the GitHub issue above, this is much more likely not caused by the Cats-Effect. :sad_pepe: Anyway, nothing else in the build brings a different version of Cats-Core causing the NSME.

[kubukoz]

@danicheg do you have a minimal repro for this issue? redis4cats 1.7.2 seems to depend on that exact CE version (3.5.7) so I don't see how it'd have any incompatibilities at all.

redis4cats does have a dependency on cats-laws and testkit 2.12.0 in its tests:

• https://github.com/profunktor/redis4cats/blob/v1.7.2/project/Dependencies.scala#L7
• https://github.com/profunktor/redis4cats/blob/v1.7.2/project/Dependencies.scala#L48-L49
• https://github.com/profunktor/redis4cats/blob/v1.7.2/build.sbt#L49-L50

but they're not part of any modules published by redis4cats, as far as I can tell.

[danicheg]

Yep, I added it here https://github.com/danicheg/redis4cats-nsme-showcase

[danicheg]

By the way, the issue with NSME would disappear if Cats-Effect updated Cats-Core to v2.10.0 (although the current version is v2.12.0), as it's published for the SN 0.4.

[kubukoz]

The repro works, but I'm still baffled that it happens when both libraries you're pulling in depend on 2.9.0...

[kubukoz]

ok I think I'm onto something. This is definitely a problem with how redis4cats was published, i.e. we can possibly shift the blame to the publishing side of sbt 😅

I downgraded all the transitive dependencies of all modules in redis4cats to use Cats 2.9 at the latest: https://github.com/profunktor/redis4cats/tree/downgraded-to-cats-29 and when I use that as a local snapshot, your repro no longer crashes.

Without this change, the build contains multiple versions of Cats:

[sbt] redis4cats  λ whatDependsOn org.typelevel cats-core_2.13
cats-core_2.13 2.10.0   cats-core_2.13 2.11.0   cats-core_2.13 2.12.0   cats-core_2.13 2.9.0    cats-core_2.13;

However, redis4cats-effects only depends on 2.9.0:

[sbt] redis4cats  λ redis4cats-effects/whatDependsOn org.typelevel cats-core_2.13
cats-core_2.13 2.9.0   cats-core_2.13;

I think sbt is still publishing against 2.9, but it's compiling and testing against 2.12, thus using the a method only present on 2.10 and above. Might be some optimization on sbt's side hitting an unhappy path...

Here's the difference in bytecode: LHS is 1.7.2-SNAPSHOT (made from the branch linked above), RHS is locally published 1.7.2.

The latter refers to the method in question, the former doesn't.

BTW there is a mention of this syntax method in the release notes for 2.10.0:

This release includes typelevel/cats#4477 which fixes the encoding of Apply (and Semigroupal) syntax to avoid allocating an intermediate "Ops" wrapper class. This affected commonly-used operators such as *>. Unfortunately libraries will need to re-compile and re-publish against Cats v2.10.0 to use the new allocation-free syntax.

All in all, I think redis4cats should add an explicit dependency on Cats Core to its core & effects modules, but the behavior of (probably sbt) compiling against a different version than what's published is still suspicious to me and probably should be investigated further.

[danicheg]

Nice investigation, @kubukoz. It would be great if you could create a discussion or issue in the SBT repository so we can get an answer from the SBT maintainers.

All in all, I think redis4cats should add an explicit dependency on Cats Core

That's why I raised my concerns about coordinating Cats-Core updates across the ecosystem in the Cats-Effect repository. That's quite bizarre that the most recently published Cats-Effect version (a month ago!) depends on the Cats-Core published ... two years ago. Perhaps Cats-Core shouldn't evolve in a way that results in Cats-Effect using a stale Cats-Core version for two years.

[kubukoz]

Looks like it was our fault all along.

I minimized the issue to this:

ThisBuild / scalaVersion := "2.13.15"
ThisBuild / organization := "com.kubukoz.demo"

val demo = project
  .settings(
    libraryDependencies ++= Seq(
      "org.typelevel" %% "cats-effect-kernel" % "3.5.7",
      "org.typelevel" %% "keypool" % "0.4.10" % Optional,
    )
  )

import cats.syntax.all._

object Demo {
  def main(args: Array[String]): Unit = println(List(1) *> List(1))
}

(available here: https://github.com/kubukoz/demos/tree/redis4cats-nsme)

Run publishLocal, then:

cs launch com.kubukoz.demo:demo_2.13:0.1.0-SNAPSHOT
Exception in thread "main" java.lang.NoSuchMethodError: 'java.lang.Object cats.syntax.ApplyOps$.$times$greater$extension(java.lang.Object, java.lang.Object, cats.Apply)'
        at Demo$.main(Demo.scala:4)
        at Demo.main(Demo.scala)

So it looks like it was the Optional scope, all along. If you add the desired keypool dependency (at least 0.4.10), it doesn't crash anymore.

At this point I don't think it's an issue in sbt, it's just our misuse of it - I don't see a good reason why we'd have any % Optional or % Runtime dependencies.

[danicheg]

Nicely done @kubukoz

[danicheg]

@armanbilge hey Arman, you asked the place to dig into the issue. Here it is.

[armanbilge]

Thanks! I thought that this is resolved? Thanks to Jakub's investigation. Is there still a problem even after redis4cats fixes itself?

[kubukoz]

After removing % Optional you can set 2.9.0 explicitly but you'll still get evicted to (at least) 2.11 because that's what the now-not-optional dependency brings.

The NSME happened because r4c compiled against 2.11 and published against 2.9, which won't happen now (both will use 2.11)

[kubukoz]

if we didn't need keypool at all, we could remove it and we'd still be compiling&publishing against 2.9, using the deprecated method, but binary compatibility of Cats would keep things working. That is to say, I don't see a flaw on Cats's side