Skip to content

Commit fb082a0

Browse files
abestelAdrien Bestel
andauthored
Fix CallbackStack.Node leak (#1318)
Some context to start: Cats Effects has been having memory leaks in CallbackStack since version 3.4.3. See for example: typelevel/cats-effect#3935 I've been facing this memory leak in an application using fs2-kafka, and found that I'm not the only one (typelevel/cats-effect#3973). Using a simple consumer > produce stream application, I monitored the size of the CallbackStack using the following command: ``` while sleep 1; do jcmd <pid> GC.class_histogram | grep 'cats.effect.CallbackStack$Node' ; done ``` I found that swapping the `F.race(shutdown, fetch)` for `fetch` stops the memory leak. This should not be an issue because the Stream is anyway interrupted on `.interruptWhen(F.race(shutdown, stopReqs.get).void.attempt)`, but I'm not 100% convinced of this. Co-authored-by: Adrien Bestel <adrien.bestel@daimlertruck.com>
1 parent ed3d15e commit fb082a0

File tree

1 file changed

+7
-11
lines changed

1 file changed

+7
-11
lines changed

modules/core/src/main/scala/fs2/kafka/KafkaConsumer.scala

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ object KafkaConsumer {
170170
stopReqs <- Deferred[F, Unit]
171171
} yield Stream
172172
.eval {
173-
def fetchPartition: F[Unit] = F
173+
val fetchPartition: F[Unit] = F
174174
.deferred[PartitionResult]
175175
.flatMap { deferred =>
176176
val callback: PartitionResult => F[Unit] =
@@ -202,19 +202,15 @@ object KafkaConsumer {
202202

203203
assigned.ifM(storeFetch, completeRevoked)
204204
} >> deferred.get
205-
F.race(shutdown, fetch)
206-
.flatMap {
207-
case Left(()) =>
208-
stopReqs.complete(()).void
209205

210-
case Right((chunk, reason)) =>
211-
val enqueueChunk = chunks.offer(Some(chunk)).unlessA(chunk.isEmpty)
206+
fetch.flatMap { case (chunk, reason) =>
207+
val enqueueChunk = chunks.offer(Some(chunk)).unlessA(chunk.isEmpty)
212208

213-
val completeRevoked =
214-
stopReqs.complete(()).void.whenA(reason.topicPartitionRevoked)
209+
val completeRevoked =
210+
stopReqs.complete(()).void.whenA(reason.topicPartitionRevoked)
215211

216-
enqueueChunk >> completeRevoked
217-
}
212+
enqueueChunk >> completeRevoked
213+
}
218214
}
219215

220216
Stream

0 commit comments

Comments
 (0)