Skip to content

Conversation

@He-Pin
Copy link
Member

@He-Pin He-Pin commented Dec 22, 2024

extracted from #1621

Motivation:
In #1566 , @queimadus points out a leak

Modification:

  1. change the builder from a buffer to a vector builder.
  2. free that builder to null after the prefix is generated.

Result:
leak fixed.

import org.apache.pekko.actor.ActorSystem
import org.apache.pekko.stream.scaladsl.{Flow, Sink, Source}
import org.apache.pekko.util.ByteString

import scala.concurrent.Await


object PekkoQuickstart extends App {
  private implicit val system: ActorSystem = ActorSystem()

  val s = Source
    .repeat(())
    .map(_ => ByteString('a' * 400000))
    .take(1000000)
    .flatMapPrefix(50000) { _ => Flow[ByteString] }

  val r = Source.empty
    .concatAllLazy(List.tabulate(30000)(_ => s): _*)
    .runWith(Sink.ignore)

  Await.result(r, scala.concurrent.duration.Duration.Inf)
  println(r.value)

//  Source
//    .repeat(s)
//    .take(30000)
//    .flatMapConcat(x => x)
//    .runWith(Sink.ignore)
//    .onComplete(println(_))

//  Source.empty
//    .concatAllLazy(List.tabulate(30000)(_ => Source.lazySource(() => s)): _*)
//    .runWith(Sink.ignore).onComplete(println(_))
}

@He-Pin He-Pin added t:stream Pekko Streams bug Something isn't working backport labels Dec 22, 2024
@He-Pin He-Pin added this to the 1.1.3 milestone Dec 22, 2024
@He-Pin He-Pin changed the title chore: Fix leak in FlatMapPrefix operator. fix: Fix leak in FlatMapPrefix operator. Dec 22, 2024
@He-Pin
Copy link
Member Author

He-Pin commented Dec 28, 2024

@pjfanning Would you like to give this a review?

@pjfanning
Copy link
Member

@pjfanning Would you like to give this a review?

Would it be possible to unit test this? We probably don't want anything that takes too much time or memory to run but it might be feasible to add some extra test coverage.

@He-Pin He-Pin removed the backport label Jan 3, 2025
@He-Pin
Copy link
Member Author

He-Pin commented Jan 3, 2025

I think I can't test it because the builder is protected inside the logic, otherwise, I have to extract the code to a dedicated class and expose it to tests.

Another way is using the MBean, but that is too much.

@He-Pin He-Pin force-pushed the fixPrefixMap branch 2 times, most recently from 9373a45 to bc37570 Compare January 3, 2025 15:39
@He-Pin He-Pin requested a review from raboof January 3, 2025 15:47
@He-Pin He-Pin merged commit ce3620f into apache:main Jan 3, 2025
9 checks passed
@He-Pin He-Pin deleted the fixPrefixMap branch January 3, 2025 16:56
@He-Pin
Copy link
Member Author

He-Pin commented Jan 3, 2025

@queimadus Thanks for the reporting, this do spend me a weekend to work out:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working t:stream Pekko Streams

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants