Runaway memory use when consuming video in livekit-rtc

[tobowers]

Select which package(s) are affected

@livekit/rtc-node

Describe the bug

When consuming video through a video stream, the memory of the process grows unbounded. This happens simply by creating a video stream, whether you consume the VideoFrame or not.

Memory growth was gigabytes per minute with a 1920x1080 video stream (h264)

Reproduction

I tested this down to going into the livekit-rtc code in VideoStream and commenting out anything within OnEvent and the memory consumption still happens.

I also tested using the code as is and consuming every frame as fast as possible. Additionally tested by not consuming any frames at all.

Logs

System Info

Darwin, livekit-rtc: 0.13.8, bun 1.2.5

LiveKit server version

LiveKit cloud

Severity

blocking all usage of LiveKit

Additional Information

No response

[davidzhao]

thanks for the report. confirmed we are seeing this and will fix on our side

[Shubhrakanti]

Hey @tobowers this should be fixed by #448

[tobowers]

It is not fixed @davidzhao - I provided a minimal repro library in slack https://github.com/tobowers/minimal-livekit-memory-leak - you can use this with the sandbox video conferencing app. after just a minute or two (8000 frames) - I'm up to 7gb of memory usage (including calling Bun.gc every second or so)

[tobowers]

I probably wasn't clear enough above, but even if you completely comment out onEvent here

node-sdks/packages/livekit-rtc/src/video_stream.ts

Lines 58 to 93 in 61763fe

	if (
	ev.message.case != 'videoStreamEvent' ||
	ev.message.value.streamHandle != this.ffiHandle.handle
	) {
	return;
	}
	const streamEvent = ev.message.value.message;
	switch (streamEvent.case) {
	case 'frameReceived':
	const rotation = streamEvent.value.rotation;
	const timestampUs = streamEvent.value.timestampUs;
	const frame = VideoFrame.fromOwnedInfo(streamEvent.value.buffer!);
	const value = { rotation, timestampUs, frame };
	if (this.queueResolve) {
	this.queueResolve({
	done: false,
	value: {
	frame: value.frame,
	timestampUs: value.timestampUs!,
	rotation: value.rotation!,
	},
	});
	this.queueResolve = null;
	} else {
	this.eventQueue.push({
	frame: value.frame,
	timestampUs: value.timestampUs!,
	rotation: value.rotation!,
	});
	}
	break;
	case 'eos':
	FfiClient.instance.off(FfiClientEvent.FfiEvent, this.onEvent);
	break;
	}

(leave the function but make it a no-op) the memory keeps increasing. Makes me think maybe it's in rust or the ffi bridge?

[Shubhrakanti]

Hey @tobowers There was another issue casing the leak. #454 should fix it.

[Shubhrakanti]

(leave the function but make it a no-op) the memory keeps increasing. Makes me think maybe it's in rust or the ffi bridge?

Yes! Weren't closing the FfiHandle correctly after receiving the VideoFrame. I ran an example similar to your's where I just consumed a video stream from a LiveKit room. Memory hovered around 300MB ( I tested for 3-4 mins).

I probably wasn't clear enough above, but even if you completely comment out onEvent here

Hmm. that's helpful. Let me run https://github.com/tobowers/minimal-livekit-memory-leak just to make sure this fix works correctly.

[Shubhrakanti]

@tobowers I ran your code with with this log statement.

  for await (const frame of stream) {
    i++;
    if (i % 200 === 0) {
      // const webP = await convertFrameToWebP(frame.frame)
      // await Bun.write(`./frames/${i}.webp`, webP)
      // console.log("frame", i, frame.frame.data.length)
      console.log("RSS memory usage", process.memoryUsage().rss / 1024 / 1024, "MB", (Date.now() - start_time) / 1000, "seconds")
    }
  }

With the fix

subscribed to track TR_VCB6XYdcLerASv human 2
RSS memory usage 255.40625 MB 7.15 seconds
RSS memory usage 277.265625 MB 13.819 seconds
RSS memory usage 285.609375 MB 20.489 seconds
RSS memory usage 287.078125 MB 27.145 seconds
RSS memory usage 347.421875 MB 33.84 seconds
RSS memory usage 363.875 MB 40.506 seconds
RSS memory usage 377.296875 MB 47.226 seconds
RSS memory usage 382.65625 MB 53.883 seconds
RSS memory usage 382.796875 MB 60.551 seconds
RSS memory usage 382.859375 MB 67.225 seconds
RSS memory usage 382.90625 MB 73.901 seconds
RSS memory usage 390.984375 MB 80.55 seconds
RSS memory usage 391.015625 MB 87.229 seconds
RSS memory usage 395.15625 MB 93.887 seconds
RSS memory usage 396.578125 MB 100.557 seconds
RSS memory usage 399.3125 MB 107.274 seconds
RSS memory usage 399.34375 MB 113.933 seconds
RSS memory usage 399.359375 MB 120.607 seconds
RSS memory usage 399.40625 MB 127.267 seconds

Without the fix

subscribed to track TR_VCB6XYdcLerASv human 2
RSS memory usage 409.15625 MB 7.305 seconds
RSS memory usage 683.171875 MB 14.102 seconds
RSS memory usage 962.578125 MB 20.741 seconds
RSS memory usage 1228.828125 MB 27.392 seconds
RSS memory usage 1484.375 MB 34.09 seconds
RSS memory usage 1483.953125 MB 40.773 seconds
RSS memory usage 1375.71875 MB 47.434 seconds
RSS memory usage 1466.671875 MB 54.063 seconds
RSS memory usage 1641.765625 MB 60.715 seconds
RSS memory usage 1913.296875 MB 67.363 seconds
RSS memory usage 2052.734375 MB 74.154 seconds
RSS memory usage 2258.671875 MB 80.689 seconds
RSS memory usage 2271.046875 MB 87.347 seconds
RSS memory usage 1646.390625 MB 94.013 seconds
RSS memory usage 1685.53125 MB 100.676 seconds
RSS memory usage 1793.703125 MB 107.379 seconds
RSS memory usage 1976.40625 MB 114.044 seconds
RSS memory usage 2114.328125 MB 120.712 seconds
RSS memory usage 2379.25 MB 127.379 seconds

So I think we should be good to go here!