How to capture screenshots and encapsulate H264 streams in a Node.js?

[luckyyyyy]

Hello, I'm not very familiar with video formats. I created a simple demo and can now obtain H264 information, but I'm facing two issues:

1. How can I get the latest screenshot from an H264 stream? Since I need to fetch it frequently, calling ffmpeg all the time is not efficient. However, I haven't found any libraries in Node.js that can convert H264 streams to images, and this kind of requirement seems quite niche in Node.js. Should I wrap a library like video-rs using NAPI to achieve this?
2. How can I simultaneously encapsulate H264 into MKV? Should I start a TCP server and use FFmpeg for this?

import { ScrcpyOptions3_1, h264SearchConfiguration } from '@yume-chan/scrcpy';
import net from 'net';
import fs from 'fs';
import path from 'path';

function createTCPStream(port = 27183, host = 'localhost') {
  const client = new net.Socket();

  return new ReadableStream({
    start(controller) {
      client.connect(port, host);
      let firstChunk = false;
      client.on('data', (data) => {
        if (!firstChunk) {
          firstChunk = true;
        } else {
          controller.enqueue(data);
        }
      });

      client.on('end', () => controller.close());
      client.on('error', (err) => controller.error(err));
    },
    cancel() {
      client.destroy();
    },
  });
}

async function main() {
  const outputDir = path.join(process.cwd());
  if (!fs.existsSync(outputDir)) {
    fs.mkdirSync(outputDir);
  }

  const outputPath = path.join(outputDir, 'output.h264');
  const writeStream = fs.createWriteStream(outputPath);
  const options = new ScrcpyOptions3_1({
    tunnelForward: true,
    cleanup: false,
    control: false,
    video: true,
    audio: false,
    sendFrameMeta: true,
    videoCodec: 'h264',
  });

  const videoSocket: ReadableStream<Uint8Array> = createTCPStream();

  const commandLineArguments: string[] = options.serialize();
  console.log('Command line arguments:', commandLineArguments);

  const { metadata: videoMetadata, stream: videoStream } = await options.parseVideoStreamMetadata(videoSocket);
  console.log('Video metadata:', videoMetadata);

  const videoPacketStream = videoStream.pipeThrough(
    options.createMediaStreamTransformer(),
  );

  let configuration: Uint8Array | undefined;
  let frameCount = 0;

  try {
    for await (const packet of videoPacketStream) {
      if (packet.type === 'configuration') {
        configuration = packet.data;
        console.log('Received configuration packet');

        try {
          const { sequenceParameterSet, pictureParameterSet } = h264SearchConfiguration(packet.data);
          console.log('SPS length:', sequenceParameterSet.byteLength);
          console.log('PPS length:', pictureParameterSet.byteLength);
        } catch (e) {
          console.error('Failed to extract SPS/PPS:', e);
        }

        continue;
      }

      if (packet.type === 'data') {
        let buffer: Uint8Array;
        if (configuration) {
          buffer = new Uint8Array(configuration.byteLength + packet.data.byteLength);
          buffer.set(configuration);
          buffer.set(packet.data, configuration.byteLength);
          configuration = undefined;
        } else {
          buffer = packet.data;
        }

        writeStream.write(Buffer.from(buffer));
        frameCount++;

        if (frameCount % 100 === 0) {
          console.log(`Processed ${frameCount} frames`);
        }
      }
    }
  } catch (error) {
    console.error('Error processing video stream:', error);
  } finally {
    writeStream.end();
    console.log(`Recording completed. Total frames: ${frameCount}`);
    console.log(`Output saved to: ${outputPath}`);
  }
}
main().catch(console.error);

[luckyyyyy]

At the same time, I found that during the first three communications of Scrcpy, the first byte sent is 00, followed by 64+12 bytes of device information, video, and resolution information. However, if I don't actively filter out the first byte, it will cause an error in parsing the video stream metadata.

[yume-chan]

How can I get the latest screenshot from an H264 stream?

You must decode the stream using a decoder. You need to save the latest configuration packet, and all data packets starting from the latest keyframe, to decode any frame.

I haven't found any libraries in Node.js that can convert H264 streams to images

I found https://github.com/Streampunk/beamcoder (seems abandoned) and https://github.com/mmomtchev/ffmpeg (pretty new). They use n-api to call FFmpeg APIs.

Using a WebAssemly port of FFmpeg might be another option, the performance might not be as good as native FFmpeg libraries.

How can I simultaneously encapsulate H264 into MKV?

You can use webm-muxer package. WebM is a subset of Matroska (MKV), and this package has a type option to produce MKV compatible files.

An example can be found in old-demo: https://github.com/tango-adb/old-demo/blob/aab87732cc685286e4c62791b04256ea4a464422/packages/demo/src/components/scrcpy/recorder.ts#L231. Some API might have changed since it was last updated.

This implementation saves latest configuration packet and all data packets starting from the latest keyframe, like I described above, to be able to start recording from any time. The downside is, it will include several extra frames at the beginning. My later version uses resetVideo command to produce a new configuration packet and a keyframe immediately, then start recording from that.

Of course you can also use FFmpeg API. However it can't be done using FFmpeg command line, because the H.264 stream doesn't contain timestamps: Genymobile/scrcpy#5742 (comment)

if I don't actively filter out the first byte, it will cause an error in parsing the video stream metadata.

You are correct. Theoretically speaking, the first 0x00 byte is sent to the first socket, so for example if video is disabled, this byte will be sent to the audio socket instead, so it's not part of the video stream. But in practice video socket is always the first socket (if not disabled) so this byte will be in there. I will mention it in the docs.

[luckyyyyy]

Thank you for your help. As you mentioned, I found the project at https://github.com/mmomtchev/ffmpeg to be amazing. Someone has already done what I want to do. I need to decode H.264 to get the latest screenshots, possibly taking 5-10 screenshots per second to capture the most recent frames. This is what I currently need, but I need to learn some keyframe-related concepts.

For packaging with MKV, using FFmpeg's command line is definitely simpler for me. I've already noticed the issue with timestamps in H.264, as mentioned in the comments. The situation improves after using the -use_wallclock_as_timestamps 1 parameter.