Advanced Topics

Build ImagePacket/GpuBufferPacket

Getting output from stream

To receive output from a running graph, you need to setup OutputStreamPoller or NativePacketCallback before calling CalculatorGraph#StartRun.
There are two main ways to do this.

Now suppose we want to get results from HolisticLandmarkCpu.

node {
  calculator: "HolisticLandmarkCpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "MODEL_COMPLEXITY:model_complexity"
  input_side_packet: "REFINE_FACE_LANDMARKS:refine_face_landmarks"
  output_stream: "POSE_LANDMARKS:pose_landmarks"
  output_stream: "WORLD_LANDMARKS:pose_world_landmarks"
  output_stream: "POSE_ROI:pose_roi"
  output_stream: "POSE_DETECTION:pose_detection"
  output_stream: "FACE_LANDMARKS:face_landmarks"
  output_stream: "LEFT_HAND_LANDMARKS:left_hand_landmarks"
  output_stream: "RIGHT_HAND_LANDMARKS:right_hand_landmarks"
}

Getting output synchronously

The first way is to use OutputStreamPoller.

Before running CalculatorGraph

var poseLandmarksStreamName = "pose_landmarks";
// NOTE: `POSE_LANDMARKS` type is `NormalizedLandmarkList`.
var poseLandmarksStreamPoler = calculatorGraph.AddOutputStreamPoller<NormalizedLandmarkList>(poseLandmarksStreamName).Value();

// You can poll multiple streams at the same time.
var poseRoiStreamName = "pose_roi";
// NOTE: `POSE_ROI` type if `NormalizedRect`;
var poseRoiStreamPoller = calculatorGraph.AddOutputStreamPoller<NormalizedRect>(poseRoiStreamName).Value();

If the output is empty (e.g. no person in the input image), MediaPipe does not send output packets to the stream by default. To get an empty packet when there's no output, specify observeTimestampBounds parameter (false by default).

🔔 Since there is no other way to determine if the output is empty, it is basically better to set it to true.

var poseLandmarksStreamName = "pose_landmarks";
var poseLandmarksStreamPoler = calculatorGraph.AddOutputStreamPoller<NormalizedLandmarkList>(poseLandmarksStreamName, true).Value();

After running CalculatorGraph

Send input packets to the input stream, and wait for the output.

ImageFramePacket inputPacket; // initialize it properly in the actual code
calculatorGraph.AddPacketToInputStream("input_video", inputPacket).AssertOk();

var poseLandmarksPacket = new NormalizedLandmarkListPacket();
if (poseLandmarksStreamPoller.Next(poseLandmarksPacket))
{
    if (poseLandmarksPacket.IsEmpty()) // when `observeTimestampBounds` is `true`, output packet can be empty
    {
        var poseLandmarks = poseLandmarksPacket.Get();
        // Do something
    }
}

🔔 OutputStreamPoller#Next will block the thread.
This means if you've not set observeOutputStream to true and the output stream becomes empty, the whole application will hang.

Getting output asynchronously

The other way is to use callbacks.

To support platforms that require AOT compilation or IL2CPP (e.g. iOS), you need to use lower level APIs.

Before running CalculatorGraph

First, you need to define the callback that you want to be called by MediaPipe.

⚠️ The critical restriction here is that the callback must be a static method.

[AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
private static IntPtr PoseLandmarksCallback(IntPtr graphPtr, IntPtr packetPtr)
{
    try
    {
        using (var packet = new NormalizedLandmarkListPacket(packetPtr, false)) // `isOwner` (the second argument) must be `false`
        {
            if (!packet.IsEmpty()) // when `observeTimestampBounds` is `true`, output packet can be empty
            {
                var output = packet.Get();
                // Do something
            }
        }
        // Return a pointer to the result status
        return Status.Ok().mpPtr;
    }
    catch (Exception e)
    {
        // Return a pointer to the result status
        return Status.FailedPrecondition(e.ToString()).mpPtr;
    }
}

The callback will receive 2 IntPtrs, the first is the pointer to the running CalculatorGraph and the second is the pointer to the output packet.

To get the output, you need to initialize an output packet from the pointer.

🔔 The output packet is managed by C++ and you don't own it.

// `isOwner` (the second argument) must be `false`
using (var packet = new NormalizedLandmarkListPacket(packetPtr, false)) {...}

Which method to use?

Adding Custom Calculators

Adding Custom APIs

Use Forked MediaPipe Repository