Auxiliary model license clarification

[Paladinium]

Looking at the models used for ditto-talkinghead with regards to License usage, some models likely prohibit commercial use such as:

• 2d106det.onnx and det_10g.onnx: Given the python classes insightface_det.py and insightface_landmark106.py, these models do not allow commercial use as noted here
• landmark203.onnx: the origin of this model is totally unknown - hence the license is unknown too.

What would really help is:

• Regarding the InsightFace models: do you have plans to replace them? Or can you at least make a recommendation what could potentially be a suitable replacement?
• Regarding Landmark203: could you clarify where the model originated from and what its license is?

Thanks.

[RobinROAR]

Thank you for your interest：）

But I'm sorry，because currently there is no effective way to circumvent this limitation, as the motion representation we use comes from Liscense and must adhere to its requirements. A potential workaround is to train a separate face reenactment module independently, and then train Ditto on top of that representation.

[Paladinium]

@RobinROAR : Thanks for your quick reply. What do you mean by "motion representation we use comes from Liscense"?

The models I was referring to were:

• 2d106det.onnx: is a face detector which returns bbox + score (+ usually 5 keypoints)
• det_10g.onnx: is a landmark regressor → returns (106,2) landmarks in image coordinates

So I don't understand what you are referring to when you mention "motion representation" as well as "train a separate face reenactment module"?

Just for the 2 models, potential alternatives could be e.g.:

• As face detector: OpenCV YuNet
• Landmark regressor: MediaPipe Face Mesh / Face Landmarker with an adapater to only pick 106 out of 468 points

Of course, it would take some time to get it work, but I only ask about the general feasibility.

[RobinROAR]

As mentioned in the paper, Ditto uses the off-the-shelf components “LivePortrait” to extract the motion representation, and then uses this motion representation to train a Diffusion Transformer model. These onnx are from the LivePortrait . If you want to replace them, you need to retrain a LivePortrait, which is a face reenactment model.