Quality degradation fine-tuning on solo and group portraits

[aamiransari23]

I attempted fine-tuning on a dataset of solo and group portrait images (33k images with corresponding 33k alpha masks). However, I observed that after each epoch, the quality of the predicted masks progressively degrades.

I am using a matting task setup with pretrained weights (model.safetensor). All images in the dataset are at 1K resolution.

Could you help identify possible reasons for this issue?

[ZhengPeng7]

Are the test images similar to the ones for fine-tuning training? If the test ones are closer to the pretraining data I used before, it makes sense. If not, fine-tuning should bring some improvement.

Now that the fine-tuning has even brought some degradations, you may consider using a smaller learning rate.

BTW, you don't have to use the model.safetensor files in the HF model pages. You can find all of them from the GitHub release or GDrive folder as given in the README.

[aamiransari23]

I fine-tuned your P3M-10k pretrained weights on my solo headshot dataset (~6k images, average resolution ~4K) to improve hair detail quality. I used FP16 training with a matting objective.

However, I’m seeing consistent degradation in hair quality:

• By epoch 5, the results are already worse than the original pretrained model
• After that, training stagnates and does not recover
• I tried both 1e-5 and 1e-6 learning rates, but the behavior remains the same

I’d like help understanding what might be causing this.

For reference, I’m attaching:

1. Composed output from the base pretrained weights (P3M-10k)
2. Composed output from my fine-tuned model (epoch 200)
3. The ground truth masks used during training

Let me know if you suspect any issues with the data, setup, or training strategy—I’m happy to provide more details.

[ZhengPeng7]

Glad to see the details.
From the case, I feel that the fine-tuned results are more binary -- see the hair regions.
You need to make sure that you set small weights or directly turn off the IoU loss, or even the BCE loss, which optimizes the model more for binary segmentation. Increasing the weights of the L1 loss should help in this kind of problem.
And if you only need the extracted foreground portrait, remember to use the fast-fg-est, which is powerful on this problem. You can even find the improved version in the GPU implementation in this project.