Very high FID and KID

[sonhm3029]

Hello authors, thanks for your great paper and code repository.

I am reproducing your result in paper with Dresscode dataset.

1. I have generate the agnostic mask
2. I am using this script to inference and generate sample:

CUDA_VISIBLE_DEVICES=0 python inference.py \
--dataset dresscode \
--data_root_path xyz
--output_dir results \
--dataloader_num_workers 8 \
--batch_size 16 \
--seed 555 \
--mixed_precision no \
--allow_tf32 \
--eval_pair \
--repaint

After all image generated, I calculated the FID and KID score using eval.py file. I have try other method like using torch-fidelity too. And the result for FID is very high: ~40-50.
I don't know why this happen. Everything works well with vitonhd dataset.

Do you or anyone encounter this issues like me or have any idea of what 's happening?

[Zheng-Chong]

Could you show some of the input and output visualization samples?

[sonhm3029]

Hello, this is some samples from upper body category - paired:

048484_0.jpg

048426_0.jpg

048413_0.jpg

0484393_0.jpg

048392_0.jpg

[sonhm3029]

This is the result I received for evaluate above upper body paired:
+-------------------+-------------------+--------------------+---------------------+
| FID | KID | SSIM | LPIPS |
+-------------------+-------------------+--------------------+---------------------+
| 57.68753901171948 | 42.18205313563561 | 0.9333221912384033 | 0.03060338646173477 |
+-------------------+-------------------+--------------------+---------------------+

[Zheng-Chong]

Thank you for providing the information. I think you can test whether fid and kid are 0 when the GT folder and Pred folder are set to the same folder. If they are not 0, then it's a clean-fid issue. In my tests, adjusting different batch_sizes affected the results.

[JunyaoHu]

This is the result I received for evaluate above upper body paired:

 +-------------------+-------------------+--------------------+---------------------+ 
| FID | KID | SSIM | LPIPS | 
+-------------------+-------------------+--------------------+---------------------+ 
| 57.68753901171948 | 42.18205313563561 | 0.9333221912384033 | 0.03060338646173477 | 
+-------------------+-------------------+--------------------+---------------------+

@sonhm3029 @Zheng-Chong

I have very similar quantitative results, have you solved it? :) The results I obtained from VITON-HD are basically consistent with the original data in the paper. It seems that the main issue still lies in DressCode dataset.

I have confirmed that I used the repaint parameter and scaled it to 512

And how to calculate metrics on whole DressCode Dataset (upper + lower + dress)?

---

GT Folder : /mnt/data/hjy/datasets/DressCode/upper_body/images_512
Pred Folder: /home/hjy/repo/vton/CatVTON/results/DressCode/dresscode-512/paired/upper_body

+-------------------+--------------------+--------------------+---------------------+
|        FID        |        KID         |        SSIM        |        LPIPS        |
+-------------------+--------------------+--------------------+---------------------+
| 57.68800930125772 | 41.540698846346366 | 0.9333102107048035 | 0.03056919574737549 |
+-------------------+--------------------+--------------------+---------------------+

---

GT Folder : /mnt/data/hjy/datasets/DressCode/lower_body/images_512
Pred Folder: /home/hjy/repo/vton/CatVTON/results/DressCode/dresscode-512/paired/lower_body

+-------------------+--------------------+--------------------+----------------------+
|        FID        |        KID         |        SSIM        |        LPIPS         |
+-------------------+--------------------+--------------------+----------------------+
| 42.50220108517664 | 26.084873648648628 | 0.9203006029129028 | 0.036912254989147186 |
+-------------------+--------------------+--------------------+----------------------+

---

GT Folder : /mnt/data/hjy/datasets/DressCode/dresses/images_512
Pred Folder: /home/hjy/repo/vton/CatVTON/results/DressCode/dresscode-512/paired/dresses

+--------------------+--------------------+--------------------+----------------------+
|        FID         |        KID         |        SSIM        |        LPIPS         |
+--------------------+--------------------+--------------------+----------------------+
| 26.210086631073978 | 14.308208340840856 | 0.8621599078178406 | 0.055189039558172226 |
+--------------------+--------------------+--------------------+----------------------+

---

by the way, I can provide the results on GT: They looks normal.

GT Folder : /mnt/data/hjy/datasets/DressCode/upper_body/images_512
Pred Folder: /mnt/data/hjy/datasets/DressCode/upper_body/images_512

+------------------------+-----------------------+--------------------+-------+
|          FID           |          KID          |        SSIM        | LPIPS |
+------------------------+-----------------------+--------------------+-------+
| 2.5715962692629546e-09 | -0.017784434434465767 | 0.9998603463172913 |  0.0  |
+------------------------+-----------------------+--------------------+-------+

---

GT Folder : /mnt/data/hjy/datasets/DressCode/lower_body/images_512
Pred Folder: /mnt/data/hjy/datasets/DressCode/lower_body/images_512

+-----------------------+----------------------+-------------------+-------+
|          FID          |         KID          |        SSIM       | LPIPS |
+-----------------------+----------------------+-------------------+-------+
| 9.573398074280703e-09 | -0.01703169419420192 | 0.999916672706604 |  0.0  |
+-----------------------+----------------------+-------------------+-------+

---

GT Folder : /mnt/data/hjy/datasets/DressCode/dresses/images_512
Pred Folder: /mnt/data/hjy/datasets/DressCode/dresses/images_512

+-----------------------+----------------------+--------------------+-------+
|          FID          |         KID          |        SSIM        | LPIPS |
+-----------------------+----------------------+--------------------+-------+
| 6.161826604511589e-10 | 0.012959787287286417 | 0.9999845027923584 |  0.0  |
+-----------------------+----------------------+--------------------+-------+

---

environment:

torch                    2.4.0+cu124
torchaudio               2.4.0+cu124
torchmetrics             1.8.2
torchvision              0.19.0+cu124
clean-fid                0.1.35
numpy                    1.26.4
scipy                    1.13.1

[tenmiles0]

Hi, I would like to ask how the correspondence between the GT and the generated results is obtained

[JunyaoHu]

Hi, I would like to ask how the correspondence between the GT and the generated results is obtained 嗨，我想问一下 GT 和生成结果之间的对应关系是如何得到的

i think you can see here。

for fvd, kid:

https://github.com/GaParmar/clean-fid/blob/e88c4d6269a4bbf04c04deeb578475b57719acee/cleanfid/fid.py#L191C1-L264C20

for ssim, lpips

https://github.com/Zheng-Chong/CatVTON/blob/7818397f25613beedb3d861a34769f607cfcf3b1/eval.py#L38C1-L45C1