Why most image translation benchmarks use FID?

[Aike2071]

FID assumes that the data feature distribution follows a multivariate Gaussian distribution, but I found that many actual datasets are not strictly Gaussian distributions.

[usert5432]

Hi @Aike2071 ,

That is an interesting question. You are absolutely correct about the FID and the feature distribution. While I do not know for sure why FID is used, I suspect there are a mix of legitimate and historical reasons for it:

1. It roughly correlates with human preferences
2. It is relatively easy to calculate
3. More importantly. Old papers used FID. Which means, when you benchmark against them, you either have to use FID, or undertake a significant task of re-evaluating all previous results with new metrics

I believe this legacy factor is the main reason why most image generation benchmarks continue to use FID, despite its theoretical limitations.

[Aike2071]

Hi @usert5432 ,
Thank you so much for your reply, I am using your uvcgan code for new dataset and it worked perfectly fine!
I want to find a quantitative indicator to describe the quality of style transfer.So I found FID and KID.since my dataset’s feature distribution is far from Gaussian distribution(the FID score changes over epochs corresponds to the fact though), I did some research and found that some datasets using FID are not Gaussian distribution either.

[usert5432]

Hi @Aike2071,

I am glad uvcgan works on your dataset! It seems, empirically, FID performs remarkably well despite its faulty assumptions, so many people still use it. I remember this old paper https://arxiv.org/pdf/1802.03446 had a good summary of alternative measures, in case if it is helpful.

Please let me know if you have further questions.

[Aike2071]

Hi @usert5432 ,
The paper is just what I need to find multi-dimensional evaluation for my work!Thanks a lot.

[Aike2071]

I have one more question if you could explain to me.in the UVCGANv2 paper, How do you calculate PSNR,SSIM and LPIPS?The calculation of these values rely on image pairs,I want to know the principle you use to pair images in the real and fake domain

[usert5432]

I have one more question if you could explain to me.in the UVCGANv2 paper, How do you calculate PSNR,SSIM and LPIPS?The calculation of these values rely on image pairs,I want to know the principle you use to pair images in the real and fake domain

Sorry, I may be misinterpreting the question -- if that is the case, please correct me if I am wrong here. But, in general, to construct pairs, we simply take an original image (say x_A). We translate it to the other domain (e.g. x_{A -> B} = G_{A -> B} (x_A)). This gives us a pair (x_A, x_{A -> B}). Then, we calculate the PSNR, SSIM, CLIP, etc scores between them.

For instance, on this image grid each example corresponds to a pair.

[Aike2071]

I see, I thought I have to pair,for example, x_{A -> B} and x_B to calculate the scores because FID is calculated between them. But in fact these scores are calculated between x_{A -> B} and x_A and they’re already in pairs