Inception Score is dependent on split_size

[CameronBraunstein]

I was trying to replicate the results of the VAR paper (https://github.com/FoundationVision/VAR). However, I saved 50k images to a .npz in order by class (the first 50 images have label 0, the next have label 1, etc.,). This results in an extremely low inception score:

$python evaluator.py VIRTUAL_imagenet256_labeled.npz <my_samples>.npz
...
Computing evaluations...
Inception Score: 62.07027053833008
FID: 3.374905178956624
sFID: 7.929707907039187
Precision: 0.85034
Recall: 0.4955

However, changing the split_size in evaluator.py gave me more expected numbers:

print("Inception Score:", evaluator.compute_inception_score(sample_acts[0]) ->
print("Inception Score:", evaluator.compute_inception_score(sample_acts[0], split_size = sample_acts[0].shape[0]))

gives this:

$python evaluator.py VIRTUAL_imagenet256_labeled.npz <my_samples>.npz
...
Computing evaluations...
Inception Score: 298.4532165527344
FID: 3.3748805195803584
sFID: 7.929708379309773
Precision: 0.85028
Recall: 0.4958

This is a non-trivial difference (62 vs 298 !). I think it was missed in the original evals due to the random order that generated samples were added in. Ideally, the code should be order agnostic (I prefer adding samples in order). I suggest either to the one-liner above for a quick fix. It looks like split_size is there for tractability - maybe a deeper fix is necessary to still get this benefit.

[PeterL1n]

I encountered the same bug under the exact setting haha.

My approach is to shuffle the entries and keep the split for a fair comparison.
I tried using a single split, but the inception score will be a lot higher than the shuffle approach.
The shuffle approach seems to yield similar results to TorchMetrics.