Consider evaluating DSSIM using CLIC-2021 peceptual quality task

[jyrkialakuijala]

See the similar issue in libjxl.

It would be great to evaluate DSSIM on the CLIC-2021 perceptual quality task. This should provide additional information to he community with respect to its performance characteristics when compared to other potentially usable perceptual quality metrics.

Please use the test data from the CLIC 2021 perceptual challenge to generate a CSV file with the decisions (see link below for the exact instructions):

https://github.com/fab-jul/clic2021-devkit/blob/main/README.md#perceptual-challenge

Please email them to George Toderici (or me) to get the final results / ranks. We'll publish these at: http://compression.cc/leaderboard/perceptual/test/

...and of course update this bug tracker.

[kornelski]

Test set reports accuracy 0.616, which is just slightly above the ms-ssim submission.

[kornelski]

@jyrkialakuijala Here's the full submission dssim3.csv.zip

[jyrkialakuijala]

Do you know if there is a way to compute three different accuracy/correlation values, one for raters testing the 0.07 bpp images, one for 0.15 bpp images and yet another for 0.3 bpp?

Motivation for this idea: I would expect different correlations for these. Also, I'd expect classic methods to work better for higher bitrates and possibly neural image quality methods to work better at the low.

[kornelski]

The small test set and the tool that is public don't use the actual scoring method, and don't expose any more info about the images:

fab-jul/clic2021-devkit#4

[kornelski]

I was unable to submit at http://compression.cc — the site very frequently times out. I think I've barely registered after a dozen of tries, but I can't log in, because the login form asks for username, and the registration didn't register any username. I've tried using e-mail as the username, but it didn't work.

[gtoderici]

Hi @kornelski - I am one of the CLIC organizers. I don't think the registration is opened anymore. We'll update to reflect this. With that said, please send me the submission file (i.e., the test set CSVs), and I'll be able to update this thread with the numbers. I have the email username@google.com (where username is gtoderici).