Add multilingual QA report support - #16242
Conversation
| DistributionMetricsRegistry: list[DistributionMetricSpec] = [ | ||
| DistributionMetricSpec("cer", "CER", True, True, (0.0, 0.3)), | ||
| DistributionMetricSpec("utmosv2", "UTMOS v2", False), | ||
| DistributionMetricSpec("pred_context_ssim", "SSIM (pred vs context)", False), |
There was a problem hiding this comment.
why do we want to remove this one? i feel it is a useful indicator for decision making.
There was a problem hiding this comment.
The report unconditionally loads pred_context_ssim for every benchmark to run statistical tests and boxplots. If either evaluation lacks that per-sample field, report generation fails (for text context only manifests I think).
There was a problem hiding this comment.
gotcha. then the ideal way is to add a condition to keep this pred_context_ssim for audio context only. what do you think?
There was a problem hiding this comment.
Should we separate it into two? One report for audio and one report for text? It would require 2 constants folders though
|
I'm good to merge to the training branch. But we should think about how to merge to main if we want to merge to main |
|
this PR fix is not enough to ensure success of supplying new eval artifact dir structure. So i would expect more fixes.
|
|
The failed artifact structures are because our inference script parallelizes things across datasets so it creates a separate directory per dataset. This way concurrent processes don't try to overwrite the same all metrics csv file. I have updated the slurm eval script to create simlinks once each benchmark eval finishes to create a directory structure that works with our existing report generation tool. |
3 participants
No description provided.