Skip to content

Comments

Automate manual audio regression comparisons#1758

Open
derselbst wants to merge 16 commits intoFluidSynth:masterfrom
derselbst:copilot/automate-audio-comparison
Open

Automate manual audio regression comparisons#1758
derselbst wants to merge 16 commits intoFluidSynth:masterfrom
derselbst:copilot/automate-audio-comparison

Conversation

@derselbst
Copy link
Member

@derselbst derselbst commented Feb 10, 2026

The manual rendering tests previously required human inspection and could not be executed in CI. Inspired by the discussion in #1747, I fired up copilot to introduce an automated workflow that compiles a reference FluidSynth revision, renders the audio with the reference and current version of FluidSynth, and compares SNR/RMS/ABS metrics, without requiring versioned "golden reference" files.

  • Allow check_rendering to run against a caller-specified fluidsynth binary via MANUAL_TEST_FLUIDSYNTH.
  • Add test/run-manual-regression.sh to build current + reference revisions, render both sets, and compare with SoX metrics and thresholds.
  • New Linux workflow rendering-regression.yml to run the regression comparison with configurable reference ref and thresholds.

Example usage:

REFERENCE_REF=HEAD~1 SNR_MIN=60 RMS_MAX=0.0001 ABS_MAX=0.01 test/run-manual-regression.sh

TODO:

  • test suite currently fails when and old version of fluidsynth is being passed new / unknown commandline options, like -o synth.limiter.active=1
  • SNR, RMS and abs diff need to be evaluated to get meaningful values

Copilot AI and others added 8 commits February 10, 2026 05:28
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
Co-authored-by: derselbst <8152480+derselbst@users.noreply.github.com>
@derselbst
Copy link
Member Author

@spessasus FYI

@derselbst derselbst closed this Feb 11, 2026
@derselbst derselbst deleted the copilot/automate-audio-comparison branch February 11, 2026 19:35
@derselbst derselbst restored the copilot/automate-audio-comparison branch February 14, 2026 13:32
@derselbst
Copy link
Member Author

Why did I close this?

@derselbst derselbst reopened this Feb 14, 2026
@spessasus
Copy link
Contributor

I have no idea! The PR looks good so far.

@derselbst derselbst added this to the 2.6 milestone Feb 18, 2026
@derselbst derselbst marked this pull request as ready for review February 18, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants