Calibration issue at a lower operation frequency

[regymm]

At the latest main branch, I noticed the calibration to fail at a lower controller frequency (62.5 MHz controller, 250 MHz 4x clock, 200 MHz refclk). The state_calibrate can't go through CHECK_STARTING_DATA, so the controller will reset itself and (indefinitely) try again.

Here's an ILA capture when this happened:

I'm using Nexys Video, with Vivado.
The default 83.333 MHz/333.333 MHz works fine. Also tried 100 MHz/400 MHz, the UART echo demo doesn't behave correctly (maybe some other minor issue) but I can still see the calibration pass to finish (state 23) in ILA.
62.5 MHz/250 MHz worked before, but the last working version I can find is commit 3c4d5db0, which seems doesn't exist now.