Fix #2034 (#2048)

* Fix #2034

* Fix build error

* Adjust threshold

* Add temporary debug prints

* Adjust threshhold

* Another threshold adjustment for macOS on GitHub Actions CI...

* Performance improvement by avoiding unnecessary chrono access

* More performance improvement to avoid unnecessary chrono access

Author: yhirose