Bluetooth refuses to connect when compiling own firmware

[Tirpitz93]

I cannot connect to bluetooth when running firmware I have compiled myself (un-changed clone of this repo).
but when I use the flash tool mentioned in the readme I can use the bundled binary firmware and it works.
is anyone else experiencing this issue.
I have also tried various versions of the BLE-keyboard library, and tried using NimBLE instead of the ESP BLE stack, no difference.

[Tirpitz93]

I think pinning the ESP framework to version 6.1 fixed this issue for me.