Idea of Ilan:
Use percentile based auto thresholds:
thresholds = np.percentile(scores, np.linspace(1, 99, 99))
Benefits are that amount of thresholds is limited to 99 and is in domain by construction.
Perhaps most importantly each step flips a fixed number of predicted labels