Skip to content

Revert "tempered posterior experiment (Temperature) with log scores"#35

Merged
hansenp merged 1 commit intodevelopfrom
revert-30-lc/tempered_posterior_experiment
Jan 26, 2026
Merged

Revert "tempered posterior experiment (Temperature) with log scores"#35
hansenp merged 1 commit intodevelopfrom
revert-30-lc/tempered_posterior_experiment

Conversation

@hansenp
Copy link
Collaborator

@hansenp hansenp commented Jan 5, 2026

For the reasons I've already mentioned in my review, I'd like to revert the merge of this PR. In my opinion, the changes don't reflect what we discussed beforehand, and I requested in my review that we discuss this in the new year. I suggest we do this together with @pnrobinson. once he returns from vacation next week.

@leokim-l
Copy link
Collaborator

I strongly disagree with this as detailed here

#30 (comment)

(As mentioned before, there is no reason why your code should be in the standard branch and mine in another secondary one, just because you were quicker to put it there... It does not seem fair to me)

Let us discuss in person and with @pnrobinson present on next Wednesday at 1pm to solve this and decide how to move forward.

@leokim-l
Copy link
Collaborator

I had the chance (thanks to everything being in one place) to extensively test both ideas. Although the rank @ K type metric is comparable, if not sometimes even a tiny tiny bit better with temperature, the PR curves look not so promising. Whether further investigation is able to clear this up and actually make the temperature version better is, to the best of my understanding, an open question, though the evidence so far is, again, not promising. This can be a task for later, however, for now, for clarity, I agree with merging this pull request and henceforth calling what is in this development branch the standard way to do this. What is seen in the plots below are some examples of temperature, where for ease of labeling T1 is the code without temperature. I also tested T=10, T=390 and the picture did not change qualitatively. @hansenp feel free to merge this pull request, thanks!

image image image image

@hansenp hansenp merged commit b2cff39 into develop Jan 26, 2026
2 checks passed
@leokim-l leokim-l deleted the revert-30-lc/tempered_posterior_experiment branch February 2, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants