improve beam search

[monperrus]

right now, we're using beam search as an off-the-shelf component.

It would be great that:

• the search embeds some kind of patch quality knowledge: the first patch generated should have a better quality than
• the search ensures some kind of diversity in the way the bug is being fixed

Ideas from UCDavis:

• by raising the "temperature" (AFAIU adding a multiplicative factor on the softmax)
• by using random sampling instead of beam search

Related work:

• Diverse beam search: Decoding diverse solutions from neural sequence models
• Improved Beam Search Diversity for Neural MachineTranslation with k-DPP Sampling

By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

The Curious Case of Neural Text Degeneration

[monperrus]

Status in OpenNMT by @javierron

Relative order of patches does not change. Two patches generated with beam size 10 will maintain their order when generated with beam size 100, however there might be a different number of patches in between the original two.