Integrate shortlist to scorer for compatiblity with Quality Estimation

[jerinphilip]

There's some mismatch of shortlist on/off with marian-scorer and bergamot-inference, with marian-scorer not having the capability to use shortlist. This affects QE's training data generation and have requested that marian-scorer be able to use shortlist and obtain logprobs to the requirements of QE.

[graemenail]

Still working on this; just wanted to acknowledge this issue and give a small update. I have something which is at least partially working. I need to verify it's actually applying the shortlist as expected, and also check for unintended side effects.

[graemenail]

Dealing with issues seemingly related to this code when scoring sentences with out-of-shortlist tokens.

e.g. --shortlist lex.s2t.gz 200 200 0

299999616.000000 ||| WordScores= -38.53593 100000000.00000 -0.00023 -32.71639 -31.97223 -18.29514 -7.01926 -28.06969 100000000.00000 -28.04831 100000000.00000 -29.33926 -0.05415 -23.04181 -22.67663 -0.00217 -3.16472 -4.53165 -2.65718 -19.06726 -34.58292 -0.00862 -29.59204 -2.37931

vs no-shortlist

-50.828621 ||| WordScores= -17.67943 -0.10885 -0.00024 -1.42122 -0.20161 -7.92681 -7.02563 -0.00431 -0.00043 -0.02556 -0.39346 -0.00009 -0.05415 -1.20754 -0.00301 -0.00221 -3.16518 -4.53655 -2.65727 -0.00129 -0.08333 -0.00869 -1.94225 -2.37950

As the size of the shortlist is increased these issues disappear.

[graemenail]

There are experimental PRs for this:

• Experimental: Allow shortlists in marian-scorer graemenail/marian-dev#2 (marian-nmt)
• Experimental: Allow shortlists in marian-scorer (browsermt) graemenail/marian-dev#3