feat: Add log probabilities and number of input tokens to additional outputs

[kthui]

What does the PR do?

Add log probabilities and number of input tokens to additional outputs.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

N/A

Where should the reviewer start?

Start with the documentation, and then move to the implementation and finally the tests.

Test plan:

New tests for the new additional outputs will be added.

• CI Pipeline ID: 21025199

Caveats:

N/A

Background

N/A

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[kthui]

For example:

input parameter logprobs = 2

text_output: the
logprobs:

[{
    '5': {'logprob': -1.0246853828430176, 'rank': 1, 'decoded_token': ' the'},
    '10': {'logprob': -2.2121853828430176, 'rank': 2, 'decoded_token': ' a'}
}]

text_output: term
logprobs:

[{
    '1385': {'logprob': -3.2646501064300537, 'rank': 1, 'decoded_token': ' term'},
    '78': {'logprob': -3.7529313564300537, 'rank': 2, 'decoded_token': ' first'}
}]

[rmccorm4]

Minor doc comments - LGTM otherwise

[rmccorm4]

@oandreeva-nv for second opinion

[oandreeva-nv]

LGTM!