Usage example for get_output()

[anutkk]

According to the documentation generator.get_output() should return the generated logits.

In practice, this is the error message I get:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[37], [line 1](vscode-notebook-cell:?execution_count=37&line=1)
----> [1](vscode-notebook-cell:?execution_count=37&line=1) generator.get_output()

TypeError: get_output(): incompatible function arguments. The following argument types are supported:
    1. (self: onnxruntime_genai.onnxruntime_genai.Generator, arg0: str) -> numpy.ndarray

Invoked with: <onnxruntime_genai.onnxruntime_genai.Generator object at 0x7decfe932ff0>

The function expects an input string. However no matter what I put, the output is array([], dtype=float64).

What is the correct way to use this method?

[anutkk]

So after digging through the C++ source code, the answer is:

logits=generator.get_output("logits")

However for some reason at the first step the maximum token is different from the output of get_next_tokens(). Not sure if this is a bug or a misunderstanding.

import onnxruntime_genai as og
import numpy as np

prompt = '''<|user|>
Please tell me the time.<|end|>
<|assistant|>'''

model=og.Model("/home/ubuntu/models/Phi-3-mini-4k-instruct-onnx/cuda/cuda-fp16/")

tokenizer = og.Tokenizer(model)

tokens = tokenizer.encode(prompt)

params=og.GeneratorParams(model)
params.input_ids = tokens

generator = og.Generator(model, params)
i = 0
while not generator.is_done():
    generator.compute_logits()
    generator.generate_next_token()  
    new_token = generator.get_next_tokens()[0]
    logits = generator.get_output("logits").squeeze()
    new_token2 = np.argmax(logits)
    print(new_token, " ", new_token2)
    i += 1
    if i > 10:
        break


print()

And the result:

306   18
29915   29915
29885   29885
9368   9368
304   304
3867   3867
1855   1855
29899   29899
2230   2230
848   848
29892   29892

[natke]

Created an issue for this as it looks like it needs to be investigated

[natke]

See #591