Refactor Usage handling #2108
Refactor Usage handling #2108
Conversation
- Remove model specific Usage implementations
- Add `Object getNativeUsage()` to Usage interface
- This will allow the model specific Usage data to be returned
- At the client side, client needs to cast the return type of `getNativeUsage` into the corresponding Usage returned by the model API
- Rename `generationTokens` to `completionTokens`
- Since `completion` token name is more common among the models, renaming generation tokens into completion tokens
- Change the prompt, completion and total token return types to Integer
- Use DefaultUsage for most of the model specific usage handling
- When initializing set the native usage to the model specific usage type
Resolves spring-projects#1407
ilayaperumalg
force-pushed
the
refactoring-usage
branch
from
482c07a to
42fd8f0
Compare
| public DefaultUsage(@JsonProperty("promptTokens") Integer promptTokens, | ||
| @JsonProperty("completionTokens") Integer completionTokens, | ||
| @JsonProperty("totalTokens") Integer totalTokens) { | ||
| this(promptTokens, completionTokens, totalTokens, null); |
There was a problem hiding this comment.
The native usage argument should not be null but Map.of, even though it is handled later.
| DefaultUsage that = (DefaultUsage) o; | ||
| return Objects.equals(this.promptTokens, that.promptTokens) | ||
| && Objects.equals(this.generationTokens, that.generationTokens) | ||
| && Objects.equals(this.completionTokens, that.completionTokens) |
There was a problem hiding this comment.
equals should also include the nativeUsage field in the calculate, as well as for hascode/tostring
| private final Long generationTokens; | ||
|
|
||
| private final Long totalTokens; | ||
| private final int totalTokens; |
| * @author Ilayaperumal Gopinathan | ||
| * @since 0.7.0 | ||
| */ | ||
| public class EmptyUsage implements Usage { |
There was a problem hiding this comment.
we should think about getting rid of this and use 'new DefaultUsage(0,0)
|
Added support to ser/deser the native usage object We should discuss getting rid of EmtpyUsage, can call DefaultUsage(0, 0) mergedin 4b64aa0 |
2 participants
Object getNativeUsage()to Usage interfacegetNativeUsageinto the corresponding Usage returned by the model APIgenerationTokenstocompletionTokenscompletiontoken name is more common among the models, renaming generation tokens into completion tokens