Open
Conversation
- display special tokens (like eot) properly - escape control characters for display properly instead of messing with repr - change output formatting of the logits viewer to be clearer and show token ids
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses the following issues with the logits viewer:
Previously, the logits viewer would display blank strings for special tokens like end of turn, eos, etc. This seems to be a llama-server bug, but we can work around it on the client-side by re-decoding blank tokens.
Previously, each token would go through repr and some buggy logic to strip apostrophes. That was a bad approach because repr can surround with either style of Python quotes - apostrophes (0x27) or quotes (0x22), depending on the content. It will also escape backslashes, quotes, and other special characters, which isn't desirable for a visual representation of tokens. Instead, we escape only control characters (ASCII C0 and Unicode C1) and leave the rest alone.
The old formatting was confusing to me - many tokens begin with spaces so it was hard to see what token would actually be emitted by the model. Token IDs are also useful if we want to ban some tokens. I've changed the format to
probability - [token] (token id)Checklist: