Feature/context token counter #608
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add contextTokens to ApiMetrics interface - Calculate context size using last API request's tokens - Display context token count in TaskHeader below total tokens - Use exact token counts instead of character estimation This helps users track the total size of their conversation context, which is useful for managing context window limits.
- Only update context tokens when both input and output tokens are non-zero - Keep previous context token count when API calls fail - Avoid resetting counter on partial or failed responses This makes the context token counter more resilient against edge cases and provides more accurate context size tracking during API failures.
MuriloFP
requested review from
ColemanRoo,
mrubens and
stea9499
as code owners
|
- Only consider API requests with valid token information - Skip messages with invalid/missing token data - Prevent counter from resetting on action approval messages - Ensure both tokensIn and tokensOut are valid numbers This makes the context token counter more stable and accurate by only updating on valid API responses with complete token data.
mrubens
reviewed
Jan 28, 2025
Co-authored-by: Matt Rubens <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added a Context token counter to the task header. It is a minimal addition that uses the same counting logic as the input and output tokens counter. It takes the total input tokens from the last message, and adds in the output tokens. This is updated every time a new message is received from the API, and covers against edge cases such as failed requests.
Type of change
How Has This Been Tested?
Tested it manually in debugging mode, added logic to prevent mistakes in edge cases.
Checklist:
Additional context
Related Issues
Reviewers
Important
Add context token counter to task header, updating with each API message in
getApiMetrics.tsand displaying inTaskHeader.tsx.contextTokenstoApiMetricsingetApiMetrics.ts, calculated as the sum oftokensInandtokensOutfrom the lastapi_req_startedmessage.contextTokensinTaskHeader.tsxto display the total context tokens in the task header.contextTokenstoTaskHeaderinChatView.tsx.contextTokensinTaskHeader.tsxunder the "Context" label.This description was created by
for 97fe13d. It will automatically update as commits are pushed.