Skip to content

feat: generate response length based on a histogram when max_tokens is defined in the request #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Aug 28, 2025
Merged

feat: generate response length based on a histogram when max_tokens is defined in the request #169

merged 6 commits into from
Aug 28, 2025

Conversation

@mayabar
Copy link
Collaborator

@mayabar mayabar commented Aug 25, 2025

  • Added logic to randomly select the response length based on a pre-defined histogram when the request includes the max_tokens property
  • This initial version applies the same histogram regardless of the specific max_tokens value
  • Future improvements will introduce dynamic buckets adapting the histogram according to different max_tokens values.

fix #167

@mayabar
when request contains max tokens, calculate length on the response ba…
6a8116b 
…sed on a histogram - intial implementation

Signed-off-by: Maya Barnea <[email protected]>
@mayabar mayabar requested review from irar2 and shmuelk August 25, 2025 11:10
@mayabar
- in case max_tokens is defined in the request, finish reason will no…
e8b0cbb 
…t be randomly selected, instead it will be stop when response length is maxTokens, otherwise - stop

- fix utils_tests

Signed-off-by: Maya Barnea <[email protected]>
shmuelk
shmuelk reviewed Aug 26, 2025
View reviewed changes
@mayabar mayabar requested a review from shmuelk August 26, 2025 08:01
shmuelk
shmuelk requested changes Aug 26, 2025
View reviewed changes
Copy link
Collaborator

@shmuelk shmuelk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding more comments to the function getResponseLengthByHistogram

@mayabar mayabar requested a review from shmuelk August 27, 2025 14:42
mayabar added 2 commits August 28, 2025 09:46
@mayabar
fix misspelling
94eda74 
Signed-off-by: Maya Barnea <[email protected]>
@mayabar
fixes in comments
6c14aa6 
Signed-off-by: Maya Barnea <[email protected]>
@shmuelk
Copy link
Collaborator

shmuelk commented Aug 28, 2025

/lgtm

/approve

@github-actions github-actions bot added the lgtm label Aug 28, 2025
@mayabar mayabar merged commit b98882a into llm-d:main Aug 28, 2025
4 checks passed
@mayabar mayabar deleted the max_tokens branch August 28, 2025 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shmuelk shmuelk shmuelk approved these changes

@github-actions github-actions[bot] github-actions[bot] approved these changes

@irar2 irar2 Awaiting requested review from irar2

Assignees

No one assigned

Labels

lgtm

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Enhance calculation of tokens number in response based on request's max_tokens parameter - simple logic

2 participants

@mayabar @shmuelk