Introduce GenerationConfig #10228
Merged
Introduce GenerationConfig #10228
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
larryliu0820
requested review from
cccclai,
iseeyuan,
jackzhxng,
kirklandsign,
lucylq,
shoumikhin and
swolchok
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10228
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ef7d4ca with merge base f911567 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
iseeyuan
reviewed
Apr 16, 2025
|
|
||
| if (warmup) { | ||
| runner.warmup(prompt, seq_len); | ||
| runner.warmup(prompt, /*max_new_tokens=*/seq_len); |
Contributor
There was a problem hiding this comment.
Would it be added in the internal runner as well?
Contributor
Author
There was a problem hiding this comment.
which internal runner?
iseeyuan
reviewed
Apr 16, 2025
iseeyuan
approved these changes
Apr 16, 2025
cccclai
reviewed
Apr 16, 2025
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 16, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
707bda8 to
74cfb7f
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
larryliu0820
added
the
release notes: api
larryliu0820
added a commit
that referenced
this pull request
Apr 16, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
74cfb7f to
5ecf7b7
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 16, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
5ecf7b7 to
834fac2
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
1 similar comment
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
larryliu0820
added a commit
that referenced
this pull request
Apr 16, 2025
Summary: Pull Request resolved: #10228 Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
834fac2 to
72cbdf1
Compare
larryliu0820
added a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
72cbdf1 to
605ff4d
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
2fc8d51 to
163ccea
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
163ccea to
e89ba89
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
larryliu0820
added a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
e89ba89 to
0357334
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
larryliu0820
added a commit
that referenced
this pull request
Apr 17, 2025
Summary: Pull Request resolved: #10228 Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
0357334 to
febbfa6
Compare
larryliu0820
added a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
febbfa6 to
4009fda
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
larryliu0820
added a commit
that referenced
this pull request
Apr 17, 2025
Summary: Pull Request resolved: #10228 Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
larryliu0820
force-pushed
the
export-D73091676
branch
from
4009fda to
9fe9659
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
9fe9659 to
4e038ea
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 17, 2025
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
4e038ea to
6004124
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
Summary: Started to implement #9341 Started to fix #8495 This PR introduces `GenerationConfig` which contains the configs that can be changed across different invocations of `generate()`. For example, `temperature` is moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we call `generate()`. Similarly we put `echo` and `warming` into the config. We also allow both `seq_len` and `max_new_tokens` to be passed through the config and we determine the value of `max_new_tokens` based on these 2 config values, pte file metadata as well as the number of prompt tokens. Reviewed By: iseeyuan Differential Revision: D73091676
facebook-github-bot
force-pushed
the
export-D73091676
branch
from
6004124 to
ef7d4ca
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D73091676 |
keyprocedure
pushed a commit
to keyprocedure/executorch
that referenced
this pull request
Apr 21, 2025
Differential Revision: D73091676 Pull Request resolved: pytorch#10228
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Started to implement #9341
Started to fix #8495
This PR introduces
GenerationConfigwhich contains the configs that can be changed across different invocations ofgenerate().For example,
temperatureis moved out from the runner constructor for it's not tied to the runner instance but instead should be adjustable every time we callgenerate().Similarly we put
echoandwarminginto the config.We also allow both
seq_lenandmax_new_tokensto be passed through the config and we determine the value ofmax_new_tokensbased on these 2 config values, pte file metadata as well as the number of prompt tokens.Differential Revision: D73091676