Make default max_seq_length in openai_api consistent to chat

iseeyuan · web-flow · commit 49687b61da10 · 2024-09-25T19:51:29.000-07:00
Those two apis should have the same default max_seq_length. There can be a better location to define this. The max_seq_length in chat is 2048, in https://github.com/pytorch/torchchat/blob/main/torchchat/generate.py#L821. The way max_seq_length defined in model looks flaky. It should be consistent to the model definition and the cache setup.
diff --git a/torchchat/usages/openai_api.py b/torchchat/usages/openai_api.py
@@ -289,7 +289,7 @@ def __init__(self, *args, **kwargs):
             )
         except:
             # can not find max_seq_length in model config, use default value
-            self.max_seq_length = 128
+            self.max_seq_length = 2048
         # The System fingerprint is a unique identifier for the model and its configuration.
         self.system_fingerprint = (
             f"{self.builder_args.device}_{self.builder_args.precision}"

Original file line number	Diff line number	Diff line change
`@@ -289,7 +289,7 @@ def __init__(self, args, *kwargs):`
`289`	`289`	`)`
`290`	`290`	`except:`
`291`	`291`	`# can not find max_seq_length in model config, use default value`
`292`		`- self.max_seq_length = 128`
	`292`	`+ self.max_seq_length = 2048`
`293`	`293`	`# The System fingerprint is a unique identifier for the model and its configuration.`
`294`	`294`	`self.system_fingerprint = (`
`295`	`295`	`f"{self.builder_args.device}_{self.builder_args.precision}"`