Skip to content

Commit ca8c468

Browse files
Jintao-HuangJintao-Huang
authored
fix vllm==0.4.* slower than vllm==0.3.* (#1035)
1 parent f3b5d42 commit ca8c468

File tree

12 files changed

+72
-38
lines changed

12 files changed

+72
-38
lines changed

‎docs/source/LLM/支持的模型和数据集.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,8 +162,6 @@
162162
|internlm2-math-7b-chat|[Shanghai_AI_Laboratory/internlm2-math-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-7b/summary)|wqkv|internlm2|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-7b](https://huggingface.co/internlm/internlm2-math-7b)|
163163
|internlm2-math-20b|[Shanghai_AI_Laboratory/internlm2-math-base-20b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-base-20b/summary)|wqkv|default-generation|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-base-20b](https://huggingface.co/internlm/internlm2-math-base-20b)|
164164
|internlm2-math-20b-chat|[Shanghai_AI_Laboratory/internlm2-math-20b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-20b/summary)|wqkv|internlm2|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-20b](https://huggingface.co/internlm/internlm2-math-20b)|
165-
|internvl-chat-v1_5|[AI-ModelScope/InternVL-Chat-V1-5](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|-|[OpenGVLab/InternVL-Chat-V1-5](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5)|
166-
|internvl-chat-v1_5-int8|[AI-ModelScope/InternVL-Chat-V1-5-int8](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5-int8/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|-|[OpenGVLab/InternVL-Chat-V1-5-int8](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-int8)|
167165
|deepseek-7b|[deepseek-ai/deepseek-llm-7b-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-base/summary)|q_proj, k_proj, v_proj|default-generation|✔|✔||-|[deepseek-ai/deepseek-llm-7b-base](https://huggingface.co/deepseek-ai/deepseek-llm-7b-base)|
168166
|deepseek-7b-chat|[deepseek-ai/deepseek-llm-7b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-chat/summary)|q_proj, k_proj, v_proj|deepseek|✔|✔||-|[deepseek-ai/deepseek-llm-7b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat)|
169167
|deepseek-moe-16b|[deepseek-ai/deepseek-moe-16b-base](https://modelscope.cn/models/deepseek-ai/deepseek-moe-16b-base/summary)|q_proj, k_proj, v_proj|default-generation|✔|✔||-|[deepseek-ai/deepseek-moe-16b-base](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base)|
@@ -297,6 +295,8 @@
297295
|yi-vl-34b-chat|[01ai/Yi-VL-34B](https://modelscope.cn/models/01ai/Yi-VL-34B/summary)|q_proj, k_proj, v_proj|yi-vl|✔|✘|transformers>=4.34|vision|[01-ai/Yi-VL-34B](https://huggingface.co/01-ai/Yi-VL-34B)|
298296
|llava-llama-3-8b-v1_1|[AI-ModelScope/llava-llama-3-8b-v1_1-transformers](https://modelscope.cn/models/AI-ModelScope/llava-llama-3-8b-v1_1-transformers/summary)|q_proj, k_proj, v_proj|llava-llama-instruct|✔|✘|transformers>=4.36|vision|[xtuner/llava-llama-3-8b-v1_1-transformers](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers)|
299297
|internlm-xcomposer2-7b-chat|[Shanghai_AI_Laboratory/internlm-xcomposer2-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-7b/summary)|wqkv|internlm-xcomposer2|✔|✘||vision|[internlm/internlm-xcomposer2-7b](https://huggingface.co/internlm/internlm-xcomposer2-7b)|
298+
|internvl-chat-v1_5|[AI-ModelScope/InternVL-Chat-V1-5](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|vision|[OpenGVLab/InternVL-Chat-V1-5](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5)|
299+
|internvl-chat-v1_5-int8|[AI-ModelScope/InternVL-Chat-V1-5-int8](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5-int8/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|vision|[OpenGVLab/InternVL-Chat-V1-5-int8](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-int8)|
300300
|deepseek-vl-1_3b-chat|[deepseek-ai/deepseek-vl-1.3b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-vl-1.3b-chat/summary)|q_proj, k_proj, v_proj|deepseek-vl|✔|✘|attrdict|vision|[deepseek-ai/deepseek-vl-1.3b-chat](https://huggingface.co/deepseek-ai/deepseek-vl-1.3b-chat)|
301301
|deepseek-vl-7b-chat|[deepseek-ai/deepseek-vl-7b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-vl-7b-chat/summary)|q_proj, k_proj, v_proj|deepseek-vl|✔|✘|attrdict|vision|[deepseek-ai/deepseek-vl-7b-chat](https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat)|
302302
|paligemma-3b-pt-448|[AI-ModelScope/paligemma-3b-pt-448](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-448/summary)|q_proj, k_proj, v_proj|paligemma|✔|✘|transformers>=4.41|vision|[google/paligemma-3b-pt-448](https://huggingface.co/google/paligemma-3b-pt-448)|

‎docs/source_en/LLM/Supported-models-datasets.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,8 +162,6 @@ The table below introcudes all models supported by SWIFT:
162162
|internlm2-math-7b-chat|[Shanghai_AI_Laboratory/internlm2-math-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-7b/summary)|wqkv|internlm2|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-7b](https://huggingface.co/internlm/internlm2-math-7b)|
163163
|internlm2-math-20b|[Shanghai_AI_Laboratory/internlm2-math-base-20b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-base-20b/summary)|wqkv|default-generation|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-base-20b](https://huggingface.co/internlm/internlm2-math-base-20b)|
164164
|internlm2-math-20b-chat|[Shanghai_AI_Laboratory/internlm2-math-20b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-math-20b/summary)|wqkv|internlm2|✔|✔|transformers>=4.35|math|[internlm/internlm2-math-20b](https://huggingface.co/internlm/internlm2-math-20b)|
165-
|internvl-chat-v1_5|[AI-ModelScope/InternVL-Chat-V1-5](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|-|[OpenGVLab/InternVL-Chat-V1-5](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5)|
166-
|internvl-chat-v1_5-int8|[AI-ModelScope/InternVL-Chat-V1-5-int8](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5-int8/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|-|[OpenGVLab/InternVL-Chat-V1-5-int8](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-int8)|
167165
|deepseek-7b|[deepseek-ai/deepseek-llm-7b-base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-base/summary)|q_proj, k_proj, v_proj|default-generation|✔|✔||-|[deepseek-ai/deepseek-llm-7b-base](https://huggingface.co/deepseek-ai/deepseek-llm-7b-base)|
168166
|deepseek-7b-chat|[deepseek-ai/deepseek-llm-7b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-llm-7b-chat/summary)|q_proj, k_proj, v_proj|deepseek|✔|✔||-|[deepseek-ai/deepseek-llm-7b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat)|
169167
|deepseek-moe-16b|[deepseek-ai/deepseek-moe-16b-base](https://modelscope.cn/models/deepseek-ai/deepseek-moe-16b-base/summary)|q_proj, k_proj, v_proj|default-generation|✔|✔||-|[deepseek-ai/deepseek-moe-16b-base](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base)|
@@ -297,6 +295,8 @@ The table below introcudes all models supported by SWIFT:
297295
|yi-vl-34b-chat|[01ai/Yi-VL-34B](https://modelscope.cn/models/01ai/Yi-VL-34B/summary)|q_proj, k_proj, v_proj|yi-vl|✔|✘|transformers>=4.34|vision|[01-ai/Yi-VL-34B](https://huggingface.co/01-ai/Yi-VL-34B)|
298296
|llava-llama-3-8b-v1_1|[AI-ModelScope/llava-llama-3-8b-v1_1-transformers](https://modelscope.cn/models/AI-ModelScope/llava-llama-3-8b-v1_1-transformers/summary)|q_proj, k_proj, v_proj|llava-llama-instruct|✔|✘|transformers>=4.36|vision|[xtuner/llava-llama-3-8b-v1_1-transformers](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers)|
299297
|internlm-xcomposer2-7b-chat|[Shanghai_AI_Laboratory/internlm-xcomposer2-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-7b/summary)|wqkv|internlm-xcomposer2|✔|✘||vision|[internlm/internlm-xcomposer2-7b](https://huggingface.co/internlm/internlm-xcomposer2-7b)|
298+
|internvl-chat-v1_5|[AI-ModelScope/InternVL-Chat-V1-5](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|vision|[OpenGVLab/InternVL-Chat-V1-5](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5)|
299+
|internvl-chat-v1_5-int8|[AI-ModelScope/InternVL-Chat-V1-5-int8](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5-int8/summary)|wqkv|internvl|✔|✘|transformers>=4.35, timm|vision|[OpenGVLab/InternVL-Chat-V1-5-int8](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-int8)|
300300
|deepseek-vl-1_3b-chat|[deepseek-ai/deepseek-vl-1.3b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-vl-1.3b-chat/summary)|q_proj, k_proj, v_proj|deepseek-vl|✔|✘|attrdict|vision|[deepseek-ai/deepseek-vl-1.3b-chat](https://huggingface.co/deepseek-ai/deepseek-vl-1.3b-chat)|
301301
|deepseek-vl-7b-chat|[deepseek-ai/deepseek-vl-7b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-vl-7b-chat/summary)|q_proj, k_proj, v_proj|deepseek-vl|✔|✘|attrdict|vision|[deepseek-ai/deepseek-vl-7b-chat](https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat)|
302302
|paligemma-3b-pt-448|[AI-ModelScope/paligemma-3b-pt-448](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-448/summary)|q_proj, k_proj, v_proj|paligemma|✔|✘|transformers>=4.41|vision|[google/paligemma-3b-pt-448](https://huggingface.co/google/paligemma-3b-pt-448)|

‎swift/llm/deploy.py‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,11 @@
1515

1616
from swift.utils import get_logger, get_main, seed_everything
1717
from .infer import merge_lora, prepare_model_template
18-
from .utils import ChatCompletionResponse # noqa
19-
from .utils import (ChatCompletionRequest, ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice,
20-
ChatCompletionStreamResponse, ChatMessage, CompletionRequest, CompletionResponse,
21-
CompletionResponseChoice, CompletionResponseStreamChoice, CompletionStreamResponse, DeltaMessage,
22-
DeployArguments, Model, ModelList, UsageInfo, inference, inference_stream, messages_to_history,
23-
random_uuid)
18+
from .utils import (ChatCompletionRequest, ChatCompletionResponse, ChatCompletionResponseChoice,
19+
ChatCompletionResponseStreamChoice, ChatCompletionStreamResponse, ChatMessage, CompletionRequest,
20+
CompletionResponse, CompletionResponseChoice, CompletionResponseStreamChoice,
21+
CompletionStreamResponse, DeltaMessage, DeployArguments, Model, ModelList, UsageInfo, inference,
22+
inference_stream, messages_to_history, random_uuid)
2423

2524
logger = get_logger()
2625

‎swift/llm/infer.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,9 @@ def llm_infer(args: InferArguments) -> None:
383383
'response': response,
384384
'history': history,
385385
}
386+
images = infer_kwargs.get('images')
387+
if images is not None:
388+
obj['images'] = images
386389
history = new_history
387390
if jsonl_path is not None:
388391
append_to_jsonl(jsonl_path, obj)
@@ -438,6 +441,8 @@ def llm_infer(args: InferArguments) -> None:
438441
request['system'] = system
439442
if images is not None:
440443
request['images'] = images
444+
if args.truncation_strategy:
445+
request['truncation_strategy'] = args.truncation_strategy
441446
request_list.append(request)
442447
resp_list = inference_vllm(llm_engine, template, request_list, use_tqdm=True)
443448
result = []
@@ -452,6 +457,9 @@ def llm_infer(args: InferArguments) -> None:
452457
'label': request.pop('label', None),
453458
'history': request['history'],
454459
}
460+
images = request.get('images')
461+
if images is not None:
462+
obj['images'] = images
455463
if jsonl_path is not None:
456464
append_to_jsonl(jsonl_path, obj)
457465
result.append(obj)
@@ -491,15 +499,15 @@ def llm_infer(args: InferArguments) -> None:
491499
response, _ = inference(
492500
model, template, stream=args.stream and args.verbose, verbose=args.verbose, **kwargs)
493501
label = data.pop('response', None)
494-
if 'truncation_strategy' in kwargs:
495-
kwargs.pop('truncation_strategy')
496502
obj = {
497503
'system': kwargs['system'],
498504
'query': kwargs['query'],
499505
'response': response,
500506
'label': label,
501507
'history': kwargs['history'],
502508
}
509+
if images is not None:
510+
obj['images'] = images
503511
if jsonl_path is not None:
504512
append_to_jsonl(jsonl_path, obj)
505513
result.append(obj)

‎swift/llm/utils/__init__.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,10 @@
1111
from .preprocess import (AlpacaPreprocessor, ClsPreprocessor, ComposePreprocessor, ConversationsPreprocessor,
1212
PreprocessFunc, RenameColumnsPreprocessor, SmartPreprocessor, SwiftPreprocessor,
1313
TextGenerationPreprocessor)
14-
from .protocol import ChatCompletionResponse # noqa
15-
from .protocol import (ChatCompletionRequest, ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice,
16-
ChatCompletionStreamResponse, ChatMessage, CompletionRequest, CompletionResponse,
17-
CompletionResponseChoice, CompletionResponseStreamChoice, CompletionStreamResponse, DeltaMessage,
18-
Model, ModelList, UsageInfo, XRequestConfig, random_uuid)
14+
from .protocol import (ChatCompletionRequest, ChatCompletionResponse, ChatCompletionResponseChoice,
15+
ChatCompletionResponseStreamChoice, ChatCompletionStreamResponse, ChatMessage, CompletionRequest,
16+
CompletionResponse, CompletionResponseChoice, CompletionResponseStreamChoice,
17+
CompletionStreamResponse, DeltaMessage, Model, ModelList, UsageInfo, XRequestConfig, random_uuid)
1918
from .template import (DEFAULT_SYSTEM, TEMPLATE_MAPPING, History, Prompt, StopWords, Template, TemplateType,
2019
get_template, register_template)
2120
from .utils import (LazyLLMDataset, LLMDataset, dataset_map, download_dataset, find_all_linears, find_embedding,

‎swift/llm/utils/argument.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1167,7 +1167,7 @@ def load_from_ckpt_dir(self) -> None:
11671167
]
11681168
for key in imported_keys:
11691169
value = getattr(self, key)
1170-
if key == 'dataset' and len(value) > 0:
1170+
if key in {'dataset', 'val_dataset'} and len(value) > 0:
11711171
continue
11721172
if key in {'dataset_test_ratio', 'system'} and value is not None:
11731173
continue
@@ -1180,6 +1180,10 @@ def load_from_ckpt_dir(self) -> None:
11801180
if self.dtype == 'AUTO':
11811181
self.dtype = sft_args.get('dtype')
11821182

1183+
# compat
1184+
if self.val_dataset is None:
1185+
self.val_dataset = []
1186+
11831187
@staticmethod
11841188
def check_ckpt_dir_correct(ckpt_dir) -> bool:
11851189
"""Check the checkpoint dir is correct, which means it must contains a `configuration.json` file.

‎swift/llm/utils/model.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2787,7 +2787,7 @@ def _new_forward(*args, **kwargs):
27872787
TemplateType.internvl,
27882788
requires=['transformers>=4.35', 'timm'],
27892789
support_flash_attn=True,
2790-
support_gradient_checkpointing=False,
2790+
tags=['multi-modal', 'vision'],
27912791
hf_model_id='OpenGVLab/InternVL-Chat-V1-5')
27922792
@register_model(
27932793
ModelType.internvl_chat_v1_5_int8,
@@ -2796,7 +2796,7 @@ def _new_forward(*args, **kwargs):
27962796
TemplateType.internvl,
27972797
requires=['transformers>=4.35', 'timm'],
27982798
support_flash_attn=True,
2799-
support_gradient_checkpointing=False,
2799+
tags=['multi-modal', 'vision'],
28002800
hf_model_id='OpenGVLab/InternVL-Chat-V1-5-int8')
28012801
def get_model_tokenizer_internvl(model_dir: str,
28022802
torch_dtype: Dtype,
@@ -2831,7 +2831,7 @@ def get_model_tokenizer_internvl(model_dir: str,
28312831
model.language_model.output.state.force_no_igemmlt = True
28322832

28332833
if model is not None:
2834-
_use_submodel_func(model, 'language_model', ['get_input_embeddings'])
2834+
_use_submodel_func(model, 'language_model', ['get_input_embeddings', 'gradient_checkpointing_enable'])
28352835
fix_internvl_inplace_bug(model)
28362836
if not hasattr(model, '__old_forward'): # Avoid double patching
28372837
forward = model.forward

‎swift/llm/utils/protocol.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ def random_uuid() -> str:
1313
class Model:
1414
id: str # model_type
1515
is_chat: bool # chat model or generation model
16+
1617
object: str = 'model'
1718
created: int = field(default_factory=lambda: int(time.time()))
1819
owned_by: str = 'swift'

‎swift/llm/utils/template.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -784,7 +784,7 @@ def data_collator(self, batch: List[Dict[str, Any]], padding_to: Optional[int] =
784784
['<|im_end|>'], INTERNLM_SYSTEM, ['<s><|im_start|>system\n{{SYSTEM}}<|im_end|>\n']))
785785

786786

787-
def replace_img_tab(query: str, history: History, replace_token: str) -> Tuple[str, History, List[str]]:
787+
def replace_img_tag(query: str, history: History, replace_token: str) -> Tuple[str, History, List[str]]:
788788
images_path = []
789789
pattern = r'<img>(.+?)</img>'
790790
new_history = []
@@ -818,7 +818,7 @@ def encode(self, example: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any
818818
history = example.pop('history', None)
819819
if history is None:
820820
history = []
821-
example['query'], example['history'], images_path = replace_img_tab(example['query'], history, '</s>')
821+
example['query'], example['history'], images_path = replace_img_tag(example['query'], history, '</s>')
822822

823823
images = []
824824
dtype = self.model.dtype
@@ -1122,7 +1122,7 @@ def encode(self, example: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any
11221122
history = example.pop('history', None)
11231123
if history is None:
11241124
history = []
1125-
example['query'], example['history'], images_path = replace_img_tab(example['query'], history, '<s>')
1125+
example['query'], example['history'], images_path = replace_img_tag(example['query'], history, '<s>')
11261126
images = []
11271127
for image_path in images_path:
11281128
image = _read_from_path(image_path)
@@ -1230,7 +1230,7 @@ def encode(self, example: Dict[str, Any]) -> Tuple[Dict[str, Any], Dict[str, Any
12301230
history = example.pop('history', None)
12311231
if history is None:
12321232
history = []
1233-
example['query'], example['history'], images_path = replace_img_tab(example['query'], history,
1233+
example['query'], example['history'], images_path = replace_img_tag(example['query'], history,
12341234
'<image_placeholder>')
12351235

12361236
inputs, _ = super().encode(example)

‎swift/llm/utils/utils.py‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -556,8 +556,12 @@ def inference_stream(model: PreTrainedModel,
556556
}
557557
template.model = model
558558
inputs, tokenizer_kwargs = template.encode(example)
559-
if len(inputs) == 0:
560-
raise ValueError('input_ids exceeds `max_length`. Please increase the value of `max_length`.')
559+
560+
truncation_strategy = kwargs.pop('truncation_strategy', 'delete')
561+
if len(inputs) == 0 and truncation_strategy == 'delete':
562+
# input_ids exceeds `max_length`. Please increase the value of `max_length`.
563+
return '', history
564+
561565
inputs.pop('labels', None)
562566
tokenizer = template.tokenizer
563567
device = next(model.parameters()).device
@@ -691,8 +695,9 @@ def inference(model: PreTrainedModel,
691695
template.model = model
692696
inputs, tokenizer_kwargs = template.encode(example)
693697

694-
truncation_strategy = kwargs.pop('truncation_strategy', None)
698+
truncation_strategy = kwargs.pop('truncation_strategy', 'delete')
695699
if len(inputs) == 0 and truncation_strategy == 'delete':
700+
# input_ids exceeds `max_length`. Please increase the value of `max_length`.
696701
return '', history
697702

698703
inputs.pop('labels', None)

0 commit comments

Comments
 (0)