Skip to content

Commit 147aa97

Browse files
hjh0119Jintao-Huang
authored andcommitted
[sp] remove deprecated tokenizer (#4934)
1 parent 630cf94 commit 147aa97

File tree

1 file changed

+1
-1
lines changed
  • swift/trainers/sequence_parallel

1 file changed

+1
-1
lines changed

swift/trainers/sequence_parallel/utils.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -500,7 +500,7 @@ def _get_per_token_logps_grpo(self, model, inputs, sp_instance):
500500
_, _, labels, _, _, _ = sp_instance.pad_and_split_inputs(None, None, input_ids.clone(), None, None, None)
501501

502502
shape1 = logits.shape[1]
503-
labels = torch.where(labels == -100, self.tokenizer.pad_token_id, labels)
503+
labels = torch.where(labels == -100, self.processing_class.pad_token_id, labels)
504504
# calculate padding size for example, 9 to 10 if sp=2
505505
padding_size = shape1 * sp_instance.sp_world_size - origin_length
506506
# left shift one token to leave the last token

0 commit comments

Comments
 (0)