Skip to content

Commit 27b57af

Browse files
committed
Change FSDP to truncate dataset
Close #719
1 parent 88bd137 commit 27b57af

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

3.test_cases/pytorch/FSDP/src/model_utils/concat_dataset.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ def __iter__(self) -> Iterable[Dict[str, bytes]]:
2828
mask_buffer = []
2929
for sample in self.hf_dataset:
3030
encoded = self.tokenizer(sample['text'],
31-
truncation=False,
31+
truncation=True,
3232
padding=False)
3333
iids = encoded['input_ids']
3434
mask = encoded['attention_mask']

0 commit comments

Comments
 (0)