Skip to content

Commit 8f1da31

Browse files
nathanrchnSaibo-creator
authored andcommitted
fix the test file
1 parent f0df053 commit 8f1da31

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

tests/long_test_with_corpus.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,10 @@ def check_codebook_consistency_during_online_generation(
228228
prompt_len = 500
229229
assert prompt_len < seq_len
230230

231+
# replace pad token with 1
232+
token_chunk = token_chunk.to(torch.int64)
233+
token_chunk[token_chunk == 50256] = 1
234+
231235
original_sequence = token_chunk.tolist()
232236

233237
# Encode the data

0 commit comments

Comments
 (0)