Skip to content

Commit 35374f7

Browse files
Fix typo in tokenizer variable name
1 parent b9b0483 commit 35374f7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

generative-proof-of-concept-CPU-preprocessing-in-memory.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -441,7 +441,7 @@ def package_non_instruct_text(text: str, desired_samples: int, max_length_tokens
441441
# data += non_instruct_samples
442442

443443
##<<##<<
444-
x, y, vocab_size = prepare_data(data_0=non_instruct_samples, tokenizer_0=toeknizer, max_seq_length=MAX_SEQ_LENGTH, prompt_length = PROMPT_LENGTH)
444+
x, y, vocab_size = prepare_data(data_0=non_instruct_samples, tokenizer_0=tokenizer, max_seq_length=MAX_SEQ_LENGTH, prompt_length = PROMPT_LENGTH)
445445
# x, y, vocab_size = prepare_data() # data)
446446

447447
print("Input IDs shape:", len(x), "x", len(x[0]) if x else 0)

0 commit comments

Comments
 (0)