Skip to content

Commit 408b915

Browse files
Update generative-proof-of-concept-CPU-preprocessing-in-memory.py
Add .batch() and .repeat() to dataset.
1 parent 9277fe8 commit 408b915

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

generative-proof-of-concept-CPU-preprocessing-in-memory.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1375,6 +1375,8 @@ def create_dataset(raw_text_samples, tokenizer, sample_expansion_batch_size=10)
13751375
return dataset
13761376

13771377
phase_i_b_dataset = create_dataset(raw_text_samples=phase_i_b_samples, tokenizer=tokenizer, sample_expansion_batch_size=10)
1378+
dataset = dataset.repeat()
1379+
dataset = dataset.batch(batch_size)
13781380

13791381

13801382
phase_i_b_history =\

0 commit comments

Comments
 (0)