Skip to content

Commit 956d8dd

Browse files
Update generative-proof-of-concept-CPU-preprocessing-in-memory.py
Fix iteration on the Dataset.
1 parent 84ec89c commit 956d8dd

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

generative-proof-of-concept-CPU-preprocessing-in-memory.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1330,7 +1330,9 @@ def _expand_next_batch(self):
13301330
end_idx = min(start_idx + self.sample_expansion_batch_size, len(self.raw_text_samples))
13311331
collect()
13321332
if start_idx >= end_idx:
1333-
raise StopIteration("No more raw samples to process.")
1333+
self.current_index = 0 # raise StopIteration("No more raw samples to process.")
1334+
start_idx = 0
1335+
end_idx = min(self.sample_expansion_batch_size, len(self.raw_text_samples))
13341336

13351337
batch_samples = self.raw_text_samples[start_idx:end_idx]
13361338
self.current_index = end_idx

0 commit comments

Comments
 (0)