Update generative-proof-of-concept-CPU-preprocessing-in-memory.py

david-thrower · web-flow · commit 51dc82e8bb90 · 2025-10-14T18:33:24.000-04:00
Fine details with tf.data.Dataset.
diff --git a/generative-proof-of-concept-CPU-preprocessing-in-memory.py b/generative-proof-of-concept-CPU-preprocessing-in-memory.py
@@ -1372,11 +1372,14 @@ def create_dataset(raw_text_samples, tokenizer, sample_expansion_batch_size=10)
                     tf.TensorSpec(shape=(1, VOCABULARY_SIZE), dtype=tf.float32)   # Nested one-hot label
                 )
             )
+            # Set dataset to allow multiple epochs:
+            dataset = dataset.repeat()
+            # Batch it
+            dataset = dataset.batch(batch_size)
             return dataset
 
         phase_i_b_dataset = create_dataset(raw_text_samples=phase_i_b_samples, tokenizer=tokenizer, sample_expansion_batch_size=10)
-        dataset = dataset.repeat()
-        dataset = dataset.batch(batch_size)
+
 
 
         phase_i_b_history =\

Original file line number	Diff line number	Diff line change
`@@ -1372,11 +1372,14 @@ def create_dataset(raw_text_samples, tokenizer, sample_expansion_batch_size=10)`
`1372`	`1372`	`tf.TensorSpec(shape=(1, VOCABULARY_SIZE), dtype=tf.float32) # Nested one-hot label`
`1373`	`1373`	`)`
`1374`	`1374`	`)`
	`1375`	`+ # Set dataset to allow multiple epochs:`
	`1376`	`+ dataset = dataset.repeat()`
	`1377`	`+ # Batch it`
	`1378`	`+ dataset = dataset.batch(batch_size)`
`1375`	`1379`	`return dataset`
`1376`	`1380`
`1377`	`1381`	`phase_i_b_dataset = create_dataset(raw_text_samples=phase_i_b_samples, tokenizer=tokenizer, sample_expansion_batch_size=10)`
`1378`		`- dataset = dataset.repeat()`
`1379`		`- dataset = dataset.batch(batch_size)`
	`1382`	`+`
`1380`	`1383`
`1381`	`1384`
`1382`	`1385`	`phase_i_b_history =\`