Update generative-proof-of-concept-CPU-preprocessing-in-memory.py

david-thrower · web-flow · commit d6fb1c099e95 · 2025-10-17T22:30:33.000-04:00
Naming reference correction ...
diff --git a/generative-proof-of-concept-CPU-preprocessing-in-memory.py b/generative-proof-of-concept-CPU-preprocessing-in-memory.py
@@ -1049,8 +1049,8 @@ def create_dataset(raw_text_samples, tokenizer, sample_expansion_batch_size=50,
             dataset = tf.data.Dataset.from_generator(
                 lambda: generator_0,
                 output_signature=(
-                    tf.TensorSpec(shape=(generator.max_seq_length,), dtype=tf.int32),  # Use generator's parameter
-                    tf.TensorSpec(shape=(generator.vocabulary_size,), dtype=tf.float32)  # Use generator's parameter
+                    tf.TensorSpec(shape=(generator_0.max_seq_length,), dtype=tf.int32),  # Use generator's parameter
+                    tf.TensorSpec(shape=(generator_0.vocabulary_size,), dtype=tf.float32)  # Use generator's parameter
                 )
             )
           

Original file line number	Diff line number	Diff line change
`@@ -1049,8 +1049,8 @@ def create_dataset(raw_text_samples, tokenizer, sample_expansion_batch_size=50,`
`1049`	`1049`	`dataset = tf.data.Dataset.from_generator(`
`1050`	`1050`	`lambda: generator_0,`
`1051`	`1051`	`output_signature=(`
`1052`		`- tf.TensorSpec(shape=(generator.max_seq_length,), dtype=tf.int32), # Use generator's parameter`
`1053`		`- tf.TensorSpec(shape=(generator.vocabulary_size,), dtype=tf.float32) # Use generator's parameter`
	`1052`	`+ tf.TensorSpec(shape=(generator_0.max_seq_length,), dtype=tf.int32), # Use generator's parameter`
	`1053`	`+ tf.TensorSpec(shape=(generator_0.vocabulary_size,), dtype=tf.float32) # Use generator's parameter`
`1054`	`1054`	`)`
`1055`	`1055`	`)`
`1056`	`1056`